Conference sharing method and device, equipment and storage medium

Abstract

The invention discloses a conference sharing method and device, equipment and a storage medium, and relates to the field of computers. The conference sharing method comprises the following steps: an agent terminal displays a first display interface according to received first rich media data sent by a first terminal, and obtains an editing operation of a user on the first display interface to obtain second rich media data; and the proxy terminal merges the first rich media data and the second rich media data and then sends the merged data to the first terminal, so that the first terminal displays a second display interface. Wherein the first display interface comprises to-be-shared content indicated by the first rich media data in the conference; and the first terminal interacts with other terminals participating in the conference through the proxy terminal. The second display interface comprises the content to be shared in the conference indicated by the first rich media data and the second rich media data.

Description

[0001] This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on July 5, 2023, with application number 202310819730.3 and application name “A conference sharing method and device”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of computers, and in particular to conference sharing methods, devices, equipment and storage media. Background Art

[0003] With the development of communication technology, video conferencing applications are becoming more and more widespread. At present, the application scenarios of video conferencing are not limited to the multiple parties participating in the conference only sharing desktops, videos, etc., but also operations such as adding tags. Take party A and party B participating in the video conference, party A and party B interact through the server, and take party A participating in the video conference sharing the desktop and adding tags as an example to illustrate. Party A often sends the shared desktop data and the marked data to the server. The server merges the shared desktop data and the marked data to obtain a video stream and sends it to party B.

[0004] However, due to hardware reasons, party B can only display according to the video stream, resulting in party A and party B participating in the video conference being unable to collaborate (such as interactive marking), which in turn leads to a poor experience for all parties participating in the video conference. Summary of the invention

[0005] The present application provides a conference sharing method, apparatus, device and storage medium, which solves the problem that the terminals participating in the above-mentioned video conference cannot perform collaborative operations, resulting in a poor user experience of participating in the video conference.

[0006] In a first aspect, the present application provides a conference sharing method, which can be applied to a computer system or to a computing device that supports the computer system to implement the conference sharing method, for example, the computing device can be a terminal. The conference sharing method may include: the proxy terminal displays a first display interface based on the first rich media data received and sent by the first terminal, and obtains the user's editing operation on the first display interface to obtain the second rich media data. The proxy terminal merges the first rich media data and the second rich media data and sends them to the first terminal for the first terminal to display the second display interface. The first display interface includes the content to be shared in the conference indicated by the first rich media data; the first terminal interacts with other terminals participating in the conference through the proxy terminal. The second display interface includes the content to be shared in the conference indicated by the first rich media data and the second rich media data.

[0007] In one possible scenario, the first terminal can only display the interface and cannot perform editing operations due to hardware reasons (hardware failure or hardware not supported) / limited functions (function not supported).

[0008] In the present application, the proxy terminal sends the first rich media data and the second rich media data merged to the first terminal, and the first terminal interacts with other terminals participating in the conference through the proxy terminal, thereby avoiding the problem that the first terminal cannot collaborate with other terminals participating in the conference due to hardware reasons or other problems in the first terminal, which makes it impossible to perform editing operations such as marking in the conference. It enables one party participating in the conference (the first terminal and the proxy terminal) to perform editing operations during the conference, and then multiple parties participating in the conference can collaborate, which improves the conference experience of the user of the first terminal.

[0009] In a possible example, the first rich media data includes one or more of the following: desktop sharing sub-data and image sub-data of the first terminal. The second rich media data may include one or more of the following: vector sub-data obtained by the user marking on the first display interface, text sub-data or image sub-data inserted on the first display interface, and desktop control sub-data inserted on the first display interface.

[0010] In a possible scenario, the editing operation includes one or more of the following: inserting annotations, texts, and images.

[0011] For example, the agent terminal receives a trigger operation of the user on the first display interface and obtains a markup inserted on the first display interface. The agent terminal receives the text inserted by the user and a designated position of the text in the first display interface, and the designated position is determined by the user through a click or a slide operation on the first interface. The agent terminal receives an image inserted by the user and a designated position of the image in the first display interface, and the designated position is determined by the user through a click or a slide operation on the first interface.

[0012] In a possible implementation manner, the conference sharing method further includes: the proxy terminal sending the first rich media data and the second rich media data to the second terminal, and the second terminal and the first terminal participate in the same conference.

[0013] In a possible example, the proxy terminal sends the first rich media data and the second rich media data to the second terminal respectively.

[0014] In another possible example, the proxy terminal combines the first rich media data and the second rich media data and sends the combined data to the second terminal.

[0015] Exemplarily, the proxy terminal may send the first rich media data and the second rich media data to the cloud server, and the cloud server forwards the first rich media data and the second rich media data to the second terminal.

[0016] In the present application, the proxy terminal sends the first rich media data and the second rich media data to the second terminal, avoiding the problem that the first terminal can only send the first rich media data to the second terminal due to hardware reasons (failure or lack of support) or functional reasons (lack of support). The proxy terminal sends the second rich media data obtained through editing operations to the second terminal, and realizes displaying the controls corresponding to the second rich media data on the second terminal, so that the first terminal and the second terminal can collaborate in the meeting, thereby improving the user experience of participating in the meeting.

[0017] In a possible implementation manner, the conference sharing method further includes: the proxy terminal displays a control component corresponding to the second rich media data on the first display interface.

[0018] In the present application, the proxy terminal can determine the positional relationship between the second rich media data and the first rich media data on the display interface while visualizing the second rich media data, and then determine whether the position of the annotation is correct, thereby improving the user experience of participating in the meeting.

[0019] For example, the first display interface also displays one or more annotations, texts, images, or desktop controls inserted by the user.

[0020] In a possible implementation manner, the rich media data is data that has been processed for anti-packet loss.

[0021] In the present application, the rich media data after anti-packet loss processing has redundant information to recover data that may be lost or damaged, thereby reducing the packet loss rate, thereby avoiding frame drops or screen distortion problems on the display interface, improving the smoothness of the display interface, and thus improving the user experience of participating in the meeting.

[0022] In a possible scenario, the proxy terminal and the first terminal are in the same local area network environment.

[0023] In a second aspect, the present application also provides a conference sharing device, which is applied to a computing device, and the conference sharing device includes various modules for executing the conference sharing method in the first aspect or any optional design of the first aspect. For example, the conference sharing device includes: a display module, an acquisition module and a sending module. Among them, the display module is used to display a first display interface according to the first rich media data received and sent by the first terminal; wherein the first display interface includes the content to be shared in the conference indicated by the first rich media data; the first terminal interacts with other terminals participating in the conference through a proxy terminal. The acquisition module is used to obtain the user's editing operation on the first display interface to obtain the second rich media data. The sending module is used to merge the first rich media data and the second rich media data and send them to the first terminal, so that the first terminal displays the second display interface, and the second display interface includes the content to be shared in the conference indicated by the first rich media data and the second rich media data.

[0024] In a possible example, the editing operation includes one or more of the following: inserting annotations, texts, and images.

[0025] In a possible scenario, the first rich media data includes one or more of the following: desktop sharing sub-data and image sub-data of the first terminal. The second rich media data may include one or more of the following: vector sub-data obtained by the user marking on the first display interface, text sub-data or image sub-data inserted on the first display interface, and desktop control sub-data inserted on the first display interface.

[0026] In a possible implementation manner, the sending module is further configured to: send the first rich media data and the second rich media data to a second terminal, and the second terminal participates in the same conference as the first terminal.

[0027] In a possible implementation manner, the display module is further configured to: display a control component corresponding to the second rich media data on the first display interface.

[0028] In a possible implementation manner, the rich media data is data that has been processed for anti-packet loss.

[0029] In a possible implementation manner, the proxy terminal and the first terminal are in the same local area network environment.

[0030] In a third aspect, the present application also provides a conference sharing method, which is applied to a conference sharing system, and the conference sharing system includes a first terminal and a proxy terminal. The first terminal interacts with other terminals participating in the conference through the proxy terminal, and the conference sharing method includes: the first terminal sends first rich media data to the proxy terminal. The proxy terminal displays a first display interface based on the first rich media data. The first display interface includes content to be shared in the conference indicated by the first rich media data. The proxy terminal obtains the user's editing operation on the first display interface and obtains the second rich media data. The proxy terminal merges the first rich media data and the second rich media data and sends them to the first terminal. The first terminal displays the second display interface, and the second display interface includes content to be shared in the conference indicated by the first rich media data and the second rich media data.

[0031] In a fourth aspect, the present application also provides a chip, comprising: an interface circuit and a power supply circuit; the interface circuit is used to obtain first rich media data sent by the first terminal, and the control circuit is used to execute the method in the above-mentioned first aspect or any possible implementation method of the first aspect.

[0032] In a fifth aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device; each computing device in the at least one computing device includes a memory and a processor, the memory is used to store computer instructions; when the processor executes the computer instructions, the method in the above first aspect or any possible implementation of the first aspect is implemented. The computing device may refer to a server, a processor included in the server, etc.

[0033] In a sixth aspect, the present application further provides a computer-readable storage medium, wherein a computer program or instruction is stored in the storage medium, and when the computer program or instruction is executed by a computing device, the method in the first aspect or any optional implementation of the first aspect is implemented.

[0034] In a seventh aspect, the present application further provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a computing device, the method in the first aspect or any optional implementation of the first aspect is implemented.

[0035] The beneficial effects of the second to seventh aspects above can be referred to the description of the first aspect or any implementation method of the first aspect, and will not be elaborated here.

[0036] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of a video conference collaboration scenario;

[0038] Figure 2 This is a schematic diagram of a weak network confrontation scenario for video conferencing;

[0039] Figure 3 An application scenario diagram of a computer system provided for this application;

[0040] Figure 4 Schematic diagram of the conference access method provided for this application Figure 1 ;

[0041] Figure 5 Schematic diagram of the conference access method provided for this application Figure 2 ;

[0042] Figure 6 A flowchart of the conference sharing method provided for this application;

[0043] Figure 7 A schematic diagram of the first display interface provided by this application;

[0044] Figure 8 A schematic diagram of a second display interface provided by this application;

[0045] Fig. 9 A schematic diagram of the process of the data synthesis method provided in this application;

[0046] Fig.10 A flowchart of the anti-packet loss processing method provided in this application;

[0047] Fig.11 Schematic diagram of the process of anti-packet loss method provided in this application Figure 1 ;

[0048] Fig.12 Schematic diagram of the process of anti-packet loss method provided in this application Figure 2 ;

[0049] Fig.13 A schematic diagram of the structure of a conference sharing device provided in this application;

[0050] Fig.14 A schematic diagram of the structure of a computing device provided in this application. DETAILED DESCRIPTION

[0051] To facilitate understanding, the technical terms involved in this application are first introduced.

[0052] A multipoint control unit (MCU) is a technical device used to solve weak network scenarios in video conferencing. In a video conference, when the terminal devices participating in the video conference face weak network environments such as limited network bandwidth, high latency or high packet loss rate, the use of an MCU can improve and optimize the transmission quality and experience of the video conference. In this application, the MCU is often used to process data during the connection and interaction of the video conference.

[0053] Session Initiation Protocol (SIP) is a text-based protocol used to establish, modify, and terminate real-time sessions, such as semantic calls, video calls, or video conferences.

[0054] H.323 is a voice and video communication protocol that is also widely used in multimedia communications on Internet Protocol (IP) networks. H.323 includes a series of protocols and specifications for establishing and managing multimedia communication sessions.

[0055] Luma, chroma data is a color space used to represent color images. Y represents luminance, and U and V represent chrominance.

[0056] Rich media data refers to a collection of one or more types of media information, and the types of media information may include: text, image, audio, video, animation, annotation, etc. In the present application, rich media data may include: one or more of vector sub-data (annotation), desktop sharing sub-data, image sub-data, desktop control sub-data, and text sub-data. Among them, desktop sharing sub-data, image sub-data, and desktop control sub-data may be video type data (video stream), and image sub-data may also be data in image format.

[0057] In online conference scenarios, there are many traditional conference terminal manufacturers, and many types of conference terminals (such as MCU) are released, and most of the conference terminals are deployed in the enterprise intranet. Take the A and B terminals participating in the online meeting as an example. In the cloud conference scenario, both A and B terminals are connected to the cloud server, and A and B terminals communicate with the cloud server through the Internet. Due to the differences between A and B terminal devices, it is often the case that A or B terminals can only share audio, video and desktop, resulting in a poor experience for users when participating in the meeting.

[0058] To solve the above problem that end a and end b can only share audio, video and desktop, a possible solution is provided below.

[0059] like Figure 1 As shown, Figure 1A schematic diagram of a video conference collaboration scenario. Figure 1 The terminals A, B and C participate in the video conference, and all of them establish connections with the cloud server 100. A conference service system is deployed in the cloud server 100.

[0060] The above conference service system includes media control service, media processing service and media adaptation service.

[0061] The media control service is used to establish connections with terminals A, B, and C, and connect terminals A, B, and C to the same conference.

[0062] The media processing service is used to receive and forward data from terminal A, terminal B, and terminal C, such as the terminal's desktop, whiteboard, mouse, and other data.

[0063] The media adaptation service is used to synthesize the desktop, whiteboard, mouse and other data of the terminal received by the media processing service, and send the synthesized data to the media processing service.

[0064] In a possible example, among terminals A, B, and C, terminal A is the main speaker, that is, terminal A performs operations such as desktop sharing and annotation, and terminal A sends data such as its desktop, whiteboard, mouse, annotation, and controls to the cloud server 100. The media processing service in the cloud server 100 receives the data sent by terminal A, and can directly forward the data sent by terminal A to terminal B; or, the cloud server 100 sends the data sent by terminal A to the media adaptation service for synthesis, and receives the synthesized data sent by the media adaptation service, and the cloud server 100 sends the synthesized data to terminals B and C.

[0065] However, in the above example, terminal B or terminal C can only view the desktop, annotations, controls and other contents shared by terminal A. Due to hardware failure, limited functions or lack of support, terminal B and terminal C cannot collaborate with terminal A when terminal A shares the desktop, annotations, controls and other contents. For example, terminal C annotates and displays the annotations based on the annotations of terminal A, resulting in a poor experience for users participating in video conferences.

[0066] The communication between end A and the cloud server, as well as between end B and the cloud server through the Internet, is prone to weak network scenarios (such as packet loss, latency, jitter, speed limit, etc.), resulting in frame drops or screen distortion when sharing audio, video, and desktop. The sharing effect between end A and end B is poor, and the user experience is poor.

[0067] To solve the above-mentioned problem of frame drops or screen distortion during audio, video and desktop sharing, poor sharing effect between end A and end B, and thus poor user experience, the following provides a possible solution.

[0068] like Figure 2 As shown, Figure 2 Schematic diagram of a weak network confrontation scenario for video conferencing. Figure 2 Terminal A and terminal B participate in the video conference. Figure 2 Figure a shows that terminal A directly establishes a connection with the cloud server 100, and terminal B establishes a connection with the cloud server 100 through MCU200.

[0069] In a possible example, terminal A establishes a connection with the media control service in the cloud server 100, and then directly sends the data of the desktop, whiteboard, mouse, annotation, control, etc. of terminal A to the media processing service in the cloud server 100, or terminal A receives the data sent by terminal B and forwarded by the cloud server 100. Since terminal A interacts directly with the cloud server 100, it will be affected by network bandwidth limitations and network stability, so it will encounter a weak network situation, resulting in a poor user experience. Terminal B establishes a connection with the media control service in the cloud server 100 through MCU200, and then sends the data of the desktop, whiteboard, mouse, annotation, control, etc. of terminal B to the media processing service in the cloud server 100 after being processed by MCU200, or receives the data sent by terminal A after being processed by MCU200.

[0070] Since the MCU200 is only deployed on terminal B participating in the video conference, when terminal A encounters a weak network scenario, the interface displayed on terminal A will drop frames or display a distorted screen, resulting in a poor user experience. MCU200 optimizes the data transmission method with cloud service 200. For example, MCU200 uses a lighter protocol and compression algorithm to reduce the size of data transmission and improve transmission efficiency, thereby completing data transmission faster in a weak network environment and reducing the possibility of transmission failure.

[0071] Figure 2 In b, a local server 300 is deployed inside the enterprise, and the local server 300 and the cloud server 100 are deployed with the same conference service system, that is, the conference service system on the cloud is sunk to the enterprise. Terminal A directly establishes a connection with the local server 300, and terminal B establishes a connection with the local server 300 through the MCU 200 thereon.

[0072] Terminal A and terminal B participating in the video conference can interact through the enterprise intranet to reduce the occurrence of weak network scenarios, but it is necessary to deploy multiple servers in the enterprise, occupying the enterprise's server resources.

[0073] In summary, the above solutions cannot effectively solve the problem that in a weak network scenario, the display interface on the terminal participating in the video conference may drop frames, resulting in a poor user experience.

[0074] To solve the problem that the terminals participating in the above-mentioned video conference cannot perform collaborative operations, resulting in a poor user experience for the participants in the video conference. The present application proposes a conference sharing method, which can be applied to a computing device. The conference sharing method includes: the proxy terminal displays a first display interface based on the first rich media data received and sent by the first terminal, and obtains the user's editing operation on the first display interface to obtain the second rich media data. The proxy terminal merges the first rich media data and the second rich media data and sends them to the first terminal for the first terminal to display the second display interface. The first display interface includes the data to be shared in the conference indicated by the first rich media data; the first terminal interacts with other terminals participating in the conference through the proxy terminal. The second display interface includes the data to be shared in the conference indicated by the first rich media data and the second rich media data.

[0075] In the present application, the proxy terminal sends the first rich media data and the second rich media data merged to the first terminal, and the first terminal interacts with other terminals participating in the conference through the proxy terminal, thereby avoiding the problem that the first terminal cannot collaborate with other terminals participating in the conference due to hardware reasons or other problems in the first terminal, which makes it impossible to perform editing operations such as marking in the conference. It enables one party participating in the conference (the first terminal and the proxy terminal) to perform editing operations during the conference, and then multiple parties participating in the conference can collaborate, which improves the conference experience of the user of the first terminal.

[0076] The conference sharing method provided by this application can be applied to Figure 3 The computer system shown in FIG. Figure 3 As shown, Figure 3 This is an application scenario diagram of a computer system provided in the present application. The computer system may include a server 300, a terminal 410, a terminal 420, and an agent terminal 500. The server 300 and the terminal 410, the server 300 and the agent terminal 500, and the agent terminal 500 and the terminal 420 may communicate via wired or wireless means.

[0077] The wired communication may be: Ethernet, optical fiber, and various peripheral component interconnect express (PCIe) buses provided inside the computer system for connecting the server 300 and the terminal 410, the server 300 and the proxy terminal 500, and the proxy terminal 500 and the terminal 420. The wireless communication may be: the Internet, wireless fidelity (WIFI), and ultra-wideband (UWB) technology, etc.

[0078] In a possible scenario, the proxy terminal 500 and the terminal 420 are in the same local area network environment.

[0079] Exemplarily, the proxy terminal 500 and the terminal 420 are located under the same firewall, which is a network security device used to monitor and control network traffic to protect the network from unauthorized access, malicious attacks, data leakage, etc. When the proxy terminal 500 and the terminal 420 are located in the same firewall, the proxy terminal 500 and the terminal 420 often belong to the same local area network (LAN) or network subnet.

[0080] Terminal 410, terminal 420 and proxy terminal 500 may be a mobile phone, tablet computer, handheld computer, personal computer (PC), cellular phone, personal digital assistant (PDA), wearable device (such as smart watch), smart home device (such as TV), car computer, game console, and augmented reality (AR) / virtual reality (VR) device, etc. The present application does not impose any special restrictions on the specific device forms of terminal 410, terminal 420 and proxy terminal 500. In this embodiment, the device forms of terminal 420 and proxy terminal 500 may be the same. For example, terminal 420 and proxy terminal 500 are both mobile phones. The device forms of terminal 420 and proxy terminal 500 may also be different, such as terminal 420 is a TV and proxy terminal 500 is a mobile phone.

[0081] Exemplarily, terminal 420 accesses the online conference through proxy terminal 500. Proxy terminal 500 displays the first display interface according to the first rich media data received from terminal 420, and obtains the user's editing operation on the first display interface to obtain the second rich media data. Proxy terminal 500 sends the data after merging the first rich media data and the second rich media data to terminal 420, which is used for terminal 420 to display the second display interface. Terminal 420 interacts with other terminals (such as terminal 410) participating in the online conference through proxy terminal 500.

[0082] It should be noted that Figure 3 The illustrated computer system architecture is only an example, and the device type or number within the system can be configured according to actual needs, and the embodiments of the present application are not limited thereto. For example, the computer system may also include more terminals.

[0083] In the first aspect, before executing the conference sharing method provided by the present application, the first terminal is also connected to the online conference (conference) through the proxy terminal. The following provides a possible implementation method of conference access. Figure 4 As shown, Figure 4 Schematic diagram of the conference access method provided for this application Figure 1 The conference access method can be executed by a computing device 600. The computing device 600 can be deployed at Figure 3 The proxy terminal 500 shown in FIG. Figure 4 As shown, the conference access method may include the following steps:

[0084] ①: The computing device 600 receives and configures the address of the first terminal and the address of the server.

[0085] Exemplarily, the computing device 600 receives and configures the address of the server input by the user, so as to establish a connection between the computing device 600 and the server, that is, the computing device 600 and the server can communicate with each other.

[0086] For example, the computing device 600 receives and configures an access address of the conference service system in the server.

[0087] The computing device 600 receives and configures the address or account information of the first terminal (terminal 420 ) to enable the computing device 600 to communicate with the terminal 420 .

[0088] In one possible scenario, the terminal 420 will also configure the address of the computing device 600 .

[0089] ②: The computing device 600 accesses the conference according to the address of the first terminal and the address of the server.

[0090] The computing device 600 establishes a connection between the terminal 420 and the server 300 through the computing device 600 according to the address of the first terminal and the address of the server, thereby enabling the computing device 600 to access the meeting, so that data can be sent between the terminal 420 and the computing device 600, and between the computing device 600 and the server 300.

[0091] In a possible implementation, the terminal 420 actively accesses the conference, and the computing device 600 accesses the conference according to the address of the first terminal and the address of the server, including: the computing device 600 receives the conference access instruction sent by the terminal 420 through SIP / H.323 according to the address of the terminal 420, and forwards the conference access instruction to the server 300 according to the address of the server 300. The computing device 600 establishes a connection with the server 300, and then the terminal 420 can access the conference through the computing device.

[0092] Exemplarily, terminal 420 creates a SIP message using the SIP format, the message includes conference access instructions and parameters, and the SIP message carries the address of computing device 600. Terminal 420 sends the SIP message to computing device 600. Computing device 600 obtains the SIP message according to the address of terminal 420. Computing device 600 adjusts the address of computing device 600 carried in the SIP message to the address of server 300, and sends the adjusted SIP to server 300. After receiving the adjusted SIP message, server 300 returns a corresponding SIP response, and computing device 600 receives the SIP response and forwards it to terminal 420, so that terminal 420 actively accesses the conference through computing device 600.

[0093] In a possible implementation, the terminal 420 passively accesses the conference, that is, the terminal 420 is called to access the conference. The computing device 600 accesses the conference according to the address of the first terminal and the address of the server, including: the computing device 600 receives the conference access instruction sent by the server 300 through SIP / H.323 according to the address of the server 300, and forwards the conference access instruction to the terminal 420 according to the address of the terminal 420, so as to realize the calling terminal 420 to access the conference.

[0094] In the present application, since the computing device 600 supports interactive annotation with other terminals of the meeting, the terminal 420 accesses the meeting through the computing device 600, thereby avoiding the problem that the terminal 420 directly accesses the meeting and may be unable to interactively annotate with other terminals participating in the meeting, thereby improving the user experience of participating in the meeting.

[0095] Another possible implementation of conference access is provided below. Figure 5 As shown, Figure 5 Schematic diagram of the conference access method provided for this application Figure 2 The conference access method can be executed by a computing device 600. The computing device 600 can be deployed at Figure 3 Alternatively, the computing device 600 may be a proxy terminal 500 as shown in FIG. Figure 3 In this implementation, after directly accessing the conference, the first terminal (which may be referred to as terminal 420) switches to accessing the conference through computing device 600. Figure 5 As shown, the conference access method includes the following steps:

[0096] ①: The computing device 600 receives and configures the server address and conference information.

[0097] The conference information includes the conference number, the identifier of the terminal 420 (such as the address), the conference password, and the like.

[0098] Exemplarily, the computing device 600 may receive the meeting information input by the user, or the computing device 600 may scan a QR code related to the meeting on the display interface of the terminal 420 to obtain the meeting information.

[0099] In a possible scenario, the computing device 600 will also receive and configure the address of the terminal 420 to enable the terminal 420 to establish a connection with the computing device 600 .

[0100] For the description of the address of the terminal 420 and the address of the server 300, see Figure 4 The contents of the address of the terminal 420 and the address of the server 300 shown in the corresponding contents are not described in detail here.

[0101] ②: The computing device 600 sends a conference access instruction to the server 300 according to the address and conference information of the server 300.

[0102] The conference access instruction is a message in the SIP / H.323 format, and includes the address and conference number of the server 300, the identifier (such as the address) of the terminal 420, the conference password, and the like.

[0103] Exemplarily, the computing device 600 sends a conference access instruction to the server 300 according to the address and conference information of the server 300 , including: the computing device 600 sends a conference access instruction carrying conference information to the server 300 according to the address of the server 300 .

[0104] ③: The server 300 renegotiates with the terminal 420 according to the conference access instruction and obtains a negotiation result.

[0105] Since the conference access instruction includes the address of terminal 420 and terminal 420 has already accessed the conference, server 300 needs to renegotiate with terminal 420 (or perform call transfer operations, etc.) to determine whether terminal 420 or terminal 420 accesses the conference through computing device 600, thereby obtaining a negotiation result.

[0106] The negotiation result includes the terminal 420 accessing the conference through the computing device 600 or maintaining the terminal 420 directly accessing the conference.

[0107] ④: Terminal 420 accesses the meeting through computing device 600 according to the negotiation result.

[0108] If the negotiation result is that terminal 420 accesses the conference through computing device 600, terminal 420 and server 300 establish a connection through computing device 600, that is, terminal 420 accesses the conference through computing device 600. Therefore, the conference is converted from terminal 420 directly accessing to terminal 420 accessing through computing device 600.

[0109] The terminal 420 subsequently sends the data directly to the computing device 600, and the computing device 600 forwards the data to the server 300 to enable interaction with other parties participating in the conference.

[0110] In the present application, since the computing device 600 supports interactive annotation with other terminals of the conference, the terminal 420 is converted from directly accessing the conference to accessing the conference through the computing device 600, thereby avoiding the problem that the terminal 420 directly accesses the conference and may be unable to interactively annotate with other terminals of the conference, thereby improving the user experience of participating in the conference.

[0111] In a second aspect, the present application provides a conference sharing method, which can be applied in Figure 3 The computer system shown in FIG. Figure 6 As shown, Figure 6 The conference sharing method provided in this application is a flowchart of the conference sharing method, which can be executed by a computing device 600, and the computing device 600 can be deployed in Figure 3 In this embodiment, the first terminal may be referred to as terminal 420, and the second terminal may be referred to as terminal 410. Figure 6 As shown, the conference sharing method includes the following steps S610 to S630.

[0112] S610 : The computing device 600 displays a first display interface according to the received first rich media data sent by the terminal 420 .

[0113] The first display interface includes the content to be shared in the conference indicated by the first rich media data, and the terminal 420 interacts with other terminals participating in the conference through the computing device 600 .

[0114] In a possible embodiment, upon receiving a sharing instruction from a user, the terminal 420 sends the first rich media data to the computing device 600 .

[0115] The user's sharing instruction may be an instruction input by the user at the terminal 420 , or an instruction triggered by the user on the computing device 600 .

[0116] Exemplarily, when terminal 420 is the main speaker in the meeting, the user triggers a sharing instruction on the computing device 600, and the computing device 600 sends the sharing instruction to the terminal 420. The terminal 420 sends the first rich media data to the computing device 600 according to the sharing instruction, thereby enabling the first rich media data to be shared with other terminals participating in the meeting.

[0117] Alternatively, when terminal 420 is a collaborating party in the meeting, that is, when the collaborative speaker shares some data, the user triggers a sharing instruction on the computing device 600, and the computing device 600 sends the sharing instruction to the terminal 420. Terminal 420 sends the first rich media data to the computing device 600 according to the sharing instruction, thereby sharing the first rich media data with other terminals participating in the meeting.

[0118] In a possible embodiment, due to hardware reasons of the terminal 420, operations such as inserting annotations or data cannot be performed directly on the terminal 420. Therefore, the computing device 600 can only receive one or more of the desktop sharing sub-data or the image sub-data of the terminal 420, so the first rich media data includes at least one of the desktop sharing sub-data and the image sub-data.

[0119] The desktop sharing sub-data is used to indicate the web page interface, program interface, or main interface of the terminal to be shared currently displayed on the desktop.

[0120] The image sub-data is used to indicate the video inserted by the user in the display interface, etc.

[0121] In a possible example, the image sub-data also includes a position where a video is inserted into the display interface, and the video may be a real-time video captured by the terminal 420 using a camera.

[0122] In a possible scenario, the desktop sharing sub-data and the image sub-data may be presented in the form of a video stream.

[0123] like Figure 7 As shown, Figure 7 This is a schematic diagram of the first display interface provided in this application. Figure 7 A in the figure shows the first display interface displayed by the computing device 600 according to the desktop sharing sub-data and the image sub-data sent by the terminal 420. Correspondingly, the desktop sharing sub-data is expressed as a shared desktop in the first display interface, and the image sub-data is expressed as a video in the first display interface, that is, the first display interface includes the shared desktop and the video.

[0124] S620: The computing device 600 obtains the user's editing operation on the first display interface to obtain second rich media data.

[0125] In a possible implementation, the computing device 600 obtains the user's editing operation on the first display interface to obtain the second rich media data, including: the computing device 600 obtains the user's editing operation on the first display interface through various input devices (keyboard, mouse, touch screen, etc.), and parses and processes the editing operation to generate corresponding input events, such as keyboard key events, mouse click events, touch screen gesture events, etc. The computing device 600 monitors the aforementioned input events and then processes the corresponding events accordingly to obtain the second media data.

[0126] Exemplarily, if the input event is a keyboard key event, the computing device 600 inserts the character indicated by the keyboard key event into the first display interface, and then obtains the text inserted into the second rich media data.

[0127] If the input event is a mouse click event, the computing device 600 sequentially connects the first coordinate indicated by the mouse click event and multiple coordinates corresponding to subsequent mouse movements to obtain the mouse movement trajectory, and then obtains the annotation inserted in the second rich media data.

[0128] If the input event is a touch screen gesture event, the computing device 600 sequentially connects the click coordinates or multiple coordinates corresponding to the slide indicated by the touch screen gesture to obtain the movement track of the mouse, and then obtains the annotation inserted in the second rich media data.

[0129] In a possible scenario, the editing operation includes: inserting one or more of annotations, texts, and images.

[0130] Exemplarily, the computing device 600 receiving the annotation inserted on the first display interface includes: the computing device 600 receives a trigger operation of the user on the first display interface to obtain the annotation inserted on the first display interface.

[0131] The trigger operation may be a click or a slide operation performed by the user on the first display interface using a mouse or a finger.

[0132] In a possible embodiment, before the computing device 600 receives the trigger operation of the user, the computing device 600 further includes: the computing device 600 receives the annotation type selected by the user on the first display interface.

[0133] The annotation type may be a bounding box, a bounding box color, a line width and color, an occlusion (such as a mosaic), and the like.

[0134] The computing device 600 receives text inserted on the first display interface, including: the computing device 600 receives the text inserted by the user and the specified position of the text in the first display interface, where the specified position is determined by the user through a click or slide operation on the first interface.

[0135] The computing device 600 receiving the image inserted on the first display interface includes: the computing device 600 receiving the image inserted by the user and a designated position of the image in the first display interface. The designated position is determined by the user through a click or slide operation on the first interface.

[0136] In a possible scenario, the second rich media data may include one or more of vector sub-data obtained by a user marking on the first display interface, text sub-data inserted on the first display interface, and image sub-data inserted on the first display interface.

[0137] The vector sub-data is used to indicate the data generated when the user marks on the display interface. For example, the user uses the mouse to click or slide on the first interface, and the terminal determines the coordinate information of the click or slide operation on the display interface, that is, the vector sub-data.

[0138] The text sub-data is used to indicate the text inserted by the user in the display interface. In a possible example, the text sub-data also includes the position where the text is inserted into the display interface.

[0139] The image sub-data is also used to indicate the image inserted by the user in the display interface. In a possible example, the image sub-data also includes the position where the image is inserted into the display interface.

[0140] The above image can be in the following formats: jpg (joint photographic group), jpeg (joint photographic experts group), png (portable network graphics), gif (graphics interchange format), bmp (bitmap), or WebP.

[0141] In another possible scenario, the second rich media data may also include desktop control sub-data inserted by the user on the first display interface.

[0142] Desktop control subdata is used to indicate a collection of widgets such as whiteboard / notepad, calculator, weather, calendar, etc.

[0143] In a possible embodiment, the computing device 600 may further display a control component corresponding to the second rich media data on the first display interface.

[0144] The computing device 600 displays on the first display interface the editing operations performed by the user of the computing device 600 on the first display interface, and obtains one or more control components in the annotations, text, and images, that is, displays one or more of the above-mentioned vector sub-data, text sub-data, image sub-data, or desktop control sub-data.

[0145] like Figure 7 As shown in b, the computing device 600 displays on the first display interface the annotations obtained by the user through clicking and moving the mouse, as well as the annotations obtained by the user through moving gestures.

[0146] The computing device 600 also displays control components for the text and the image inserted by the user on the first display interface.

[0147] Figure 7 The content indicated by the dotted line shown in b is the control component corresponding to the second rich media data.

[0148] In the present application, the computing device 600 can determine the positional relationship between the second rich media data and the first rich media data on the display interface while visualizing the second rich media data, and further determine whether the position of the annotation is correct, thereby improving the user experience of participating in the meeting.

[0149] In a possible embodiment, the computing device 600 obtaining the user's editing operation on the first display interface includes: the computing device 600 obtaining the user's boundary operation on the first display interface at the computing device 600.

[0150] In one possible example, the user at the computing device 600 and the user at the terminal 420 are the same user.

[0151] In another possible example, the user at the computing device 600 and the user at the terminal 420 are different users.

[0152] S630: The computing device 600 combines the first rich media data and the second rich media data and sends the combined data to the terminal 420, so that the terminal 420 displays the second display interface.

[0153] The second display interface includes the content to be shared in the conference indicated by the first rich media data and the second rich media data.

[0154] In a possible implementation, the computing device 600 combines the first rich media data and the second rich media data and sends the combined data to the terminal 420 , including: after the computing device 600 combines the first rich media data and the second rich media data to obtain a video stream, sending the video stream to the terminal 420 .

[0155] For the content of the terminal 420 after the computing device 600 combines the first rich media data and the second rich media data, please refer to the following Fig. 9 The description is not repeated here.

[0156] In one possible example, Figure 8 As shown, Figure 8The second display interface of the terminal 420 shows the content to be shared corresponding to the first rich media data and the second rich media data, such as shared desktop, video, annotation, text, image.

[0157] In a possible embodiment, the computing device 600 receives the first rich media data sent by the terminal 420. After the computing device 600 obtains the second rich media data, the computing device 600 sends the first rich media data and the second rich media data to the terminal 410. The terminal 410 and the terminal 420 are connected to the same conference.

[0158] In a possible implementation, the computing device 600 sends the first rich media data and the second rich media data to the terminal 410 respectively.

[0159] In another possible implementation, the computing device 600 combines the first rich media data and the second rich media data and sends the combined data to the terminal 410 .

[0160] For the content of the computing device 600 combining the first rich media data and the second rich media data and sending them to 410, please refer to the following Fig. 9 The description that the computing device 600 combines the first rich media data and the second rich media data and sends the combined data to 420 is not described in detail here.

[0161] The computing device 600 sends the first rich media data and the second rich media data to the terminal 410, which are used to display one or more of the desktop shared sub-data, image sub-data, vector sub-data obtained by the user annotating the desktop shared sub-data and image sub-data through the proxy terminal, text sub-data, and desktop control sub-data inserted by the user in the desktop shared sub-data and image sub-data through the proxy terminal of the terminal 420 on the terminal 410, thereby avoiding the problem that the terminal 420 can only share the desktop shared sub-data and desktop control sub-data with the terminal 410 due to hardware reasons, and thus enabling the user of the terminal 420 to annotate, insert text and other data in the meeting, so that the terminal 420 and the terminal 410 can collaborate in the meeting, thereby improving the user experience of participating in the meeting.

[0162] With respect to the content in which the computing device 600 combines the first rich media data and the second rich media data and sends the combined data to the terminal 420 , a possible embodiment is provided below.

[0163] like Fig. 9 As shown, Fig. 9 A schematic diagram of the flow chart of the data synthesis method provided in this application. Fig. 9 The content shown may include the following steps S910 to S930.

[0164] S910 , the computing device 600 performs data conversion on the first vector sub-data of the terminal 420 in the second rich media data to obtain first brightness-chrominance data.

[0165] The first vector sub-data of the terminal 410 indicates the data obtained by the user marking on the display interface of the computing device 600 .

[0166] Exemplarily, the user controls the movement trajectory or coordinates of the mouse on the display interface of the computing device 600, or the user clicks or slides on the display interface of the computing device 600.

[0167] In one possible implementation, the computing device 600 performs data conversion on the first vector sub-data of the terminal 420 in the second rich media data to obtain the first brightness-chrominance data, including: the computing device 600 converts the first vector data into grayscale values, and then maps the grayscale values ​​to the brightness range and determines the chrominance value, thereby obtaining the first brightness-chrominance sub-data corresponding to the first vector data.

[0168] Exemplarily, the computing device 600 uses a grayscale transformation function to convert the movement trajectory or coordinates in the first vector data into corresponding grayscale values. The computing device 600 maps the grayscale value to the brightness range through a linear or nonlinear brightness transformation function, such as the computing device 600 multiplies the original grayscale value by a scaling factor to expand or reduce the brightness range. The computing device 600 uses a set constant to represent the chromaticity value, thereby obtaining the first brightness-chromaticity sub-data.

[0169] In a possible scenario, the first vector sub-data further includes a label type, and the computing device 600 determines the first luminance-chrominance sub-data according to the color in the label type.

[0170] Exemplarily, the computing device 600 determines the YUV of the pixel point in the moving trajectory or coordinates according to the annotation type in the first vector data.

[0171] For example, the computing device 600 determines red (red, R), green (green, G), and blue (blue, B) according to the color in the labeled type, and then uses the following formula Y=0.29*R+0.58*G+0.11*B, U=-0.14*R-0.28*G+0.43*B, V=0.61*R-0.51*G-0.10*B to obtain the brightness-chrominance sub-data A.

[0172] In a possible scenario, the first luminance-chrominance data includes a plurality of YUV data, such as one or more YUV data corresponding to each second.

[0173] S920: The computing device 600 decodes the image sub-data and / or the desktop sharing sub-data of the terminal 420 in the first rich media data to obtain a plurality of video frames.

[0174] Since the desktop sharing sub-data or the image sub-data is expressed in the form of a video stream, the computing device 600 needs to decode the video stream to obtain a plurality of video frames (images).

[0175] In a possible example, the computing device 600 decodes the desktop sharing sub-data or the image sub-data through a decoder to obtain a plurality of video frames. The decoder may be deployed inside the computing device 600 or be separately provided.

[0176] In a possible scenario, the desktop sharing sub-data and the image sub-data are included in a set of video streams.

[0177] S930 , the computing device 600 encodes the multiple video frames and the first luminance-chrominance data and sends them to the terminal 420 .

[0178] In one possible implementation, the computing device 600 encodes multiple video frames and first brightness-chrominance data and sends them to the terminal 420, including: the computing device 600 combines the multiple video frames with the first brightness-chrominance data, and then encodes them to obtain a video stream, and then the computing device 600 sends the video stream to the terminal 420 for displaying a second display interface on the terminal 420.

[0179] Exemplarily, the computing device 600 combines the video frame A in the plurality of video frames with the YUV-A in the first luminance-chrominance data to obtain a combined image A. The video frame A corresponds to the YUV-A, and the video frame A and the YUV-A may be data generated in the same second or in the same 1 / 2 second during the conference, which is not limited in this application.

[0180] The computing device 600 combines the multiple video frames with the multiple YUV data in the first luminance-chrominance data one by one according to the above example to obtain multiple combined images. The computing device 600 encodes the multiple combined images using an encoder in the computing device 600 to obtain a video stream.

[0181] In a possible scenario, the computing device 600 receives third rich media data obtained by the user of the terminal 410 marking on the display interface corresponding to the first rich media data and the second rich media data. The computing device 600 also performs data conversion on the vector sub-data a in the third rich media data in the above S810, and merges it with the result obtained by the conversion of the first vector sub-data to obtain the second brightness-color data. The computing device 600 decodes the image sub-data and / or desktop sharing sub-data of the terminal 420 in the first rich media data to obtain multiple video frames. The computing device 600 encodes the multiple video frames and the second brightness-color data and sends them to the terminal 420.

[0182] In a possible embodiment, the present application also provides a conference sharing method, which can be applied in Figure 3 The conference sharing method can be executed by a computing device 600. The computing device 600 can be deployed at Figure 3 In this embodiment, the first terminal may be referred to as terminal 420, and the second terminal may be referred to as terminal 410. The conference sharing method includes the following steps:

[0183] ①: The computing device 600 displays the display interface a according to the received rich media data a sent by the terminal 410.

[0184] ②: The computing device 600 obtains the user's editing operation on the display interface a and obtains the rich media data b.

[0185] ③: The computing device 600 combines the rich media data a and the rich media data b and sends them to the terminal 420, so that the terminal 420 displays the display interface b.

[0186] This embodiment and Figure 6 The conference sharing methods shown differ in that:

[0187] Difference 1: The computing device 600 receives the rich media data a sent by the terminal 410 .

[0188] The rich media data a includes one or more of: desktop sharing sub-data, image sub-data, desktop control sub-data of terminal 410, vector sub-data obtained by the user of terminal 410 annotating the desktop sharing sub-data, image sub-data, and desktop control sub-data of terminal 410, and text sub-data inserted by the user of terminal 410 into the desktop sharing sub-data, image sub-data, and desktop control sub-data of terminal 410. The terminal 410 is any terminal among the other terminals participating in the conference.

[0189] For the description of vector sub-data, desktop control sub-data, desktop sharing sub-data, text sub-data, and image sub-data, see Figure 6The contents of the first rich media data and the second rich media data shown in the corresponding embodiment are not described in detail here.

[0190] Difference 2: After obtaining the rich media data b, the computing device 600 sends the rich media data b to the terminal 410 .

[0191] The computing device 600 sends rich media data b to the terminal 410 for displaying annotations or inserted texts made by the proxy terminal and the user of the terminal 420 on the terminal 410, thereby realizing collaboration between the terminal 420 and the terminal 410 and improving the user experience of the conference participants.

[0192] Difference 3: The computing device 600 combines the rich media data a and the rich media data b and sends them to the terminal 420, including: the computing device 600 performs data conversion on the vector sub-data a of the terminal 410 in the rich media data a and the vector sub-data b of the terminal 420 in the rich media data b to obtain third luminance-chrominance data. The computing device 600 decodes one or more of the desktop control sub-data, desktop sharing sub-data, or image sub-data of the terminal 410 in the rich media data a to obtain multiple video frames. The computing device 600 encodes the multiple video frames and the third luminance-chrominance data and sends them to the terminal 420.

[0193] For the data conversion of the computing device 600 on the vector sub-data a of the terminal 410 in the rich media data a and the data conversion of the computing device 600 on the vector sub-data b of the terminal 420 in the rich media data b, please refer to the above Fig. 9 The description of the data conversion performed by the computing device 600 on the first vector sub-data of the terminal 420 in the second rich media data is not repeated here.

[0194] The computing device 600 performs data conversion on the vector sub-data a of the terminal 410 in the rich media data a to obtain the luminance-chrominance sub-data a; the computing device 600 performs data conversion on the vector sub-data b of the terminal 420 in the rich media data b to obtain the luminance-chrominance sub-data b. The computing device 600 combines the luminance-chrominance sub-data a and the luminance-chrominance sub-data b to obtain third luminance-chrominance data.

[0195] In one possible implementation, the computing device 600 combines the luminance-chrominance sub-data A and the luminance-chrominance sub-data B to obtain third luminance-chrominance data, including: the computing device 600 adds the YUV values ​​in the luminance-chrominance sub-data a and the luminance-chrominance sub-data b respectively to obtain the third luminance-chrominance data.

[0196] In a possible scenario, if a pixel region has both luminance-chrominance sub-data a and luminance-chrominance sub-data b, only luminance-chrominance sub-data a or luminance-chrominance sub-data b is retained to obtain third luminance-chrominance data.

[0197] In order to solve the problem of frame drop or screen distortion in the display interface of one or more parties participating in the online conference in a weak network scenario, the present application provides a possible embodiment, wherein the rich media data is data that has been processed for anti-packet loss.

[0198] In view of the above-mentioned rich media data being data after anti-packet loss processing, the following provides a possible implementation method, such as Fig.10 As shown, Fig.10 600 is a flowchart of the anti-packet loss processing method provided in this application. The anti-packet loss processing can be performed by a computing device 600. The computing device 600 can be deployed at Figure 3 The proxy terminal 500 shown in FIG. Fig.10 As shown, the anti-packet loss processing may include the following:

[0199] Fig.10 Figure a shows that the computing device 600 receives data a sent by the server 300 and processes the data a using an anti-packet loss algorithm to obtain first rich media data.

[0200] Fig.10 Middle b shows: the computing device 600 sends the first rich media data processed by the anti-packet loss algorithm and / or the second rich media data processed by the anti-packet loss algorithm to the server 300 .

[0201] In a possible example, the anti-packet loss algorithm may be a forward error correction (FEC) algorithm, which is an algorithm used to detect and correct errors during data transmission. The computing device 600 uses FEC to add redundant information to the data, and the receiver can use the redundant information to recover the data that may be lost or damaged, thereby reducing the packet loss rate, improving the fluency of the display interface, and thus improving the user's experience in participating in the meeting.

[0202] The computing device 600 will use FEC to process data before sending it, such as using FEC to encode the data (encoder, enc) before sending it, and the second rich media data is obtained by enc processing the data using FEC, and this process can be called FECenc. The computing device 600 will also use FEC to process the data after receiving it, such as using FEC to decode the data (decoder, dec) to obtain the first rich media data, and this process can be called FECdec.

[0203] In a possible embodiment, the anti-packet loss algorithm also includes jitter buffering. After receiving the data, the computing device 600 first performs jitter buffering (JB) to obtain a more uniform data stream.

[0204] JB is an algorithm for processing data jitter in network transmission. The computing device 600 uses JB to adjust and reorder the received data so that the computing device 600 can process a more uniform data stream when processing the data subsequently.

[0205] Exemplarily, before receiving the data b sent by the terminal 420, the computing device 600 first performs JB processing and then performs FECdec.

[0206] In a possible embodiment, the anti-packet loss algorithm also includes the configuration of fragextra (fragment additional parameters). Before the computing device 600 performs secenc on the data to be sent (the data before secenc on the first rich media data and / or the second rich media data), it can also adjust the data transmission parameters of the data to be sent according to the configured fragextra parameters.

[0207] The fragextra parameters are used to describe and control the transmission and processing of P frames. Fragextra includes the number of slices, MB (macroblock) distribution, QP (quantization parameter) value, etc.

[0208] The computing device 600 can improve the transmission efficiency and quality of the data to be sent, detect the transmission delay and loss of the data to be sent, and thus improve the user experience of participating in the meeting by adjusting the fragextra parameters.

[0209] In a possible embodiment, the anti-packet loss algorithm further includes a pacing algorithm, and the computing device 600 adjusts the transmission rate of the first rich media data and / or the second rich media data through pacing.

[0210] The pacing is used to limit the sending rate of data packets, so that the computing device 600 can gradually adapt to the receiving capability of the terminal 420 or the terminal 410, and avoid the loss or accumulation of data packets.

[0211] The computing device 600 uses pacing to ensure continuous transmission of the first rich media data and / or the second rich media data, thereby preventing the display interface of the terminal 420 or the terminal 410 from freezing or being distorted.

[0212] In a possible embodiment, upon detecting that a problem occurs during the transmission of the first rich media data and / or the second rich media data, the computing device 600 may retransmit the first rich media data and / or the second rich media data.

[0213] In one possible implementation, the anti-packet loss algorithm also includes a retransmission algorithm (retransmission, retrans). The computing device 600 uses retrans. When packet loss or errors occur in the transmission of the first rich media data and / or the second rich media data, retrans will attempt to resend the lost or damaged data packets to ensure the transmission of the complete first rich media data and / or the second rich media data.

[0214] Retrans will make a judgment based on the feedback information or network status of terminal 420 or terminal 410 and decide whether to retransmit a specific data packet. This can improve the reliability and quality of the transmission of the first rich media data and / or the second rich media data, reduce the freeze or screen damage problems that may occur on the display interface of terminal 420 or terminal 410, and improve the user experience of participating in the meeting.

[0215] In view of the fact that the rich media data in the above embodiment is data that has been processed for anti-packet loss, two possible examples are provided below.

[0216] In a first possible example, the computing device 600 combines the first rich media data and the second rich media data and sends the combined data to the server 300. Fig.11 As shown, Fig.11 Schematic diagram of the process of anti-packet loss method provided in this application Figure 1 , Fig.11 The following is shown:

[0217] ①: After the computing device 600 encodes the video stream, it adjusts fragextra to control the transmission and processing of P frames in the video stream.

[0218] ②: The computing device 600 uses a forward error correction algorithm to perform forward error correction coding processing on the video stream after fragextra adjustment.

[0219] ③: The computing device 600 uses a speed adjustment algorithm to adjust the transmission rate of the video stream after forward error correction processing.

[0220] The computing device 600 sends the video stream processed by the above ①-③ to the server 300. If packet loss or error occurs during the transmission, the following steps will be performed:

[0221] ④: The computing device 600 uses a retransmission algorithm to retransmit the video stream processed by the above ①-③ to ensure the transmission of a complete video stream.

[0222] In a possible scenario, after the computing device 600 receives the data sent by the terminal 420 , it will perform jitter buffering.

[0223] In a second possible example, the computing device 600 sends the merged first rich media data to the terminal 420 as an example. Fig.12 As shown, Fig.12 Schematic diagram of the process of anti-packet loss method provided in this application Figure 2 , Fig.12 The following is shown:

[0224] ①: After the computing device 600 receives the data c sent by the server 300, it performs jitter buffering on the data c.

[0225] ②: The computing device 600 uses a forward error correction algorithm to perform forward error correction decoding processing on the data c after jitter buffering to obtain decoded data.

[0226] Exemplarily, the data c sent by the server 300 is the data c sent by the terminal 410 to the server 300, and the data c includes desktop sharing sub-data, desktop control sub-data, image sub-data, vector sub-data, and text sub-data of the terminal 410.

[0227] It is understandable that, in order to implement the functions in the above-mentioned embodiments, the computing device 600 includes hardware structures and / or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0228] Combined with the above Figures 3 to 12 , describes in detail the conference sharing method provided by this embodiment, and will now be combined with Fig.13 , describing the conference sharing device provided according to this embodiment.

[0229] Fig.13 A schematic diagram of the structure of a conference sharing device provided in the present application. The conference sharing device can be used to implement the functions of the computing device in the above method embodiment, and thus can also achieve the beneficial effects of the above method embodiment. In this embodiment, the conference sharing device can be a module (such as a chip) of the computing device.

[0230] like Fig.13As shown, the conference sharing device 1300 includes a display module 1310, an acquisition module 1320 and a sending module 1330. The conference sharing device 1300 is used to implement the above Figures 4 to 12 The functions in the method embodiments shown in .

[0231] The display module 1310 is configured to display a first display interface according to the received first rich media data sent by the first terminal, wherein the first display interface includes the content to be shared in the conference indicated by the first rich media data; the first terminal interacts with other terminals participating in the conference through the proxy terminal.

[0232] The acquisition module 1320 is used to acquire the user's editing operation on the first display interface to obtain second rich media data.

[0233] The sending module 1330 is used to merge the first rich media data and the second rich media data and send them to the first terminal, so that the first terminal displays a second display interface, and the second display interface includes the content to be shared in the conference indicated by the first rich media data and the second rich media data.

[0234] It is worth noting that the computing device 600 according to the embodiment of the present application may correspond to the conference sharing device 1300 in the embodiment of the present application, and may correspond to executing the method according to the embodiment of the present application. Figures 4 to 12 The corresponding corresponding subjects, and the operations and / or functions of each module in the conference sharing device 1300 are respectively to achieve Figures 4 to 12 For the sake of brevity, the corresponding processes of each method in the corresponding embodiments are not repeated here.

[0235] in addition, Fig.13 The conference sharing device 1300 shown can also be implemented by a communication device, where the communication device may refer to the computing device 600 in the aforementioned embodiment, or, when the communication device is a chip or a chip system applied to the computing device 600, the conference sharing device 1300 can also be implemented by a chip or a chip system.

[0236] An embodiment of the present application further provides a chip system, which includes a control circuit and an interface circuit, and the interface circuit is used to obtain the first rich media data sent by the terminal 420 to implement the function of the computing device 600 in the above method.

[0237] The embodiment of the present application also provides a conference sharing method, which is applied to the above-mentioned computer system (conference sharing system), and the conference sharing system includes a first terminal (terminal 420) and a proxy terminal 500, and the first terminal interacts with other terminals participating in the conference through the proxy terminal. When the proxy terminal executes the steps in the conference sharing method, it can be executed by a computing device 600 deployed in the proxy terminal 500. The method includes:

[0238] The terminal 420 sends the first rich media data to the proxy terminal.

[0239] The computing device 600 displays a first display interface according to the first rich media data; wherein the first display interface includes the content to be shared in the conference indicated by the first rich media data.

[0240] The computing device 600 obtains the user's editing operation on the first display interface to obtain second rich media data.

[0241] The computing device 600 combines the first rich media data and the second rich media data and sends the combined data to the terminal 420 .

[0242] Terminal 420 displays a second display interface, where the second display interface includes the content to be shared in the conference indicated by the first rich media data and the second rich media data.

[0243] It is worth noting that the conference sharing method of this embodiment may also include the above Figure 4-Figure 12 For the sake of brevity, the steps performed by the terminal 420 and the computing device 600 are not described in detail here.

[0244] The present application also provides a computing device. Fig.14 As shown, Fig.14 A schematic diagram of the structure of a computing device provided in the present application, the computing device includes: a bus 1402, a processor 1404, a memory 1406 and a communication interface 1408. The processor 1404, the memory 1406 and the communication interface 1408 communicate through the bus 1402. The computing device can be a server or a terminal device. It is worth noting that the present application does not limit the number of processors and memories in the computing device.

[0245] The bus 1402 may be, but is not limited to: a PCIe bus, a universal serial bus (USB), or an inter-integrated circuit bus (I2C), an EISA bus, an UB, a CXL, a CCIX, etc. The bus 1402 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.14 The bus 1402 may include a path for transmitting information between various components of the computing device (eg, memory 1406, processor 1404, communication interface 1408).

[0246] The processor 1404 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0247] The memory 1406 may include a volatile memory, such as a random access memory (RAM). The memory 1406 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0248] The memory 1406 stores executable program codes, and the processor 1404 executes the executable program codes to respectively implement the functions of the display module, the acquisition module, and the sending module, thereby implementing the conference sharing method. That is, the memory 1406 stores instructions for executing the conference sharing method.

[0249] Alternatively, the memory 1406 stores executable codes, and the processor 1404 executes the executable codes to respectively implement the functions of the display module, the acquisition module, and the sending module, thereby implementing the conference sharing method. That is, the memory 1406 stores instructions for executing the conference sharing method.

[0250] The communication interface 1408 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device and other devices or a communication network.

[0251] The present application also provides a computing device cluster. The computing device cluster includes at least one Fig.14 The same instructions for executing the conference sharing method may be stored in the memory of one or more computing devices in the computing device cluster.

[0252] The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0253] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network.

[0254] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the conference sharing method.

[0255] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the conference sharing method.

[0256] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, or a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0257] In each embodiment of the present application, if there is no special explanation and logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships. The various digital numbers involved in the embodiments of the present application are only used to distinguish for the convenience of description and are not used to limit the scope of the embodiments of the present application. The size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic.

Claims

1. A conference sharing method, characterized in that: The method comprises: The proxy terminal displays a first display interface according to the received first rich media data sent by the first terminal; The first display interface includes the content to be shared in the conference indicated by the first rich media data; the first terminal interacts with other terminals participating in the conference through the proxy terminal; The proxy terminal obtains the user's editing operation on the first display interface to obtain second rich media data; The proxy terminal combines the first rich media data and the second rich media data and sends them to the first terminal for the first terminal to display a second display interface, where the second display interface includes the content to be shared in the conference indicated by the first rich media data and the second rich media data.

2. The method according to claim 1, characterized in that The editing operation includes one or more of the following: inserting annotations, texts, and images.

3. The method according to claim 1 or 2, characterized in that: The first rich media data includes one or more of the following: desktop sharing sub-data and image sub-data of the first terminal.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: The proxy terminal sends the first rich media data and the second rich media data to a second terminal, and the second terminal participates in the same conference as the first terminal.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The proxy terminal displays a control component corresponding to the second rich media data on the first display interface.

6. The method according to any one of claims 1 to 5, characterized in that Rich media data is data that has been processed for anti-packet loss.

7. The method according to any one of claims 1 to 6, characterized in that The proxy terminal and the first terminal are in the same local area network environment.

8. A conference sharing device, characterized in that: The conference sharing device is deployed on a proxy terminal, and the device includes: A display module, configured to display a first display interface according to first rich media data received and sent by a first terminal; wherein the first display interface includes content to be shared in a conference indicated by the first rich media data; and the first terminal interacts with other terminals participating in the conference through the proxy terminal; An acquisition module, used to acquire the user's editing operation on the first display interface to obtain second rich media data; A sending module is used to merge the first rich media data and the second rich media data and send them to the first terminal, so that the first terminal displays a second display interface, and the second display interface includes the content to be shared in the conference indicated by the first rich media data and the second rich media data.

9. The device according to claim 8, characterized in that The editing operation includes one or more of the following: inserting annotations, texts, and images.

10. The device according to claim 8 or 9, characterized in that The first rich media data includes one or more of the following: desktop sharing sub-data and image sub-data of the first terminal.

11. The device according to any one of claims 8 to 10, characterized in that The sending module is further used for sending the first rich media data and the second rich media data to a second terminal, where the second terminal participates in the same conference as the first terminal.

12. The device according to any one of claims 8 to 11, characterized in that The display module is further used to: display a control component corresponding to the second rich media data on the first display interface.

13. The device according to any one of claims 8 to 12, characterized in that Rich media data is data that has been processed for anti-packet loss.

14. The device according to any one of claims 8 to 13, characterized in that The proxy terminal and the first terminal are in the same local area network environment.

15. A conference sharing method, characterized in that: Applied to a conference sharing system, the conference sharing system includes a first terminal and a proxy terminal; the first terminal interacts with other terminals participating in the conference through the proxy terminal; the method includes: The first terminal sends first rich media data to the proxy terminal; The proxy terminal displays a first display interface according to the first rich media data; wherein the first display interface includes content to be shared in the conference indicated by the first rich media data; The proxy terminal obtains the user's editing operation on the first display interface to obtain second rich media data; The proxy terminal combines the first rich media data and the second rich media data and sends the combined data to the first terminal; The first terminal displays a second display interface, where the second display interface includes the first rich media data and content to be shared in the conference indicated by the second rich media data.

16. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that: The storage medium stores a computer program or instruction, and when the computer program or instruction is executed by a processing device, the method according to any one of claims 1 to 7 is implemented.

18. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed on a processing device, the method according to any one of claims 1 to 7 is implemented.

Images