Method and device for collaborative editing of documents in multimedia conference scenarios

Abstract

The present specification provides a method and device for collaborative document editing in a multimedia conference scenario, wherein the method is applied to a media stream server, wherein multimedia stream channels for the multimedia conference are respectively established between the media stream server and all collaborative editing clients, and each collaborative editing client also establishes a collaborative editing channel with the collaborative editing server; the method comprises: receiving collaborative editing data for a shared document sent by a first collaborative editing client that initiates document sharing through a multimedia stream channel established with the first collaborative editing client; and synchronizing the collaborative editing data to other collaborative editing clients through a multimedia stream channel established with other collaborative editing clients.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of mobile Internet technology, and in particular, to a method and device for collaborative editing of documents in a multimedia conference scenario. Background Art

[0002] With the development of mobile Internet technology, collaborative document editing in multimedia conference scenarios is becoming more and more popular among users. During a multimedia conference, users can share a document during the conference and edit the shared document. All participating users can view the edited content of the shared document in real time.

[0003] In related technologies, in order for participating users to view the edited content of shared documents in real time, it is necessary to establish a transmission channel between each user client and the server based on the RPC (Remote Procedure Call) protocol, so as to achieve synchronization of collaborative editing data of each client based on the channel. The RPC protocol is a connection-oriented reliable transmission protocol based on TCP (Transmission Control Protocol), so it is necessary to maintain a stable connection between each client and the server to achieve synchronization of collaborative editing content. However, with the increase in the number of participating users, fluctuations in the network environment, or in a weak network environment, the stable connection will be disturbed, causing the synchronization of shared documents to be stuck. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a method and device for collaboratively editing a document in a multimedia conference scenario.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of the present specification, a method for collaborative document editing in a multimedia conference scenario is proposed. The method is applied to a media stream server, and multimedia stream channels for the multimedia conference are respectively established between the media stream server and all collaborative editing clients, and each collaborative editing client also establishes a collaborative editing channel with the collaborative editing server; the method includes:

[0007] Receiving collaborative editing data for a shared document sent by the first collaborative editing client through a multimedia stream channel established with the first collaborative editing client that initiates document sharing;

[0008] The collaborative editing data is synchronized to other collaborative editing clients through the multimedia stream channel established with other collaborative editing clients.

[0009] According to a second aspect of one or more embodiments of the present specification, a method for collaborative editing of documents in a multimedia conference scenario is proposed, the method being applied to a collaborative editing client that initiates document sharing, the collaborative editing client that initiates document sharing respectively establishing a multimedia streaming channel and a collaborative editing channel with the outside; the method comprising:

[0010] During the multimedia conference, generating collaborative editing data for a shared document;

[0011] The collaborative editing data is synchronized to other collaborative editing clients participating in the multimedia conference through the multimedia stream channel.

[0012] According to a third aspect of one or more embodiments of the present specification, a device for collaborative document editing in a multimedia conference scenario is proposed, the device is applied to a media stream server, a multimedia stream channel for the multimedia conference is established between the media stream server and all collaborative editing clients, and each collaborative editing client also establishes a collaborative editing channel with the collaborative editing server, the device includes:

[0013] A receiving unit, configured to receive collaborative editing data for a shared document sent by the first collaborative editing client through a multimedia stream channel established with the first collaborative editing client that initiates document sharing;

[0014] The synchronization unit is used to synchronize the collaborative editing data to other collaborative editing clients through the multimedia stream channel established between the collaborative editing clients.

[0015] According to a fourth aspect of one or more embodiments of the present specification, a device for collaborative editing of documents in a multimedia conference scenario is provided, wherein the device is applied to a collaborative editing client that initiates document sharing, wherein the collaborative editing client that initiates document sharing respectively establishes a multimedia streaming channel and a collaborative editing channel with the outside; the device comprises:

[0016] A generating unit, used for generating collaborative editing data for a shared document during the multimedia conference;

[0017] The collaborative editing data synchronization unit is used to synchronize the collaborative editing data to other collaborative editing clients participating in the multimedia conference through the multimedia stream channel.

[0018] According to a fifth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect and / or the second aspect are implemented.

[0019] According to the sixth aspect of one or more embodiments of the present specification, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect and / or the second aspect when executing the program.

[0020] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0021] In the embodiments of the present specification, the collaborative editing data is transmitted by borrowing the existing multimedia streaming channel in the multimedia conference. The multimedia streaming channel is a channel that can transmit data without establishing a connection. It has a high data transmission rate and can effectively avoid the synchronization jamming problem of shared documents caused by the increase in the number of participating users, the fluctuation of the network environment or the weak network environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification, and for ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0023] Figure 1 This is a schematic diagram of an architecture for collaborative editing of documents in a multimedia conference scenario provided by an exemplary embodiment of this specification;

[0024] Figure 2 is a flowchart of a method for collaborative editing of documents in a multimedia conference scenario provided by an exemplary embodiment of this specification;

[0025] Figure 3 is a schematic diagram of a multimedia stream channel and a collaborative editing channel provided by an exemplary embodiment of this specification;

[0026] Figure 4 It is a structural diagram of a data synchronization message provided by an exemplary embodiment of this specification;

[0027] Figure 5 is a schematic diagram of the structure of another data synchronization message provided by an exemplary embodiment of this specification;

[0028] Figure 6 is a flowchart of a method for collaborative editing of documents in a multimedia conference scenario provided by an exemplary embodiment of this specification;

[0029] Figure 7 is a schematic structural diagram of an electronic device provided by an exemplary embodiment of this specification;

[0030] Figure 8 It is a block diagram of a device for collaboratively editing documents in a multimedia conference scenario provided by an exemplary embodiment of this specification;

[0031] Fig. 9 It is a block diagram of a device for collaboratively editing documents in a multimedia conference scenario provided by an exemplary embodiment of this specification. DETAILED DESCRIPTION

[0032] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0033] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0034] With the development of mobile Internet technology, collaborative document editing in multimedia conference scenarios is becoming more and more popular among users. During a multimedia conference, users can share a document while conducting a multimedia conference and can edit the shared document. For example, users can turn pages, scroll paragraphs, input text, etc. of the shared document, and all participating users can view the edited content of the shared document in real time.

[0035] In related technologies, in order for participating users to see the edited content of a shared document in real time, it is necessary to establish a transmission channel between each user client and the server based on the RPC (Remote Procedure Call) protocol, so as to achieve synchronization of collaborative editing data of each client based on the channel. The RPC protocol is a connection-oriented reliable transmission protocol based on TCP (Transmission Control Protocol), so it is necessary to maintain a stable connection between each client and the server to achieve synchronization of collaborative editing content. However, with the increase in the number of participating users, fluctuations in the network environment, or in a weak network environment, this stable connection will be disturbed, causing synchronization of shared documents to be stalled.

[0036] In view of this, the present specification provides a method for collaborative document editing in a multimedia conference scenario, by using the existing multimedia stream channel in the multimedia conference to transmit synchronous collaborative editing data, and the multimedia stream channel is a channel that can transmit data without establishing a connection and the multimedia stream channel has a higher data transmission rate, that is, the multimedia stream channel is a message-oriented, lightweight transmission channel. It is precisely because the multimedia stream channel is a message-oriented transmission channel, not a connection-oriented transmission channel, that it does not need to consider whether the data packet reaches the final destination. Therefore, in the case of a poor network environment, the transmission speed of the multimedia stream channel can be several times higher than the RPC channel described above. Therefore, the method for collaborative document editing described in this specification can effectively avoid the problem of shared document jamming caused by the increase in the number of participating users, fluctuations in the network environment, or in a weak network environment.

[0037] Figure 1 FIG. 1 is a schematic diagram of an architecture for collaborative editing of documents in a multimedia conference scenario provided by an exemplary embodiment of this specification. Figure 1 As shown, the system architecture may include a media stream server 11, a collaborative editing server 12, a network 13, and several terminal electronic devices, such as a PC 14 and a mobile phone 15.

[0038] The media stream server 11 can be deployed on a physical server including an independent host, or the server can be a virtual server (such as a cloud server) carried by a host cluster. During the operation of the physical server or virtual server, the media stream server 11 can be configured with a method for collaborative document editing in a multimedia conference scenario described in this specification, and receive the collaborative editing data sent by the collaborative editing client that initiates document sharing through the multimedia stream channel established between the media stream server 11 and each client, and synchronize it to other collaborative editing clients. The transmission speed of the multimedia stream channel is much higher than the transmission channel established based on the RPC protocol as described above. Therefore, based on the multimedia stream channel that does not require connection, it can effectively avoid the synchronization jamming problem of shared documents caused by the increase in the number of participating users, the fluctuation of the network environment or in a weak network environment.

[0039] The network 13 for interaction between the PC 14, the mobile phone 15, the media stream server 11 and the collaborative editing server 12 may include various types of wired or wireless networks. In one embodiment, the network may include a public switched telephone network (PSTN) and the Internet.

[0040] The terminal electronic device can be a PC 14, a mobile phone 15, and a wearable device (such as a smart watch and smart glasses), and this specification does not limit this. The terminal electronic device can run a client configured with a method for collaborative document editing in a multimedia conference scenario described in this specification, and the collaborative editing client that initiates document sharing can send corresponding collaborative editing data through a multimedia stream channel established with the media stream server 11. Correspondingly, other collaborative editing clients can receive the above collaborative editing data through the multimedia stream channel established with the media stream server 11, thereby realizing the synchronization operation of shared documents of each collaborative editing client.

[0041] It should be noted that although Figure 1 In the illustrated embodiment, in the multimedia conference scenario, there is a media stream server 11, but in some embodiments, because the multimedia stream channel may not rely on the establishment between the media stream server 11 and the collaborative editing client, the media stream server 11 may not be included at this time. In other words, in some multimedia conference scenarios, a multimedia stream channel may be established between various collaborative editing clients, and this specification does not limit this. However, even if the media stream server 11 is not included, because of the existence of the multimedia stream channel that does not require connection, the above-mentioned multimedia stream channel can still be used to realize the transmission and synchronization of collaborative editing data between various collaborative editing clients, and the problem of shared document freezes caused by the increase in the number of participating users, fluctuations in the network environment, or in a weak network environment can still be effectively avoided.

[0042] Based on the above system architecture, a method for collaborative document editing in a multimedia conference scenario described in this specification can be configured by utilizing the existing multimedia stream channels in the multimedia conference to transmit synchronous collaborative editing data. The multimedia stream channel is a channel that can transmit data at a higher transmission speed without establishing a connection. In an environment with poor network conditions, the transmission speed of the multimedia stream channel can be several times higher than that of the RPC channel described above, which can effectively avoid the problem of shared document freezes caused by an increase in the number of participating users, fluctuations in the network environment, or in a weak network environment.

[0043] Figure 2 The present invention is a flowchart of a method for collaborative document editing in a multimedia conference scenario provided by an exemplary embodiment, which is applied to a media stream server, wherein multimedia stream channels for the multimedia conference are respectively established between the media stream server and all collaborative editing clients, and each collaborative editing client also establishes a collaborative editing channel with the collaborative editing server; the method comprises the following steps:

[0044] Step 201: Receive collaborative editing data for a shared document sent by a first collaborative editing client that initiates document sharing through a multimedia stream channel established with the first collaborative editing client.

[0045] The collaborative editing data may be used to characterize the specific operations performed by the user on the shared document. For example, the operation may be scrolling the document page, switching to the next page, paragraph selection, text symbol highlighting, or text input, etc. This specification does not limit this.

[0046] In one embodiment, a multimedia stream channel for multimedia conferencing is established between the media stream server and all collaborative editing clients, and the multimedia stream channel can be constructed based on UDP (User Datagram Protocol). The UDP protocol is a transaction-oriented, connectionless, simple, lightweight protocol that can complete the sending and receiving of messages at a very high speed. In addition, in a poor network environment, the transmission speed of the multimedia stream channel can be several times higher than the RPC channel described above. Figure 3 As shown, Figure 3 This is a schematic diagram of a multimedia stream channel and a collaborative editing channel provided by an exemplary embodiment. Among them, the media stream server 11 respectively establishes a multimedia stream channel with the PC 14 configured with the collaborative editing client and the mobile phone 15 configured with the collaborative editing client, and the multimedia stream channel can be established based on the UDP protocol. Each collaborative editing client also establishes a collaborative editing channel with the collaborative editing server, and the collaborative editing channel can be established based on the RPC (Remote Procedure Call) protocol. The RPC protocol is a connection-oriented reliable transmission protocol based on the TCP protocol, which can ensure that the communicating parties can reliably receive or send related data. Figure 3 As shown, the collaborative editing server 12 establishes collaborative editing channels with the PC 14 equipped with the collaborative editing client and the mobile phone 15 equipped with the collaborative editing client, respectively. The collaborative editing channel can be established based on the RPC protocol. By converting the collaborative editing data originally transmitted through the collaborative editing channel to be transmitted through the multimedia stream channel, the speed of message sending or receiving can be effectively improved. Even if there is a large increase in the number of users participating in the multimedia conference, the network environment fluctuates, or in a weak network environment, the speed of message transmission can be guaranteed to avoid synchronization jams of shared documents.

[0047] In one embodiment, the collaborative editing channel can be used to transmit non-collaborative editing type data generated for a shared document. For example, when the current user needs to perform a sharing operation on the shared document, the shared document data generated for the shared document at this time can be transmitted through the collaborative editing channel. For another example, when the current user needs to set permissions for the document, the permission setting data generated for the shared document at this time can be transmitted through the collaborative editing channel. In other words, the method for collaborative editing of documents in a multimedia conference scenario described in this specification can still retain the collaborative editing channel, but the data transmitted by the channel is no longer collaborative editing data, but non-collaborative editing type data, and the collaborative editing data is transmitted through the multimedia stream channel. By transmitting non-collaborative editing data through the collaborative editing channel, the collaborative editing server can process non-collaborative editing data. On the one hand, the reliability of non-collaborative editing data transmission can be guaranteed, ensuring that the user's non-editing operations can be effectively executed; on the other hand, retaining the collaborative editing server can reduce the work pressure of the media stream server, reduce the load of the media stream server, and enable the document sharing system in the multimedia conference scenario to have reasonable job distribution.

[0048] Because the collaborative editing data described in this specification uses an existing multimedia stream channel, there are inevitably two types of data in the channel, one is the collaborative editing data, and the other is the media stream data. In order to enable the media stream server to distinguish the above two types of data, this specification provides three effective methods.

[0049] In one embodiment, the media stream server can receive a data synchronization message sent by the first collaborative editing client according to a preset sending cycle, and the data synchronization message includes a collaborative editing data synchronization message sent within the first time period of the above-mentioned preset sending cycle and a media stream data synchronization message sent within the second time period of the above-mentioned preset sending cycle. After the media stream server receives the collaborative editing data synchronization message, the collaborative editing data can be parsed from the collaborative editing data synchronization message. For example, the preset sending cycle can be 1 minute, and the first time period T1 can be the first 30 seconds within the 1 minute, and correspondingly, the second time period T2 can be the last 30 seconds within the 1 minute, that is, the first collaborative editing client can send a collaborative editing data synchronization message within the first 30 seconds, and send a media stream data synchronization message within the last 30 seconds. Therefore, the media stream server can distinguish between collaborative editing data and media stream data according to different time periods corresponding to the preset sending cycle, which can effectively avoid fault errors or system crashes caused by data confusion.

[0050] In one embodiment, the media stream server can receive a data synchronization message sent by the first collaborative editing client, wherein the data synchronization message includes a marker position and a data payload, wherein the data payload includes collaborative editing data and media stream data arranged in sequence according to a preset order. Therefore, the media stream server can divide the data payload according to the marker position to obtain collaborative editing data. Figure 4 As shown, Figure 4 1 is a structural diagram of a data synchronization message provided by an exemplary embodiment. The message structure can be divided into a message header and data. The first coordinated editing client can add a corresponding mark position in the message header when sending the data synchronization message. Further, the data part of the data synchronization message sent by the first collaborative editing client arranges the collaborative editing data and the media stream data in sequence according to a preset order, so that the media stream server can effectively distinguish the collaborative editing data from the media stream data according to the mark position. It should be noted that although in Figure 4 In the embodiment shown, the collaborative editing data is placed in front and the media stream data is placed in the back, but in some embodiments, the media stream data can also be placed before the collaborative editing data, and this specification does not limit this. By marking the position, the media stream server can efficiently and accurately distinguish the collaborative editing data from the media stream data, which can effectively avoid faults or system crashes caused by data confusion.

[0051] In one embodiment, the media stream server may receive a data synchronization message sent by the first collaborative editing client, the data synchronization message including a data payload with a type identifier, the type identifier being used to distinguish the collaborative editing data and the media stream data included in the data payload; thus, the media stream server may obtain the collaborative editing data from the data payload according to the type identifier. Figure 5 As shown, Figure 5 This is a schematic diagram of another structure of a data synchronization message provided by an exemplary embodiment. The first collaborative editing client can add a corresponding type identifier in the data payload of the sent data synchronization message, for example, add type identifier A to the collaborative editing data and add type identifier B to the media stream data, so that the media stream server can obtain the collaborative editing data according to the type identifier. It should be noted that although Figure 5In the illustrated embodiment, both the media stream data and the collaboratively edited data carry type identifiers, but it is understandable that only one type of data may be provided with a type identifier to achieve effective distinction between the media stream data and the collaboratively edited data. For example, only the collaboratively edited data may be provided with a type identifier, while the media stream data may not be provided with a type identifier. In this case, the media stream server may still be able to distinguish between the media stream data and the collaboratively edited data, and this specification does not impose any restrictions on this. Through the marking effect of the type identifier, the media stream server may efficiently and accurately distinguish between the collaboratively edited data and the media stream data, and may effectively avoid fault reports or system crashes caused by data confusion.

[0052] When data is transmitted based on a multimedia stream channel, the network link layer often has a limit on the length of the data message, which is also called MTU (Maximum Transmission Unit). Generally, the MTU set for Ethernet is 1500 bytes. Therefore, in order to comply with the MTU, in one embodiment, the media stream server can receive multiple data synchronization messages containing collaborative editing data blocks respectively, and the messages contain sequence numbers; the media stream server obtains the collaborative editing data blocks respectively, and reorganizes the collaborative editing data blocks according to the sequence numbers contained in the messages where the collaborative editing data blocks are located, so as to obtain the collaborative editing data. In other words, in order to avoid the failure to successfully transmit the message due to the limitation of MTU, the collaborative editing data in a message can be split into corresponding collaborative editing data blocks and assembled into multiple messages, so as to meet the requirements of the maximum transmission unit, thereby achieving successful sending and receiving of the message. Furthermore, the corresponding sequence number can be added to the message, so that the media stream server can reorganize the collaborative editing data blocks in order based on the sequence number, so as to obtain the collaborative editing data. By fragmenting and reassembling messages, the sending and receiving of messages can meet the MTU requirements and improve the success rate of message transmission.

[0053] Step 202: Synchronize the collaborative editing data to other collaborative editing clients through the multimedia stream channel established with other collaborative editing clients.

[0054] After the media stream server receives the collaborative editing data for the shared document sent by the first collaborative editing client, the media stream server can synchronize the received collaborative editing data to other collaborative editing clients, so that the other collaborative editing clients can display the same content as the first collaborative editing client. At this time, the media stream server can synchronize the collaborative editing data to other clients in two ways. First, the media stream server can parse the collaborative editing data from the data synchronization message containing the collaborative editing data sent by the first collaborative editing client, and synchronize the collaborative editing data to other collaborative editing clients; or, second, the media stream server can synchronize the data synchronization message containing the collaborative editing data sent by the first collaborative editing client to other collaborative editing clients, so that other collaborative editing clients can parse the message to obtain the collaborative editing data. In other words, the media stream server can directly send the parsed collaborative editing data to other collaborative editing clients through the multimedia stream channel; the media stream server can also send the data synchronization message sent by the first collaborative editing client to other collaborative editing clients, and let other collaborative editing clients themselves parse the message and obtain the collaborative editing data. However, regardless of method one or method two, the collaborative editing data can be synchronized to other collaborative editing clients, so this manual does not limit this.

[0055] In one embodiment, in order to improve the success rate of collaborative editing data synchronization, the first collaborative editing client can send data synchronization messages redundantly, that is, the first collaborative editing client can repeatedly send the same data synchronization message according to a preset number. For example, the first collaborative editing client can send the same data synchronization message to the media stream server 5 times through the multimedia stream channel. This ensures that the media stream server can successfully receive the corresponding message, thereby improving the success rate of receiving the data synchronization message.

[0056] In order to further improve the success rate of collaborative editing data synchronization, this specification may also introduce an additional confirmation mechanism based on the UDP protocol. Specifically, when the media stream server does not receive the reception confirmation returned by other collaborative editing clients for the collaborative editing data, it resynchronizes the collaborative editing data to other collaborative editing clients; or, when receiving a retransmission request for the specified collaborative editing data sent by other collaborative editing clients, it resends the specified collaborative editing data to other collaborative editing clients that sent the retransmission request for the specified collaborative editing data. In other words, after receiving the collaborative editing data, other collaborative editing clients may send a reception confirmation receipt to the media stream server, so that the media stream server confirms that other collaborative editing clients have received the collaborative editing data. Assuming that the media stream server does not receive the reception confirmation receipt, it may resend the collaborative editing data to other collaborative editing data clients. The collaborative editing client may also send a retransmission request to the media stream server for the specified collaborative editing data, so that the media stream server resends the specified collaborative editing data. It can be seen that the media stream server can take corresponding response measures according to the reception confirmation receipt or retransmission request sent by the client, which not only ensures the message transmission rate but also avoids the failure of shared document synchronization caused by message loss, thereby improving the success rate of real-time synchronization of shared documents.

[0057] The media stream server described in this specification can be deployed in a media stream service cluster, which includes a first type of media stream server for processing multimedia streams and a second type of media stream server for processing collaborative editing data; at this time, the collaborative editing data can be synchronized to other collaborative editing clients based on the second type of media stream server; wherein the first type of media stream is used to forward the multimedia data generated by the multimedia conference between collaborative editing clients. By deploying the media stream server in a media stream service cluster, the response speed of the server can be effectively improved, and the cluster can specifically set different servers to process different data, thereby further improving the speed of data processing.

[0058] It can be seen from the above embodiments that the method for collaborative document editing in a multimedia conference scenario described in this specification transmits synchronous collaborative editing data by borrowing the existing multimedia stream channel in the multimedia conference. The multimedia stream channel is a channel that can transmit data without establishing a connection and has a high data transmission rate. In an environment with poor network environment, the transmission speed of the multimedia stream channel can be several times higher than the RPC channel described above. Therefore, it can effectively avoid the problem of shared document jamming caused by the increase in the number of participating users, fluctuations in the network environment or in a weak network environment.

[0059] Figure 6The present invention is a flowchart of a method for collaborative document editing in a multimedia conference scenario provided by an exemplary embodiment. The method is applied to a collaborative editing client that initiates document sharing. The collaborative editing client that initiates document sharing establishes a multimedia streaming channel and a collaborative editing channel with the outside respectively. The method includes:

[0060] Step 601: During the multimedia conference, collaborative editing data for a shared document is generated.

[0061] In one embodiment, the collaborative editing data can be used to characterize the specific operations of the user in editing the shared document. For example, the operation can be scrolling the page of the document, going to the next page, paragraph selection, text symbol highlighting or text input, etc. This specification does not limit this.

[0062] Step 602: Synchronize the collaborative editing data to other collaborative editing clients participating in the multimedia conference through the multimedia stream channel.

[0063] In one embodiment, the collaborative editing client that initiates document sharing can send data synchronization messages through the multimedia stream channel according to a preset sending cycle, so that other collaborative editing clients of the multimedia conference synchronize the collaborative editing data, and the data synchronization message includes a collaborative editing data synchronization message sent within a first time period of the preset sending cycle and a media stream data synchronization message sent within a second time period of the preset sending cycle. For example, the preset sending cycle can be 1 minute, and the first time period T1 can be the first 30 seconds within the 1 minute, and correspondingly, the second time period T2 can be the last 30 seconds within the 1 minute, that is, the collaborative editing client that initiates document sharing can send collaborative editing data synchronization messages within the first 30 seconds and send media stream data synchronization messages within the last 30 seconds. Therefore, the media stream server can distinguish between collaborative editing data and media stream data according to different time periods corresponding to the preset sending cycle, which can effectively avoid fault errors or system crashes caused by data confusion.

[0064] In one embodiment, the collaborative editing client that initiates document sharing can send a data synchronization message through a multimedia stream channel to synchronize the collaborative editing data with other collaborative editing clients of the multimedia conference, and the data synchronization message includes a mark position and a data payload, and the data payload includes collaborative editing data and media stream data arranged in sequence according to a preset order. The collaborative editing client that initiates document sharing can add a corresponding mark position in the message header of the data synchronization message so that the media stream server can efficiently and accurately distinguish the collaborative editing data and media stream data based on the mark position, which can effectively avoid fault errors or system crashes caused by data confusion.

[0065] In one embodiment, the collaborative editing client that initiates document sharing can send a data synchronization message through the multimedia stream channel to synchronize the collaborative editing data with other collaborative editing clients of the multimedia conference, and the data synchronization message includes a data payload with a type identifier, and the type identifier is used to distinguish the collaborative editing data and media stream data included in the data payload. The collaborative editing client that initiates document sharing can add a corresponding type identifier in the data payload so that the media stream server can efficiently and accurately distinguish the collaborative editing data and media stream data based on the type identifier, which can effectively avoid fault errors or system crashes caused by data confusion.

[0066] As mentioned above, when data is transmitted based on a multimedia stream channel, the network link layer often has a limit on the length of the data message, which is also called MTU (Maximum Transmission Unit). Generally, the MTU set for Ethernet is 1500 bytes. Therefore, in order to meet the MTU requirements, the collaborative editing client that initiates document sharing can split the collaborative editing data into multiple collaborative editing data blocks; and the collaborative editing client that initiates document sharing can add multiple collaborative editing data blocks to multiple data synchronization messages respectively, and each data synchronization message contains a sequence number, which is used to characterize the order of the corresponding collaborative editing data block in the collaborative editing data. Finally, the collaborative editing client that initiates document sharing can send multiple data synchronization messages containing collaborative editing data blocks through the multimedia stream channel. By fragmenting and reassembling the message, the sending and receiving of the message can meet the MTU requirements, improve the success rate of message transmission, and avoid the failure of sending or receiving messages due to non-compliance with the MTU requirements.

[0067] It should be noted that the establishment of multimedia circulation is not limited to the media stream server and the client that initiates document sharing, that is, the multimedia stream channel can be established between the collaborative editing client that initiates document sharing and the media stream server; wherein, the media stream server also establishes multimedia stream channels with other collaborative editing clients participating in the multimedia conference; or, the multimedia stream channel is established between the collaborative editing client that initiates document sharing and other collaborative editing clients participating in the multimedia conference, because in the multimedia conference scenario, the establishment of the multimedia stream channel can include the media stream server or not include the media stream server, but even if it does not include the media stream server, because of the existence of the multimedia stream channel that does not require connection, the above-mentioned multimedia stream channel can still be used to realize the transmission and synchronization of collaborative editing data between various collaborative editing clients, and the problem of shared document jamming caused by the increase in the number of participating users, the fluctuation of the network environment or the weak network environment can still be effectively avoided. In short, no matter how the multimedia stream channel is established, the channel can be used to transmit collaborative editing data. Therefore, the method of document collaborative editing in a multimedia conference scenario described in this specification has a wide range of applicable scenarios.

[0068] Corresponding to the above method embodiments, this specification also provides device embodiments.

[0069] Figure 7 This is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Please refer to Figure 7 At the hardware level, the electronic device includes a processor 701, a network interface 702, a memory 703, a non-volatile memory 704, and an internal bus 705, and may also include hardware required for other functions. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it. Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0070] Figure 8 The present invention is a block diagram of a device for collaborative document editing in a multimedia conference scenario provided by an exemplary embodiment. The device is applied to a media stream server. A multimedia stream channel for the multimedia conference is established between the media stream server and all collaborative editing clients, and each collaborative editing client also establishes a collaborative editing channel with the collaborative editing server. The device includes:

[0071] The receiving unit 801 is configured to receive collaborative editing data for a shared document sent by a first collaborative editing client that initiates document sharing through a multimedia stream channel established with the first collaborative editing client.

[0072] The synchronization unit 802 is used to synchronize the collaborative editing data to other collaborative editing clients through the multimedia stream channel established between the collaborative editing clients.

[0073] Optionally, the receiving unit 801 is specifically configured to:

[0074] Receive the data synchronization message sent by the first collaborative editing client according to a preset sending cycle, the data synchronization message including the collaborative editing data synchronization message sent within a first time period of the preset sending cycle and the media stream data synchronization message sent within a second time period of the preset sending cycle; parse and obtain the collaborative editing data from the collaborative editing data synchronization message.

[0075] Optionally, the receiving unit 801 is specifically configured to:

[0076] Receive a data synchronization message sent by the first collaborative editing client, the data synchronization message including a mark position and a data payload, the data payload including collaborative editing data and media stream data arranged in sequence according to a preset order; divide the data payload according to the mark position to obtain the collaborative editing data.

[0077] Optionally, the receiving unit 801 is specifically configured to:

[0078] Receive a data synchronization message sent by the first collaborative editing client, the data synchronization message including a data payload with a type identifier, the type identifier being used to distinguish collaborative editing data and media stream data included in the data payload; and obtain the collaborative editing data from the data payload according to the type identifier.

[0079] Optionally, the receiving unit 801 is specifically configured to:

[0080] Respectively receive a plurality of data synchronization messages containing collaboratively edited data blocks, each message containing a sequence number; respectively obtain the collaboratively edited data blocks, and reorganize the collaboratively edited data blocks according to the sequence numbers contained in the messages in which the collaboratively edited data blocks are located, to obtain the collaboratively edited data.

[0081] Optionally, the synchronization unit 802 is specifically configured to:

[0082] Parse the collaborative editing data from a data synchronization message containing the collaborative editing data sent by the first collaborative editing client, and synchronize the collaborative editing data to other collaborative editing clients; or synchronize the data synchronization message containing the collaborative editing data sent by the first collaborative editing client to other collaborative editing clients, so that the other collaborative editing clients can parse the message to obtain the collaborative editing data.

[0083] Optionally, the device further comprises:

[0084] The resynchronization unit 803 is used to resynchronize the collaborative editing data to the other collaborative editing clients when no reception confirmation is received from the other collaborative editing clients for the collaborative editing data; or, when a retransmission request for the specified collaborative editing data is received from the other collaborative editing clients, retransmit the specified collaborative editing data to the other collaborative editing clients that sent the retransmission request for the specified collaborative editing data.

[0085] Optionally, the collaborative editing channel is used to transmit non-collaborative editing type data generated for a shared document.

[0086] Optionally, the multimedia streaming channel is constructed based on the UDP protocol, and the collaborative editing channel is constructed based on the RPC protocol.

[0087] Optionally, the media stream server is deployed in a media stream service cluster, which includes a first type of media stream server for processing multimedia streams and a second type of media stream server for processing collaborative editing data.

[0088] Optionally, the synchronization unit 802 is specifically configured to:

[0089] The second type of media stream server synchronizes the collaborative editing data to other collaborative editing clients;

[0090] The first type of media stream is used to forward multimedia data generated by the multimedia conference between collaborative editing clients.

[0091] Fig. 9 The present invention is a block diagram of a device for collaborative document editing in a multimedia conference scenario provided by an exemplary embodiment. The device is applied to a collaborative editing client that initiates document sharing. The collaborative editing client that initiates document sharing establishes a multimedia streaming channel and a collaborative editing channel with the outside respectively. The device includes:

[0092] The generating unit 901 is used to generate collaborative editing data for a shared document during the multimedia conference.

[0093] The collaborative editing data synchronization unit 902 is used to synchronize the collaborative editing data to other collaborative editing clients participating in the multimedia conference through the multimedia stream channel.

[0094] Optionally, the collaborative editing data synchronization unit 902 is specifically used for:

[0095] Data synchronization messages are sent through the multimedia stream channel according to a preset sending cycle so that other collaborative editing clients of the multimedia conference synchronize the collaborative editing data. The data synchronization messages include collaborative editing data synchronization messages sent within the first time period of the preset sending cycle and media stream data synchronization messages sent within the second time period of the preset sending cycle.

[0096] Optionally, the collaborative editing data synchronization unit 902 is specifically used for:

[0097] A data synchronization message is sent through the multimedia stream channel to synchronize the collaborative editing data with other collaborative editing clients of the multimedia conference. The data synchronization message includes a mark position and a data payload. The data payload includes collaborative editing data and media stream data arranged in sequence according to a preset order.

[0098] Optionally, the collaborative editing data synchronization unit 902 is specifically used for:

[0099] A data synchronization message is sent through the multimedia stream channel to synchronize the collaborative editing data with other collaborative editing clients of the multimedia conference. The data synchronization message includes a data payload with a type identifier, and the type identifier is used to distinguish between the collaborative editing data and media stream data included in the data payload.

[0100] Optionally, the collaborative editing data synchronization unit 902 is specifically used for:

[0101] The collaboratively edited data is split into multiple collaboratively edited data blocks; the multiple collaboratively edited data blocks are respectively added to multiple data synchronization messages, and each data synchronization message contains a sequence number, which is used to characterize the order of the corresponding collaboratively edited data block in the collaboratively edited data; and the multiple data synchronization messages containing the collaboratively edited data blocks are sent through the multimedia stream channel.

[0102] Optionally, the multimedia stream channel is established between the collaborative editing client that initiates document sharing and the media stream server; wherein, the media stream server also establishes multimedia stream channels with other collaborative editing clients participating in the multimedia conference; or, the multimedia stream channel is established between the collaborative editing client that initiates document sharing and other collaborative editing clients participating in the multimedia conference.

[0103] Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the relevant method, and will not be elaborated here.

[0104] It can be seen from the technical solutions provided in the above specification that the method for elastically adjusting the parallelism of task processing described in the specification can enable the task scheduler to flexibly adjust the parallelism of task processing in the task executor based on the delay information fed back by the task executor. When the delay is greater than the upper limit of the normal delay threshold range, the parallelism of task processing is quickly increased so that the task executor can speed up data processing and ensure that the delay of task processing by the task executor is within the control time limit; at the same time, when the delay is less than the lower limit of the normal delay threshold range, the parallelism of task processing is slowly reduced. On the premise of ensuring the delay, the system resources occupied by the task executor are released to ensure the reasonable allocation of resources of the real-time computing system. In other words, the elastic adjustment method of parallelism disclosed in this specification ensures the timeliness of data processing while also ensuring the rationality of resource allocation of the real-time computing system.

[0105] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0106] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0107] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0108] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0109] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0110] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A method for collaborative document editing in a multimedia conference scenario, characterized in that: Applied to a media stream server, a multimedia stream channel for the multimedia conference is established between the media stream server and all collaborative editing clients, and each collaborative editing client also establishes a collaborative editing channel with the collaborative editing server; the method includes: Receiving collaborative editing data for a shared document sent by the first collaborative editing client through a multimedia stream channel established with the first collaborative editing client that initiates document sharing; The collaborative editing data is synchronized to other collaborative editing clients through the multimedia stream channel established with other collaborative editing clients, so as to realize the synchronization operation of the shared documents of each collaborative editing client.

2. The method according to claim 1, characterized in that The receiving the collaborative editing data sent by the first collaborative editing client includes: Receiving a data synchronization message sent by the first collaborative editing client according to a preset sending cycle, the data synchronization message including a collaborative editing data synchronization message sent within a first time period of the preset sending cycle and a media stream data synchronization message sent within a second time period of the preset sending cycle; The collaborative editing data is parsed and acquired from the collaborative editing data synchronization message.

3. The method according to claim 1, characterized in that The receiving the collaborative editing data sent by the first collaborative editing client includes: Receiving a data synchronization message sent by the first collaborative editing client, wherein the data synchronization message includes a marker position and a data payload, and the data payload includes collaborative editing data and media stream data arranged in sequence according to a preset order; The data payload is divided according to the marking positions to obtain the collaborative editing data.

4. The method according to claim 1, characterized in that: The receiving the collaborative editing data sent by the first collaborative editing client includes: receiving a data synchronization message sent by the first collaborative editing client, wherein the data synchronization message includes a data payload with a type identifier, wherein the type identifier is used to distinguish collaborative editing data and media stream data included in the data payload; The collaborative editing data is obtained from the data payload according to the type identifier.

5. The method according to claim 1, characterized in that The receiving the collaborative editing data sent by the first collaborative editing client includes: Receiving a plurality of data synchronization messages containing collaboratively edited data blocks respectively, wherein the messages contain sequence numbers; The collaboratively edited data blocks are acquired respectively, and the collaboratively edited data blocks are reorganized according to the sequence numbers contained in the messages in which the collaboratively edited data blocks are located respectively, so as to obtain the collaboratively edited data.

6. The method according to claim 1, characterized in that The step of synchronizing the collaborative editing data to other collaborative editing clients includes: parsing the collaborative editing data from a data synchronization message containing the collaborative editing data sent by the first collaborative editing client, and synchronizing the collaborative editing data to other collaborative editing clients; or The data synchronization message containing the collaborative editing data sent by the first collaborative editing client is synchronized to other collaborative editing clients, so that the other collaborative editing clients parse the message to obtain the collaborative editing data.

7. The method according to claim 1, characterized in that The method further comprises: If no receipt confirmation is received from the other collaborative editing client regarding the collaborative editing data, resynchronizing the collaborative editing data to the other collaborative editing client; or When receiving a resending request for the designated collaborative editing data sent by the other collaborative editing clients, the designated collaborative editing data is resent to the other collaborative editing clients that sent the resending request for the designated collaborative editing data.

8. The method according to claim 1, characterized in that The collaborative editing channel is used to transmit non-collaborative editing type data generated for a shared document.

9. The method according to claim 1, characterized in that: The multimedia stream channel is constructed based on the UDP protocol, and the collaborative editing channel is constructed based on the RPC protocol.

10. The method according to claim 1, characterized in that The media stream server is deployed in a media stream service cluster, which includes a first type of media stream server for processing multimedia streams and a second type of media stream server for processing collaborative editing data; The step of synchronizing the collaborative editing data to other collaborative editing clients includes: The second type of media stream server synchronizes the collaborative editing data to other collaborative editing clients; The first type of media stream is used to forward multimedia data generated by the multimedia conference between collaborative editing clients.

11. A method for collaborative document editing in a multimedia conference scenario, characterized in that: The method is applied to a collaborative editing client that initiates document sharing, wherein the collaborative editing client that initiates document sharing respectively establishes a multimedia stream channel and a collaborative editing channel with the outside; the method comprises: During the multimedia conference, generating collaborative editing data for a shared document; The collaborative editing data is synchronized to other collaborative editing clients participating in the multimedia conference through the multimedia stream channel, so as to realize the synchronization operation of the shared documents of each collaborative editing client.

12. The method according to claim 11, characterized in that The step of synchronizing the collaborative editing data to other collaborative editing clients participating in the multimedia conference through the multimedia stream channel includes: Data synchronization messages are sent through the multimedia stream channel according to a preset sending cycle so that other collaborative editing clients of the multimedia conference synchronize the collaborative editing data. The data synchronization messages include collaborative editing data synchronization messages sent within the first time period of the preset sending cycle and media stream data synchronization messages sent within the second time period of the preset sending cycle.

13. The method according to claim 11, characterized in that The step of synchronizing the collaborative editing data to other collaborative editing clients participating in the multimedia conference through the multimedia stream channel includes: A data synchronization message is sent through the multimedia stream channel to synchronize the collaborative editing data with other collaborative editing clients of the multimedia conference. The data synchronization message includes a mark position and a data payload. The data payload includes collaborative editing data and media stream data arranged in sequence according to a preset order.

14. The method according to claim 11, characterized in that The step of synchronizing the collaborative editing data to other collaborative editing clients participating in the multimedia conference through the multimedia stream channel includes: A data synchronization message is sent through the multimedia stream channel to synchronize the collaborative editing data with other collaborative editing clients of the multimedia conference. The data synchronization message includes a data payload with a type identifier, and the type identifier is used to distinguish between the collaborative editing data and media stream data included in the data payload.

15. The method according to claim 11, characterized in that The step of synchronizing the collaborative editing data to other collaborative editing clients participating in the multimedia conference through the multimedia stream channel includes: Splitting the collaborative editing data into a plurality of collaborative editing data blocks; Adding the multiple collaboratively edited data blocks to multiple data synchronization messages respectively, and each data synchronization message includes a sequence number, which is used to represent the order of the corresponding collaboratively edited data block in the collaboratively edited data; The multiple data synchronization messages containing the collaboratively edited data blocks are sent through the multimedia stream channel.

16. The method according to claim 11, characterized in that The multimedia stream channel is established between the collaborative editing client that initiates the document sharing and the media stream server; wherein the media stream server also establishes multimedia stream channels with other collaborative editing clients participating in the multimedia conference; or, The multimedia stream channel is established between the collaborative editing client that initiates the document sharing and other collaborative editing clients that participate in the multimedia conference.

17. A device for collaborative document editing in a multimedia conference scenario, characterized in that: Applied to a media stream server, a multimedia stream channel for the multimedia conference is established between the media stream server and all collaborative editing clients, and each collaborative editing client also establishes a collaborative editing channel with the collaborative editing server, the device includes: A receiving unit, configured to receive collaborative editing data for a shared document sent by the first collaborative editing client through a multimedia stream channel established with the first collaborative editing client that initiates document sharing; The synchronization unit is used to synchronize the collaborative editing data to other collaborative editing clients through the multimedia stream channel established with other collaborative editing clients, so as to realize the synchronization operation of the shared documents of each collaborative editing client.

18. A device for collaborative document editing in a multimedia conference scenario, characterized in that: The device is applied to a collaborative editing client that initiates document sharing, wherein the collaborative editing client that initiates document sharing respectively establishes a multimedia stream channel and a collaborative editing channel with the outside; the device comprises: A generating unit, used for generating collaborative editing data for a shared document during the multimedia conference; The collaborative editing data synchronization unit is used to synchronize the collaborative editing data to other collaborative editing clients participating in the multimedia conference through the multimedia stream channel, so as to realize the synchronization operation of the shared documents of each collaborative editing client.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

20. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 16 are implemented.

Images