Media synchronization method and system based on data collaboration

Abstract

The invention relates to a media synchronization method and system based on data collaboration, and belongs to the field of video communication, and the method comprises the following steps: an enterprise user initiates an audio call request to a calling platform and forwards the audio call request to a called mobile phone; the called mobile phone receives the request and establishes an audio communication channel; the enterprise user triggers a data collaboration instruction and forwards the data collaboration instruction to the calling platform through the data collaboration platform; the calling platform initiates a one-way video call request to the called mobile phone according to the instruction; after receiving the request, the called mobile phone establishes a data channel with the data collaboration platform and returns a response to the calling platform; the calling platform forwards the response information to the enterprise user through the data collaboration platform; the enterprise user triggers a data pushing instruction and forwards the data pushing instruction to the calling platform through the data collaboration platform; the calling platform pushes a video stream and generates a video index notification according to the instruction and the bandwidth of the called mobile phone; the call platform forwards the notification to the enterprise user through the data collaboration platform; and the enterprise user synchronously plays the same video file according to the notification.

Description

Technical Field

[0001] The present invention belongs to the field of video communication, and in particular relates to a media synchronization method and system based on data collaboration. Background Art

[0002] With the development of modern communication technology and the complexity of application scenarios, especially in data collaboration scenarios, the interaction between enterprises and customers is increasingly dependent on real-time audio and video synchronization. In order to improve user experience and ensure smooth communication between enterprise customer service and called mobile phone users, the call platform needs to have efficient data collaboration and media synchronization capabilities.

[0003] In the prior art, the commonly used method is to play audio and video data to the enterprise user and the called mobile phone at the same time through the calling platform. This method requires video negotiation between the caller (enterprise user) and the called (mobile phone user), and between the calling platform and the called mobile phone to achieve playback synchronization.

[0004] However, playing audio and video data simultaneously will significantly increase the bandwidth usage of the call platform and greatly increase the signaling complexity, thus increasing the burden on the system.

[0005] In the above technical solution, the method of playing audio and video data simultaneously will cause a significant increase in the bandwidth occupancy of the call platform, and the signaling complexity will also be greatly increased, increasing the burden on the system. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the purpose of the invention is to provide a media synchronization method and system based on data collaboration, wherein the data collaboration instruction can be actively triggered by the enterprise user, and the data collaboration instruction is forwarded to the calling platform through the data collaboration platform; the calling platform initiates a one-way video call request to the called mobile phone according to the data collaboration instruction; the called mobile phone accepts the one-way video call request and establishes a data collaboration channel with the data collaboration platform, thereby avoiding the phenomenon of significant increase in bandwidth occupancy of the calling platform due to the simultaneous playback of audio and video data, and greatly reducing the signaling complexity, thereby reducing the burden on the system.

[0007] A first aspect of the present invention provides a media synchronization method based on data collaboration, comprising:

[0008] S1, the enterprise user initiates an audio call request to the calling platform, and the calling platform forwards the audio call request to the called mobile phone;

[0009] S2, the called mobile phone accepts the audio call request and establishes an audio communication channel with the enterprise user;

[0010] S3, the enterprise user actively triggers the data collaboration instruction, and forwards the data collaboration instruction to the call platform through the data collaboration platform;

[0011] S4, the calling platform initiates a one-way video call request to the called mobile phone according to the data coordination instruction;

[0012] S5, the called mobile phone accepts the one-way video call request, establishes a data collaboration channel with the data collaboration platform, and returns a response message to the calling platform;

[0013] S6, the call platform forwards the response information to the enterprise user through the data collaboration platform;

[0014] S7, the enterprise user actively triggers a data push instruction, and forwards the data push instruction to the call platform through the data collaboration platform;

[0015] S8, the calling platform pushes the video stream to the called mobile phone according to the data push instruction and the bandwidth reported by the called mobile phone, plays the video file of the corresponding bit rate in real time, and generates a video index notification;

[0016] S9, the call platform forwards the video index notification to the enterprise user through the data collaboration platform;

[0017] S10, the enterprise user plays the same video file as the called mobile phone synchronously according to the video index notification;

[0018] S11, repeating sub-steps S7 to S10, the enterprise user and the called mobile phone perform real-time voice interaction.

[0019] Furthermore, after S8 and before S9, the method further includes:

[0020] S12, mixing the audio stream of the video file with the audio stream of the enterprise user through the call platform.

[0021] Furthermore, the S12 specifically includes:

[0022] S1201, standardize the audio stream of the video file and the audio stream of the enterprise user to obtain a standardized timestamp:

[0023] ntp = f(rtp t s) = sample_rate / 1000*rtp t s+offset

[0024] Among them, ntp means NTP timestamp, that is, standardized timestamp, f(rtp t s) represents the function that converts the RTP timestamp into a standardized timestamp. t s represents the RTP timestamp, sample_rate represents the audio stream sampling rate, and offset represents the offset;

[0025] S1202: Perform audio frame trimming according to the audio stream of the video file and the standardized timestamp of the audio stream of the enterprise user:

[0026]

[0027] Where diff represents the error between the standardized timestamp of the audio stream of the current frame video file and the standardized timestamp of the audio stream of the current frame enterprise user. v Indicates the standardized timestamp of the audio stream of the current frame video file, ntp a Indicates the standardized timestamp of the audio stream of the enterprise user in the current frame;

[0028] S1203: Mixing the trimmed audio stream of the video file and the audio stream of the enterprise user:

[0029] A m =α*A v +β*A a

[0030] Among them, A m represents the audio stream after mixing, α represents the weight coefficient of the audio stream of the video file, A v represents the audio stream of the video file, β represents the weight coefficient of the audio stream of the enterprise user, and A a Represents the audio stream of enterprise users.

[0031] Furthermore, the enterprise user and the calling platform, as well as the calling platform and the called mobile phone, establish an audio and / or video communication session through the SIP protocol, and use the SDP protocol to describe the audio and / or video parameters in the session;

[0032] The calling platform and the data collaboration platform perform network communication via the HTTPS protocol.

[0033] Furthermore, the video index notification includes file name information and timestamp offset information.

[0034] A second aspect of the present invention provides a media synchronization system based on data collaboration, comprising:

[0035] A first forwarding module is used for an enterprise user to initiate an audio call request to a calling platform, and forward the audio call request to a called mobile phone through the calling platform;

[0036] An establishing module, used for the called mobile phone to accept the audio call request and establish an audio communication channel with the enterprise user;

[0037] A second forwarding module is used for the enterprise user to actively trigger the data collaboration instruction and forward the data collaboration instruction to the call platform through the data collaboration platform;

[0038] An initiating module, used for the calling platform to initiate a one-way video call request to the called mobile phone according to the data coordination instruction;

[0039] A return module, used for the called mobile phone to accept the one-way video call request, establish a data collaboration channel with the data collaboration platform, and return a response message to the calling platform;

[0040] A third forwarding module, configured for the call platform to forward the response information to the enterprise user via the data collaboration platform;

[0041] A fourth forwarding module, used for the enterprise user to actively trigger a data push instruction, and forward the data push instruction to the call platform through the data collaboration platform;

[0042] A generation module, used for the calling platform to push the video stream to the called mobile phone according to the data push instruction and the bandwidth reported by the called mobile phone, play the video file of the corresponding bit rate in real time, and generate a video index notification;

[0043] A fifth forwarding module, configured for the call platform to forward the video index notification to the enterprise user via the data collaboration platform;

[0044] The synchronization module is used for the enterprise user to synchronously play the same video file as the called mobile phone according to the video index notification.

[0045] Furthermore, the media synchronization system based on data collaboration also includes:

[0046] The processing module is used to mix the audio stream of the video file with the audio stream of the enterprise user through the call platform.

[0047] Furthermore, the processing module is specifically used for:

[0048] The audio stream of the video file and the audio stream of the enterprise user are standardized to obtain a standardized timestamp:

[0049] ntp = f(rtp t s) = sample_rate / 1000*rtp t s+offset

[0050] Among them, ntp means NTP timestamp, that is, standardized timestamp, f(rtp t s) represents the function that converts the RTP timestamp into a standardized timestamp. t s represents the RTP timestamp, sample_rate represents the audio stream sampling rate, and offset represents the offset;

[0051] Perform audio frame trimming according to the standardized timestamps of the audio stream of the video file and the audio stream of the enterprise user:

[0052]

[0053] Where diff represents the error between the standardized timestamp of the audio stream of the current frame video file and the standardized timestamp of the audio stream of the current frame enterprise user. v Indicates the standardized timestamp of the audio stream of the current frame video file, ntp a Indicates the standardized timestamp of the audio stream of the enterprise user in the current frame;

[0054] The trimmed audio stream of the video file and the audio stream of the enterprise user are mixed:

[0055] A m =α*A v +β*A a

[0056] Among them, A m represents the audio stream after mixing, α represents the weight coefficient of the audio stream of the video file, A v represents the audio stream of the video file, β represents the weight coefficient of the audio stream of the enterprise user, and A a Represents the audio stream of enterprise users.

[0057] Furthermore, the enterprise user and the calling platform, as well as the calling platform and the called mobile phone, establish an audio and / or video communication session through the SIP protocol, and use the SDP protocol to describe the audio and / or video parameters in the session;

[0058] The calling platform and the data collaboration platform perform network communication via the HTTPS protocol.

[0059] Furthermore, the video index notification includes file name information and timestamp offset information.

[0060] The beneficial effects of the present invention are as follows:

[0061] The method and system of the present invention actively trigger the data collaboration instruction through the enterprise user, and forward the data collaboration instruction to the calling platform through the data collaboration platform. The calling platform initiates a one-way video call request to the called mobile phone according to the data collaboration instruction. The called mobile phone accepts the one-way video call request and establishes a data collaboration channel with the data collaboration platform, thereby avoiding the phenomenon of significantly increasing the bandwidth occupancy of the calling platform due to the simultaneous playback of audio and video data, and greatly reducing the signaling complexity, thereby reducing the burden on the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0063] Figure 1 A flow chart of a media synchronization method based on data collaboration provided by an embodiment of the present invention;

[0064] Figure 2 A structural schematic diagram of a media synchronization system based on data collaboration provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0066] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0067] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0068] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0069] The present invention proposes a media synchronization method and system based on data collaboration, which solves the technical problem that the method of simultaneously playing audio and video data in the prior art will cause a significant increase in the bandwidth occupancy of the call platform and a significant increase in signaling complexity, thereby increasing the burden on the system.

[0070] Method Embodiment

[0071] Reference Manual Attached Figure 1 , showing a flow chart of a media synchronization method based on data collaboration provided by an embodiment of the present invention.

[0072] An embodiment of the present invention provides a media synchronization method based on data collaboration, including:

[0073] Specifically, the method includes steps S1 to S12.

[0074] S1, the enterprise user initiates an audio call request to the calling platform, and the calling platform forwards the audio call request to the called mobile phone.

[0075] In the present invention, the initiation of an audio call request first establishes a basic communication channel between the enterprise user and the called mobile phone, ensuring that both parties can interact through voice, laying the foundation for subsequent video synchronization and data collaboration. By transferring audio requests through the call platform and centrally managing communication signaling, the entire process is more unified and easy to control. This helps the call platform process audio, video and data streams under a unified framework.

[0076] S2: The called mobile phone accepts the audio call request and establishes an audio communication channel with the enterprise user.

[0077] In the present invention, by establishing an audio communication channel, it is ensured that real-time voice interaction can be achieved between the enterprise user and the called mobile phone. This is the basis of the entire collaborative process and provides the necessary prerequisite for subsequent video collaboration and data synchronization. The establishment of the audio channel provides a stable communication foundation for subsequent video synchronization and data push. Enterprise users can trigger data collaboration instructions as needed during an audio call, thereby guiding the synchronization of video content.

[0078] S3, the enterprise user actively triggers the data collaboration instruction and forwards the data collaboration instruction to the call platform through the data collaboration platform.

[0079] In the present invention, enterprise users can flexibly trigger data collaboration according to business needs, rather than relying on preset automatic processes. This active triggering mechanism ensures that enterprise users can decide when to start video synchronization or data push according to the actual situation of the call, increasing the flexibility of control. The forwarding of data collaboration instructions is centrally processed by the data collaboration platform, reducing the complexity of signaling interactions directly between enterprise users and the call platform. This can reduce the signaling load of the entire system, thereby improving communication efficiency.

[0080] S4: The calling platform initiates a one-way video call request to the called mobile phone according to the data coordination instruction.

[0081] In the present invention, through a one-way video call request, the call platform only pushes the video stream to the called mobile phone, without returning the video stream from the called mobile phone. This method greatly reduces the bandwidth resources required for two-way video streaming, especially in high-concurrency scenarios, which helps to improve the overall performance and resource utilization of the call platform. One-way video calls reduce the burden of data transmission and processing, and reduce jitter, delay and synchronization problems caused by two-way video streaming. This makes the system more stable, especially in poor network conditions, one-way video can more effectively maintain call quality.

[0082] S5, the called mobile phone accepts the one-way video call request, establishes a data collaboration channel with the data collaboration platform, and returns a response message to the calling platform.

[0083] In the present invention, by establishing a data collaboration channel, the called mobile phone can receive synchronized index information and video streams, so that the video playback between the enterprise user and the called user can be synchronized. This data collaboration mechanism avoids the complexity and instability caused by direct transmission through platform transfer. After establishing the data collaboration channel, the called mobile phone will return a response message to the calling platform. This feedback mechanism ensures that the calling platform can understand the connection status and video reception status of the called mobile phone in real time, provides a confirmation basis for subsequent video push and data synchronization, and ensures the accuracy and timing consistency of the collaboration between the two ends.

[0084] S6, the call platform forwards the response information to the enterprise user through the data collaboration platform.

[0085] In the present invention, the enterprise user can know in real time whether the called mobile phone has successfully established the video channel and data collaboration channel through the received response information, so as to trigger subsequent operations according to the actual situation, such as starting to push video files or adjusting the collaboration strategy, which enhances the flexibility and control ability of the operation. The response information is forwarded by the data collaboration platform, which avoids the complex signaling interaction between the enterprise user and the called mobile phone. The call platform centrally manages and coordinates the signaling, which not only simplifies the process, but also reduces the burden of direct communication between the parties, and improves the efficiency and stability of the system.

[0086] S7, the enterprise user actively triggers the data push instruction and forwards the data push instruction to the call platform through the data collaboration platform.

[0087] In the present invention, enterprise users can flexibly decide when to push data (such as video streams) according to actual business needs. The mechanism of actively triggering data push instructions enables enterprise users to accurately control the start time of data synchronization, ensuring that the collaborative operation is highly matched with the user's business scenario. Data push instructions are transferred through the data collaboration platform to ensure that the transmission of instructions in the system is synchronous and consistent. This ensures that the call platform can respond to push instructions in a timely manner, thereby accurately pushing videos or other data, and realizing synchronous playback of both the caller and the called party.

[0088] S8, the calling platform pushes the video stream to the called mobile phone according to the data push instruction and the bandwidth reported by the called mobile phone, plays the video file of the corresponding bit rate in real time, and generates a video index notification.

[0089] It should be noted that the video index notification includes file name information and timestamp offset information.

[0090] In the present invention, the calling platform dynamically adjusts the pushed video bit rate according to the bandwidth reported by the called mobile phone, ensuring that the video stream can be adaptively optimized according to network conditions. This method can avoid freezes or video quality degradation caused by network fluctuations, thereby providing users with a smoother video viewing experience. Generating a video index notification, including file name information and timestamp offset information, can ensure that the enterprise user and the called mobile phone are synchronized when playing the same video. This index notification ensures the time consistency of different terminal devices when playing content, which helps to achieve accurate media synchronization and avoid asynchrony problems.

[0091] S9, the call platform forwards the video index notification to the enterprise user through the data collaboration platform.

[0092] In the present invention, the index notification is forwarded by the data collaboration platform, and the enterprise user does not need to directly interact with the called mobile phone or the calling platform. The collaboration platform simplifies the entire process as an intermediary, uniformly manages the process of information transmission, and reduces the complexity of direct interaction between multiple parties.

[0093] S10, the enterprise user plays the same video file as the called mobile phone synchronously according to the video index notification.

[0094] In the present invention, through the file name and timestamp offset information in the video index notification, the enterprise user can accurately and synchronously play the same video file as the called mobile phone, ensuring that users at both ends achieve accurate synchronization when watching the same video. Synchronous playback of video files enables enterprise users to interact with the called user in real time through voice or other forms. The consistency of video content makes the interaction smoother, and both parties can communicate based on the same visual content, improving the overall interactive experience.

[0095] S11, repeat sub-steps S7 to S10, and the enterprise user and the called mobile phone perform real-time voice interaction.

[0096] In the present invention, by repeating S7 to S10, enterprise users can continuously push data and synchronize videos according to actual business scenarios, so that audio and video remain synchronized in real time during the entire interactive process. The synchronization of audio and video is crucial to providing an immersive interactive experience, especially in scenarios where discussions or collaboration based on visual content are required. Each video push dynamically adjusts the video bit rate according to the current network bandwidth of the called phone to ensure smooth video playback. By repeating these steps, the quality and stability of the video stream can be continuously optimized to avoid poor viewing experience caused by network fluctuations.

[0097] It should be noted that the audio and / or video communication sessions are established between the enterprise user and the calling platform, as well as between the calling platform and the called mobile phone, through the SIP protocol, and the audio and / or video parameters in the session are described using the SDP protocol.

[0098] The call platform and the data collaboration platform communicate over the network via the HTTPS protocol.

[0099] Among them, the SIP protocol (Session Initiation Protocol) is a network protocol used to establish, modify and terminate multimedia communication sessions. It is widely used in applications such as VoIP (Internet phone), video calls, instant messaging, etc. The main function of SIP is to establish a connection between two or more communicating parties and manage the signal exchange during the session, such as the start, control and end of the session. The SIP protocol supports multiple forms of communication, including voice, video, text, etc., and has strong flexibility, and can achieve cross-platform and cross-device communication on the Internet.

[0100] Among them, the SDP protocol (Session Description Protocol) is a format standard for describing multimedia communication sessions. It is usually used in conjunction with other protocols such as the SIP protocol to provide audio and video parameter information in the session. The main function of SDP is to describe the detailed information of the session, such as media type (audio, video), codec, media format, transmission protocol, bandwidth requirements, and network address. This information helps the communicating parties or multiple parties understand how to transmit data when establishing a connection, thereby ensuring the compatibility and smoothness of audio and video communications. SDP itself does not transmit data, but provides metadata information for the session.

[0101] Among them, the HTTPS protocol (HyperText Transfer Protocol Secure) is an encrypted version of the HTTP protocol, which is used to ensure the security of data transmitted between the client (such as the browser) and the server. HTTPS provides data confidentiality, integrity and authentication by adding SSL / TLS (Secure Sockets Layer / Transport Layer Security) encryption on the basis of HTTP to prevent information from being stolen or tampered with during transmission. It is widely used in scenarios where user privacy and sensitive information need to be protected, such as online payment, login, etc., to ensure data security.

[0102] In the present invention, the SIP protocol, as a widely used multimedia communication protocol, is compatible with a variety of devices and platforms and supports voice, video and other communication forms. This standardized communication method enables enterprise users, call platforms and called mobile phones to efficiently realize audio and video communications, thereby improving the compatibility and interoperability of the system. The SDP protocol describes the audio and video parameters in the session, including information such as codecs, bandwidth, and media formats. Through the SDP protocol, the call platform can accurately negotiate multimedia parameters to ensure smooth transmission of audio and video data between different devices. The HTTPS protocol is used between the call platform and the data collaboration platform to ensure secure data transmission. HTTPS uses encryption mechanisms to prevent data from being intercepted or tampered with, thereby protecting user privacy and sensitive information security, especially in the video push and data synchronization links, effectively preventing network attacks and ensuring data integrity and confidentiality.

[0103] In a possible implementation manner, after S8 and before S9, the method further includes:

[0104] S12, mixing the audio stream of the video file with the audio stream of the enterprise user through the call platform.

[0105] In the present invention, by mixing the audio stream of the video file with the real-time audio stream of the enterprise user, the enterprise user's explanation or guidance can be heard at the same time when the called mobile phone user plays the video. This can enhance the user's sense of immersion, allowing the called user to receive voice guidance or explanations from the enterprise user while watching the video, thereby improving the overall interactive experience and the effectiveness of information transmission. The mixing process can synchronize the audio in the video with the voice of the enterprise user in real time, avoiding the situation where the called user is inconsistent or disconnected from the video content when listening to the enterprise user's explanation. Through mixing, the enterprise user's explanation can be seamlessly connected with the video content, ensuring that both parties are in the same rhythm when discussing the key content in the video, and ensuring the coherence and fluency of communication.

[0106] In a possible implementation, S12 specifically includes sub-steps S1201 to S1203:

[0107] S1201, standardize the audio stream of the video file and the audio stream of the enterprise user to obtain a standardized timestamp:

[0108] ntp = f(rtp t s) = sample_rate / 1000*rtp t s+offset

[0109] Among them, ntp means NTP timestamp, that is, standardized timestamp, f(rtp t s) represents the function that converts the RTP timestamp into a standardized timestamp. ts represents the RTP timestamp, sample_rate represents the audio stream sampling rate, and offset represents the offset.

[0110] S1202: Perform audio frame trimming according to the audio stream of the video file and the standardized timestamp of the audio stream of the enterprise user:

[0111]

[0112] Where diff represents the error between the standardized timestamp of the audio stream of the current frame video file and the standardized timestamp of the audio stream of the current frame enterprise user. v Indicates the standardized timestamp of the audio stream of the current frame video file, ntp a Indicates the normalized timestamp of the current frame of the enterprise user's audio stream.

[0113] S1203: Mixing the audio stream of the trimmed video file and the audio stream of the enterprise user:

[0114] A m =α*A v +β*A a

[0115] Among them, A m represents the audio stream after mixing, α represents the weight coefficient of the audio stream of the video file, A v represents the audio stream of the video file, β represents the weight coefficient of the audio stream of the enterprise user, and A a Represents the audio stream of enterprise users.

[0116] In the present invention, standardized timestamp and frame pruning operations greatly improve the accuracy of audio stream mixing processing. After the timestamp of the audio frame is standardized, the consistency of the audio stream can be ensured at the millisecond level. Frame pruning selectively discards or retains audio frames according to the timestamp difference, further improving the accuracy of the audio stream, making the entire audio mixing process more accurate, thereby reducing possible audio delays or jams. Through the conversion processing of RTP timestamps and NTP timestamps, combined with the pruning mechanism of audio frames, problems caused by network delays or audio frame asynchrony can be effectively reduced. In particular, for errors greater than 100ms, effective audio frame processing can be performed through a discarding strategy to ensure the low-latency characteristics of real-time mixing processing. Through accurate audio stream pruning and mixing processing, the transmission of invalid audio streams can be reduced, redundant data can be avoided, and bandwidth can be saved.

[0117] System Example

[0118] Reference Manual Attached Figure 2 , showing a structural diagram of a media synchronization system based on data collaboration provided by an embodiment of the present invention.

[0119] The present invention proposes a media synchronization system 20 based on data collaboration, comprising:

[0120] The first forwarding module 201 is used for an enterprise user to initiate an audio call request to a calling platform, and forward the audio call request to a called mobile phone through the calling platform.

[0121] Establishing module 202 is used for the called mobile phone to accept the audio call request and establish an audio communication channel with the enterprise user.

[0122] The second forwarding module 203 is used for the enterprise user to actively trigger the data collaboration instruction, and forward the data collaboration instruction to the call platform through the data collaboration platform.

[0123] The initiating module 204 is used for the calling platform to initiate a one-way video call request to the called mobile phone according to the data coordination instruction.

[0124] The return module 205 is used for the called mobile phone to accept the one-way video call request, establish a data collaboration channel with the data collaboration platform, and return a response message to the calling platform.

[0125] The third forwarding module 206 is used for the calling platform to forward the response information to the enterprise user through the data collaboration platform.

[0126] The fourth forwarding module 207 is used for the enterprise user to actively trigger the data push instruction, and forward the data push instruction to the call platform through the data collaboration platform.

[0127] The generating module 208 is used for the calling platform to push the video stream to the called mobile phone according to the data push instruction and the bandwidth reported by the called mobile phone, play the video file of the corresponding bit rate in real time, and generate a video index notification.

[0128] The fifth forwarding module 209 is used for the calling platform to forward the video index notification to the enterprise user through the data collaboration platform.

[0129] The synchronization module 210 is used for the enterprise user to synchronously play the same video file as the called mobile phone according to the video index notification.

[0130] Furthermore, the media synchronization system based on data collaboration also includes:

[0131] The processing module 211 is used to mix the audio stream of the video file with the audio stream of the enterprise user through the call platform.

[0132] Furthermore, the processing module 211 is specifically used for:

[0133] Standardize the audio stream of the video file and the audio stream of the enterprise user to obtain a standardized timestamp:

[0134] ntp = f(rtp t s) = sample_rate / 1000*rtp t s+offset

[0135] Among them, ntp means NTP timestamp, that is, standardized timestamp, f(rtp t s) represents the function that converts the RTP timestamp into a standardized timestamp. t s represents the RTP timestamp, sample_rate represents the audio stream sampling rate, and offset represents the offset.

[0136] Perform audio frame trimming based on the standardized timestamps of the audio stream of the video file and the audio stream of the enterprise user:

[0137]

[0138] Where diff represents the error between the standardized timestamp of the audio stream of the current frame video file and the standardized timestamp of the audio stream of the current frame enterprise user. v Indicates the standardized timestamp of the audio stream of the current frame video file, ntp a Indicates the normalized timestamp of the current frame of the enterprise user's audio stream.

[0139] Mix the audio stream of the trimmed video file and the audio stream of the enterprise user:

[0140] A m =α*A v +β*A a

[0141] Among them, A m represents the audio stream after mixing, α represents the weight coefficient of the audio stream of the video file, A v represents the audio stream of the video file, β represents the weight coefficient of the audio stream of the enterprise user, and A a Represents the audio stream of enterprise users.

[0142] Furthermore, audio and / or video communication sessions are established between the enterprise user and the calling platform, as well as between the calling platform and the called mobile phone, through the SIP protocol, and the audio and / or video parameters in the session are described using the SDP protocol.

[0143] The call platform and the data collaboration platform communicate over the network via the HTTPS protocol.

[0144] Furthermore, the video index notification includes file name information and timestamp offset information.

[0145] The present invention provides a data collaboration-based media synchronization system 20, which can implement the various steps and technical effects of the above-mentioned data collaboration-based media synchronization method. To avoid repetition, the present invention will not elaborate on them.

[0146] The beneficial effects of the present invention are as follows:

[0147] The method and system of the present invention actively trigger the data collaboration instruction through the enterprise user, and forward the data collaboration instruction to the calling platform through the data collaboration platform. The calling platform initiates a one-way video call request to the called mobile phone according to the data collaboration instruction. The called mobile phone accepts the one-way video call request and establishes a data collaboration channel with the data collaboration platform, thereby avoiding the phenomenon of significantly increasing the bandwidth occupancy of the calling platform due to the simultaneous playback of audio and video data, and greatly reducing the signaling complexity, thereby reducing the burden on the system.

[0148] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, but not to limit the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A media synchronization method based on data collaboration, characterized in that: include: S1, the enterprise user initiates an audio call request to the calling platform, and the calling platform forwards the audio call request to the called mobile phone; S2, the called mobile phone accepts the audio call request and establishes an audio communication channel with the enterprise user; S3, the enterprise user actively triggers the data collaboration instruction, and forwards the data collaboration instruction to the call platform through the data collaboration platform; S4, the calling platform initiates a one-way video call request to the called mobile phone according to the data coordination instruction; S5, the called mobile phone accepts the one-way video call request, establishes a data collaboration channel with the data collaboration platform, and returns a response message to the calling platform; S6, the call platform forwards the response information to the enterprise user through the data collaboration platform; S7, the enterprise user actively triggers a data push instruction, and forwards the data push instruction to the call platform through the data collaboration platform; S8, the calling platform pushes the video stream to the called mobile phone according to the data push instruction and the bandwidth reported by the called mobile phone, plays the video file of the corresponding bit rate in real time, and generates a video index notification; S9, the call platform forwards the video index notification to the enterprise user through the data collaboration platform; S10, the enterprise user plays the same video file as the called mobile phone synchronously according to the video index notification; S11, repeating sub-steps S7 to S10, the enterprise user and the called mobile phone perform real-time voice interaction.

2. The media synchronization method based on data collaboration according to claim 1, characterized in that: After S8 and before S9, the method further includes: S12, mixing the audio stream of the video file with the audio stream of the enterprise user through the call platform.

3. The media synchronization method based on data collaboration according to claim 2, characterized in that: The S12 specifically includes: S1201, standardize the audio stream of the video file and the audio stream of the enterprise user to obtain a standardized timestamp: ntp=f(rtp t s)=sample_rate / 1000*rtp t s+offset Among them, ntp means NTP timestamp, that is, standardized timestamp, f(rtp t s) represents the function that converts the RTP timestamp into a standardized timestamp. t s represents the RTP timestamp, sample_rate represents the audio stream sampling rate, and offset represents the offset; S1202: Perform audio frame trimming according to the audio stream of the video file and the standardized timestamp of the audio stream of the enterprise user: Where diff represents the error between the standardized timestamp of the audio stream of the current frame video file and the standardized timestamp of the audio stream of the current frame enterprise user. v Indicates the standardized timestamp of the audio stream of the current frame video file, ntp a Indicates the standardized timestamp of the audio stream of the enterprise user in the current frame; S1203: Mixing the trimmed audio stream of the video file and the audio stream of the enterprise user: A m =α*A v +β*A a Among them, A m represents the audio stream after mixing, α represents the weight coefficient of the audio stream of the video file, A v represents the audio stream of the video file, β represents the weight coefficient of the audio stream of the enterprise user, and A a Represents the audio stream of enterprise users.

4. The media synchronization method based on data collaboration according to claim 1, characterized in that: The enterprise user and the calling platform, as well as the calling platform and the called mobile phone, establish an audio and / or video communication session through the SIP protocol, and use the SDP protocol to describe the audio and / or video parameters in the session; The calling platform and the data collaboration platform perform network communication via the HTTPS protocol.

5. The media synchronization method based on data collaboration according to claim 1, characterized in that: The video index notification includes file name information and timestamp offset information.

6. A media synchronization system based on data collaboration, characterized in that: The data collaboration-based media synchronization system is used to implement the data collaboration-based media synchronization method according to any one of claims 1 to 5, wherein the data collaboration-based media synchronization system specifically includes: A first forwarding module is used for an enterprise user to initiate an audio call request to a calling platform, and forward the audio call request to a called mobile phone through the calling platform; An establishing module, used for the called mobile phone to accept the audio call request and establish an audio communication channel with the enterprise user; A second forwarding module is used for the enterprise user to actively trigger the data collaboration instruction and forward the data collaboration instruction to the call platform through the data collaboration platform; An initiating module, used for the calling platform to initiate a one-way video call request to the called mobile phone according to the data coordination instruction; A return module, used for the called mobile phone to accept the one-way video call request, establish a data collaboration channel with the data collaboration platform, and return a response message to the calling platform; A third forwarding module, configured for the call platform to forward the response information to the enterprise user via the data collaboration platform; A fourth forwarding module, used for the enterprise user to actively trigger a data push instruction, and forward the data push instruction to the call platform through the data collaboration platform; A generation module, used for the calling platform to push the video stream to the called mobile phone according to the data push instruction and the bandwidth reported by the called mobile phone, play the video file of the corresponding bit rate in real time, and generate a video index notification; A fifth forwarding module, configured for the call platform to forward the video index notification to the enterprise user via the data collaboration platform; The synchronization module is used for the enterprise user to synchronously play the same video file as the called mobile phone according to the video index notification.

7. The media synchronization system based on data collaboration according to claim 6, characterized in that: The media synchronization system based on data collaboration also includes: The processing module is used to mix the audio stream of the video file with the audio stream of the enterprise user through the call platform.

8. The media synchronization system based on data collaboration according to claim 7, characterized in that: The processing module is specifically used for: The audio stream of the video file and the audio stream of the enterprise user are standardized to obtain a standardized timestamp: ntp=f(rtp t s)=sample_rate / 1000*rtp t s+offset Among them, ntp means NTP timestamp, that is, standardized timestamp, f(rtp t s) represents the function that converts the RTP timestamp into a standardized timestamp. t s represents the RTP timestamp, sample_rate represents the audio stream sampling rate, and offset represents the offset; Perform audio frame trimming according to the standardized timestamps of the audio stream of the video file and the audio stream of the enterprise user: Where diff represents the error between the standardized timestamp of the audio stream of the current frame video file and the standardized timestamp of the audio stream of the current frame enterprise user. v Indicates the standardized timestamp of the audio stream of the current frame video file, ntp a Indicates the standardized timestamp of the audio stream of the enterprise user in the current frame; The trimmed audio stream of the video file and the audio stream of the enterprise user are mixed: A m =α*A v +β*A a Among them, A m represents the audio stream after mixing, α represents the weight coefficient of the audio stream of the video file, A v represents the audio stream of the video file, β represents the weight coefficient of the audio stream of the enterprise user, and A a Represents the audio stream of enterprise users.

9. The media synchronization system based on data collaboration according to claim 6, characterized in that: The enterprise user and the calling platform, as well as the calling platform and the called mobile phone, establish an audio and / or video communication session through the SIP protocol, and use the SDP protocol to describe the audio and / or video parameters in the session; The calling platform and the data collaboration platform perform network communication via the HTTPS protocol.

10. The media synchronization system based on data collaboration according to claim 6, characterized in that: The video index notification includes file name information and timestamp offset information.

Images