Distributed cross-domain audio and video sharing synchronization system and method

Abstract

The invention discloses a distributed cross-domain audio and video sharing synchronization system and method, and the system comprises a plurality of cross-domain playing domains, the playing domain which initiates audio and video sharing is used as a main playing domain, the playing domain which accepts invitation is used as a slave playing domain, and one device in each playing domain is used as an intra-domain main device. The intra-domain master device belongs to a first level of a distributed level, other devices in each playing domain serve as intra-domain slave devices, the intra-domain slave devices belong to a second level of the distributed level, and transmission of transmission signaling messages is carried out between the master playing domain and the slave playing domains through a cross-domain signaling service. The main playing domain main end equipment comprises a first data acquisition and analysis module, a first control playing control module and a first cross-domain signaling module; and the slave playing domain master terminal equipment comprises a second data acquisition and analysis module, a second playing control module, a second cross-domain signaling module and a second synchronous control module. The method and the device are used for realizing audio and video content sharing playing in a cross-domain multi-screen fusion scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of audio and video playback, and in particular relates to a distributed cross-domain audio and video sharing synchronization system and method. Background Art

[0002] With the development of technology, the application scenarios of multimedia are changing with each passing day. From the original TV screen, to mobile terminals, to outdoor large screens, and then to the multi-screen integration of the Internet of Everything. With the advancement of network communication technology, emerging application scenarios such as home theater and cockpit entertainment have a technical basis for realization. This multi-screen integration in an independent space has a natural demand for multi-screen simultaneous broadcasting in the field of video playback. For example, in a smart car cockpit, multiple screen devices in different locations can share audio and video content such as movies, TV series, variety shows, and music, which can enhance the riding experience. Furthermore, audio and video content can also be shared and synchronized between multiple independent multi-screen integration spaces across domains, such as watching movies together between cars and watching movies together between home theaters.

[0003] Most media content contains both video and audio. To share media content between multiple spatial devices across domains, it is necessary to share and synchronize video and audio data. One common screen sharing method in existing technologies is screen projection, which is two-screen sharing between a small screen and a large screen. For example, a mobile terminal shares video content to a smart TV. This method can only achieve one-to-one screen projection, and cannot achieve sharing between multiple screen devices or cross-domain sharing. Summary of the invention

[0004] In view of the above problems, the technical problem to be solved by the present invention is to provide a distributed cross-domain audio and video sharing synchronization system and method for realizing shared playback of audio and video content in a cross-domain multi-screen fusion scenario.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] On the one hand, the present invention provides a distributed cross-domain audio and video sharing synchronization system, including multiple cross-domain playback domains, the playback domain that initiates audio and video sharing is used as the main playback domain, the playback domain that accepts the invitation is used as the slave playback domain, a device in each playback domain is used as a master device in the domain, the master device in the domain belongs to the first level of the distributed hierarchy, and other devices in each playback domain are used as slave devices in the domain, the slave devices in the domain belong to the second level of the distributed hierarchy. The main playback domain and the slave playback domain are used to transmit signaling messages through a cross-domain signaling service. The main playback domain master device includes a first data acquisition and analysis module, a first playback control module and a first cross-domain signaling module; the slave playback domain master device includes a second data acquisition and analysis module, a second playback control module, a second cross-domain signaling module and a second synchronization control module.

[0007] In one possible implementation, the first data acquisition and parsing module and the second data acquisition and parsing module are used to request audio and video data, and are used for format parsing and separation of audio and video data; the first playback control module and the second playback control module are used for decoding and playback of audio and video data; the first cross-domain signaling module and the second cross-domain signaling module are used to send signaling to and receive signaling from a cross-domain signaling service, and the cross-domain signaling service is used to transmit signaling messages between master-end devices within a cross-domain domain; the second synchronization control module is used for playback synchronization calculation between master-end devices within a cross-domain domain, and provides reference information for the synchronization control of the second playback control module.

[0008] Another aspect of the present invention provides a distributed cross-domain audio and video sharing synchronization method, using the distributed cross-domain audio and video sharing system as described above, wherein the audio and video data comes from a streaming media content platform, and comprises the following steps:

[0009] After the master device in the main playback domain obtains the streaming media playback address of the audio and video data, the address is set to the first data acquisition and parsing module and the second data acquisition and parsing module. The first data acquisition and parsing module and the second data acquisition and parsing module access the streaming media content platform according to the streaming media address and download the streaming media data. At the same time, the basic audio and video information of the downloaded streaming media data is parsed and separated from the audio and video.

[0010] The first playback control module and the second playback control module request the parsed audio data and video data from the data parsing and acquiring module, send the acquired audio data to the audio decoder for decoding, and send the video data to the video decoder for decoding; after the first playback control module and the second playback control module acquire the decoded audio and video, they synchronously control the audio and video, send the audio to play, and send the video to render;

[0011] The master device in the main playback domain initiates a simulcast sharing invitation to the master device in the slave playback domain through the cross-domain signaling service. The request carries the playback address and start time of the current streaming media video. The start time is the current playback time of the master device in the main playback domain when the invitation is sent. When the master device in the slave playback domain accepts the invitation, the master device in the slave playback domain initializes the player using the playback address and start time, and requests streaming media data starting from the start time from the media content platform.

[0012] During the simulcast sharing process, the master device in the main playback domain synchronizes the current playback time as the synchronization reference time SyncRefTime to the master device in the slave playback domain at a certain frequency;

[0013] When the master device in the slave playback domain successfully starts playing, a synchronization reference timeline is created; the synchronization reference timeline initialization time StartTime is calculated using the start time StartPlayTime transmitted from the main playback domain, the network transmission delay NetLatency and the player initialization time PrepareTime of the master device in the slave playback domain, and the calculation method is: StartTime = StartPlayTime + NetLatency + PrepareTime; during the playback process, the synchronization reference time SyncRefTime sent by the main playback domain is continuously received from the slave playback domain, and the master device in the slave playback domain uses the synchronization reference time and the network transmission delay to calculate the update time UpdateTime of the synchronization reference timeline, and uses the update time to update the synchronization reference timeline, and the update time calculation method is: UpdateTime = SyncRefTime + NetLatency. During the time period when the synchronization reference time is not updated, the synchronization reference timeline automatically runs according to the natural time.

[0014] In a possible implementation, the master playback domain and the slave playback domain perform playback operations and state synchronization through a cross-domain signaling service, the synchronization signaling performs two-way communication, and the playback operation can be triggered by either the master playback domain or the slave playback domain.

[0015] In one possible implementation, when a Seek jump is triggered in the main playback domain or the slave playback domain, the master end in the domain sends the Seek message and the Seek time point to other domains, and then the master-end player in the domain performs the Seek operation on its own end; other domains immediately execute the Seek operation after receiving the Seek message and the Seek time point; after each domain succeeds in Seek, it synchronizes the Seek completion message to other domains, and resumes playback when it confirms that the Seek of other domains is successful and its own Seek is also successful.

[0016] In a possible implementation, when the main playback domain or the slave playback domain triggers a Pause operation, the master end in the domain sends a Pause message to other domains, and other domains immediately respond to and execute a Pause operation after receiving the Pause message, thereby achieving a pause effect.

[0017] In a possible implementation, when Buffering is triggered by the main playback domain or the slave playback domain, the master end in the domain sends a BufferingStart buffering start message to other domains. After receiving the BufferingStart buffering start message, other domains pause playback and wait to receive a BufferingEnd buffering end message before resuming playback.

[0018] In a possible implementation, when a stream switching operation is performed in the main playback domain or the secondary playback domain, the master end within the domain that triggers the stream switching will send the new playback address and the start playback time to other master ends within the domain. After receiving the message, all master ends within the domain will immediately stop playing and use the new address and start playback time to open a new playback.

[0019] In a possible implementation, when sharing cross-domain audio and video simultaneous playback, the main playback domain provides the synchronization reference time, and the secondary playback domain synchronizes the audio and video with reference to the synchronization reference time provided by the main playback domain.

[0020] In a possible implementation, after the secondary playback domain receives the video data, it sends it to the decoder for decoding. After successful decoding, synchronization processing is performed on the current decoded frame:

[0021] After the secondary playback domain obtains the first-frame video decoding data, if the synchronization reference time axis has not been started, it waits for the synchronization reference time axis to start. If the synchronization reference time axis has already started, it enters the synchronization processing logic;

[0022] A_TimeVid = Clock_Time + VidTolerance_Time. When VidFrame_Time > A_TimeVid, it indicates that the current video frame is ahead of the video being played in the main playback domain. At this time, the playback rate needs to be slowed down until resynchronization;

[0023] B_TimeVid = Clock_Time - VidTolerance_Time. When VidFrame_Time < B_TimeVid, it indicates that the current video frame is behind the video being played in the main playback domain. At this time, the playback rate needs to be increased until resynchronization;

[0024] When B_TimeVid < VidFrame_Time < A_TimeVid, that is, VidFrame_Time is between B_TimeVid and A_TimeVid, it indicates that the current video frame is within the allowable synchronization error range of the video frame being played in the main playback domain and is in a synchronized state. The video is played at the normal playback frame rate;

[0025] Among them, the video frame timestamp is VidFrame_Time, the synchronization reference time axis time is Clock_Time, the acceptable synchronization error threshold for the video is VidTolerance_Time, the acceptable synchronization error threshold for audio and video can be set according to actual requirements, and the network transmission delay is Network_Latency.

[0026] In a possible implementation, after receiving audio data from the playback domain, it is sent to the decoder for decoding. After successful decoding, synchronization processing is performed on the current decoded frame. The detailed process of audio synchronization is as follows:

[0027] After obtaining the first-frame audio decoding data from the playback domain, if the synchronization reference timeline has not been started, wait for the synchronization reference timeline to start. If the synchronization reference timeline has already started, enter the synchronization processing logic;

[0028] A_TimeAud = Clock_Time + AudTolerance_Time + AudDevice_Latency. When AudFrame_Time > A_TimeAud, it indicates that the current audio frame is ahead of the audio being played in the main playback domain. At this time, the playback rate needs to be slowed down until resynchronization;

[0029] B_TimeAud = Clock_Time - AudTolerance_Time + AudDevice_Latency. When AudFrame_Time < B_TimeAud, it indicates that the current audio frame is behind the audio being played in the main playback domain. At this time, the playback rate needs to be increased until resynchronization;

[0030] When B_TimeAud < AudFrame_Time < A_TimeAud, that is, AudFrame_Time is between B_TimeAud and A_TimeAud, it indicates that the current audio frame is within the allowable synchronization error range of the audio frame being played in the main playback domain and is in a synchronized state. The audio can be played at the normal playback rate;

[0031] Among them, the audio frame timestamp is AudFrame_Time, the synchronization reference timeline time is Clock_Time, the acceptable synchronization error threshold for audio is AudTolerance_Time, the acceptable synchronization error threshold for audio and video can be set according to actual needs, the network transmission delay is Network_Latency, and the device hardware audio delay time is AudDevice_Latency. The device hardware audio delay refers to the time consumed from the input of the audio decoding data to the sound output from the device sound card.

[0032] The present invention has the following beneficial effects:

[0033] (1) Without using any third-party time system, directly using the presentation time (PTS) built into the video data, thus ensuring the unity of the time system without relying on a dedicated time server;

[0034] (2) The master and slave playback domains perform bidirectional state synchronization, which can effectively handle the impact of network jitter on the synchronization effect of the master and slave playback domains.

[0035] (3) It can realize the simultaneous playback of streaming media videos in a cross-domain heterogeneous network environment, achieving a viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the structure of a distributed cross-domain audio and video sharing and synchronization system according to an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of the structure of a distributed cross-domain audio and video sharing and synchronization system according to another embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Reference Figure 1, shown is a structural schematic diagram of a distributed cross-domain audio and video sharing synchronization system according to an embodiment of the present invention, including multiple cross-domain playback domains, the playback domain that initiates audio and video sharing is used as the main playback domain, the playback domain that accepts the invitation is used as the slave playback domain, and a device in each playback domain is used as a master device in the domain. The master device in the domain belongs to the first level of the distributed hierarchy, that is, the master device of the main playback domain and the master device of the slave playback domain included in the figure. Other devices in each playback domain are slave devices in the domain, and the slave devices in the domain belong to the second level of the distributed hierarchy, that is, the slave device of the main playback domain and the slave device of the slave playback domain included in the figure. The main playback domain and the slave playback domain are used to transmit signaling messages through a cross-domain signaling service. The master device of the main playback domain includes a first data acquisition and analysis module, a first playback control module and a first cross-domain signaling module; the master device of the slave playback domain includes a second data acquisition and analysis module, a second playback control module, a second cross-domain signaling module and a second synchronization control module. The first data acquisition and parsing module and the second data acquisition and parsing module are used to request audio and video data, and are used for format analysis and separation of audio and video data; the first playback control module and the second playback control module are used for decoding and playback of audio and video data; the first cross-domain signaling module and the second cross-domain signaling module are used to send signaling to and receive signaling from a cross-domain signaling service. The cross-domain signaling service is used to transmit signaling messages between master-end devices within a cross-domain domain. The signaling messages include but are not limited to playback operation instructions, such as selecting another master end within the domain, pausing playback (Pause), skipping playback (Seek), synchronizing playback time information, playback buffering, etc. The cross-domain signaling service transmits text information with a very small amount of data; the second synchronization control module is used for playback synchronization calculation between master-end devices within a cross-domain domain, and provides reference information for the synchronization control of the second playback control module.

[0040] In one embodiment of the present invention, see Figure 2 ,The audio and video data come from the streaming content platform.

[0041] The following is a flowchart of a distributed cross-domain audio and video sharing and synchronization method based on a distributed cross-domain audio and video sharing and synchronization system according to another embodiment of the present invention, assuming that the audio and video data comes from a streaming media content platform, including the following steps:

[0042] Continue to see Figure 2 After the master device in the main playback domain obtains the streaming media playback address of the audio and video data from the streaming media content platform, the streaming media playback address is set to the first data acquisition and parsing module and the second data acquisition and parsing module. The first data acquisition and parsing module and the second data acquisition and parsing module access the streaming media content platform according to the streaming media playback address and download the streaming media data, and at the same time parse the basic audio and video information of the downloaded streaming media data, and parse and separate the audio and video from the streaming media data;

[0043] The first playback control module and the second playback control module request the parsed audio data and video data from the data parsing and acquiring module, send the acquired audio data to the audio decoder for decoding (i.e., the first audio decoder and the second audio decoder), and send the video data to the video decoder for decoding (i.e., the first video decoder and the second video decoder); after the first playback control module and the second playback control module acquire the decoded audio and video, they synchronously control the audio and video, send the audio to be played, and send the video to be rendered;

[0044] The master device in the main playback domain initiates a simulcast sharing invitation to the master device in the slave playback domain through the cross-domain signaling service. The request carries the playback address and start time of the current streaming media video. The start time is the current playback time of the master device in the main playback domain when the invitation is sent. When the master device in the slave playback domain accepts the invitation, the master device in the slave playback domain initializes the player using the playback address and start time, and requests streaming media data starting from the start time from the media content platform.

[0045] During the simulcast sharing process, the master device in the main playback domain synchronizes the current playback time as the synchronization reference time SyncRefTime to the master device in the slave playback domain at a certain frequency;

[0046] When the master device in the slave playback domain successfully starts playing, a synchronization reference timeline is created; the synchronization reference timeline initialization time StartTime is calculated using the start time StartPlayTime transmitted from the main playback domain, the network transmission delay NetLatency and the player initialization time PrepareTime of the master device in the slave playback domain, and the calculation method is: StartTime = StartPlayTime + NetLatency + PrepareTime; during the playback process, the synchronization reference time SyncRefTime sent by the main playback domain is continuously received from the slave playback domain, and the master device in the slave playback domain uses the synchronization reference time and the network transmission delay to calculate the update time UpdateTime of the synchronization reference timeline, and uses the update time to update the synchronization reference timeline, and the update time calculation method is: UpdateTime = SyncRefTime + NetLatency. During the time period when the synchronization reference time is not updated, the synchronization reference timeline automatically runs according to the natural time.

[0047] In one embodiment of the present invention, the operation synchronization counter state synchronization logic of the master playback domain and the slave playback domain is as follows:

[0048] The main playback domain and the slave playback domain perform playback operations and status synchronization through cross-domain signaling services. The synchronization signaling communicates bidirectionally, and the playback operation can be triggered by the main playback domain or by the playback domain.

[0049] When the main playback domain or the slave playback domain triggers a Seek jump, the master end in the domain sends the Seek message and the Seek time point to other domains, and then the master end player in the domain performs the Seek operation on its own end; other domains immediately execute the Seek operation after receiving the Seek message and the Seek time point; after each domain succeeds in Seek, it synchronizes the Seek completion message to other domains, and resumes playback after confirming that the Seek of other domains is successful and its own Seek is also successful.

[0050] When the main playback domain or the slave playback domain triggers the Pause operation, the master end in the domain sends the Pause message to other domains. After receiving the Pause message, other domains immediately respond and perform the Pause operation to achieve the pause effect.

[0051] When the main playback domain or the slave playback domain triggers Buffering, the master end in the domain sends a BufferingStart buffering start message to other domains. After receiving the BufferingStart buffering start message, other domains pause playback and wait to receive the BufferingEnd buffering end message before resuming playback.

[0052] When the main playback domain or the slave playback domain performs a stream switching operation, the master end in the domain that triggers the stream switching sends the new playback address and start time to the master ends in other domains. After receiving the message, all the master ends in the domains immediately stop playback and start new playback using the new address and start time.

[0053] In one embodiment of the present invention, when cross-domain audio and video are simultaneously broadcast and shared, the main playback domain provides a synchronization reference time, and the slave playback domain performs audio and video synchronization with reference to the synchronization reference time provided by the main playback domain.

[0054] Furthermore, in one embodiment of the present invention, after receiving the video data from the playback domain, the video data is sent to the decoder for decoding. After the decoding is successful, the video frame synchronization process of the current decoded frame is performed as follows:

[0055] After getting the first frame of video decoding data from the playback domain, if the synchronization reference time axis has not been started, wait for the synchronization reference time axis to be started. If the synchronization reference time axis has been started, enter the synchronization processing logic;

[0056] A_TimeVid=Clock_Time+VidTolerance_Time, when VidFrame_Time>A_TimeVid, it means that the current video frame is ahead of the video being played in the main playback domain. At this time, the playback rate needs to be slowed down until resynchronization;

[0057] B_TimeVid = Clock_Time - VidTolerance_Time. When VidFrame_Time < B_TimeVid, it indicates that the current video frame lags behind the video being played in the main playback domain. At this time, the playback rate needs to be increased until resynchronization occurs;

[0058] When B_TimeVid < VidFrame_Time < A_TimeVid, that is, VidFrame_Time is between B_TimeVid and A_TimeVid, it indicates that the current video frame is within the allowable synchronization error range of the video frame being played in the main playback domain, and is in a synchronized state. The video is played at the normal playback frame rate.

[0059] Among them, VidFrame_Time is the video frame timestamp, Clock_Time is the time on the synchronization reference timeline, and VidTolerance_Time is the acceptable synchronization error threshold for the video. The acceptable synchronization error threshold for the video can be set according to actual requirements.

[0060] Furthermore, in an embodiment of the present invention, after receiving the audio data from the playback domain, it is sent to the decoder for decoding. After successful decoding, synchronization processing is performed on the current decoded frame. The detailed process of audio synchronization processing is as follows:

[0061] After obtaining the first-frame audio decoding data from the playback domain, if the synchronization reference timeline has not been started, wait for the synchronization reference timeline to start. If the synchronization reference timeline has already started, enter the synchronization processing logic;

[0062] A_TimeAud = Clock_Time + AudTolerance_Time + AudDevice_Latency. When AudFrame_Time > A_TimeAud, it indicates that the current audio frame is ahead of the audio being played in the main playback domain. At this time, the playback rate needs to be slowed down until resynchronization occurs;

[0063] B_TimeAud = Clock_Time - AudTolerance_Time + AudDevice_Latency. When AudFrame_Time < B_TimeAud, it indicates that the current audio frame lags behind the audio being played in the main playback domain. At this time, the playback rate needs to be increased until resynchronization occurs;

[0064] When B_TimeAud < AudFrame_Time < A_TimeAud, that is, AudFrame_Time is between B_TimeAud and A_TimeAud, it indicates that the current audio frame is within the allowable synchronization error range of the audio frame being played in the main playback domain and is in a synchronized state. The audio can be played at the normal playback rate;

[0065] Among them, the audio frame timestamp is AudFrame_Time, the synchronization reference time axis time is Clock_Time, the acceptable synchronization error threshold for audio is AudTolerance_Time, and the acceptable synchronization error thresholds for audio and video can be set according to actual requirements. The network transmission delay is Network_Latency, and the device hardware audio delay time is AudDevice_Latency. The device hardware audio delay refers to the time consumed from the input of the audio decoded data to the sound being played out from the device sound card.

[0066] For the processing of audio playback being ahead or behind, the general processing method is to wait when it is ahead and drop frames to catch up when it is behind. However, the characteristic of audio is its continuity. Waiting and dropping frames to catch up will destroy the continuity of the audio, making the audio sound intermittent and not smooth. In order to achieve the synchronization effect while maintaining the continuity of the audio, in an embodiment of the present invention, the following strategy is adopted to implement audio synchronization processing in the case of being ahead or behind:

[0067] First, refine the audio synchronization error threshold. The audio synchronization error threshold is subdivided into AudTolerance_TimeMax and AudTolerance_TimeMin, where AudTolerance_TimeMax > AudTolerance_TimeMin. AudTolerance_TimeMax is used to determine whether it is in a synchronized state; AudTolerance_TimeMin is used to determine whether synchronization has been re - achieved during the synchronization process to end the synchronization process. The advantage of this processing is to prevent the critical situation of frequent conversion between synchronized and unsynchronized states in the case of only one threshold; this threshold is not fixed and can be configured according to the actual situation, as long as AudTolerance_TimeMax > AudTolerance_TimeMin is ensured. The audio synchronization process introduces a synchronization judgment threshold interval [AudTolerance_TimeMin, AudTolerance_TimeMax]. AudTolerance_TimeMax is used to determine whether it is unsynchronized, and AudTolerance_TimeMin is used to determine whether the synchronization process is completed, which can eliminate the frequent conversion between synchronized and unsynchronized states that occurs at the critical position of synchronization judgment.

[0068] Secondly, for the processing of audio playing ahead and behind, considering maintaining the continuity of the audio, an audio speed - up / slow - down processing algorithm is adopted. While ensuring the sound speed change, the sampling rate, fundamental frequency, and formants of the voice do not change, so as to achieve the effect of changing speed without changing pitch. In addition, for the synchronization process, multiple levels of speed - up / slow - down values can be defined according to the severity of the unsynchronization to achieve a progressive processing during the synchronization process, making the audio change relatively subtly during the processing and reducing the user's perception; the speed - up / slow - down value is defined as Speed_Factor.

[0069] When playing ahead, AudFrame_Time > A0_TimeAud, A0_TimeAud = Clock_Time + AudTolerance_TimeMax + AudDevice_Latency; at this time, configure Speed_Frctor < 1.0, and achieve synchronization by slowing down the playing rate. During the synchronization process, when AudFrame_Time < A1_TimeAud, A1_TimeAud = Clock_Time + AudTolerance_TimeMin + AudDevice_Latency, it is determined that synchronization has been re - achieved, and the slow - down synchronization process ends, and the normal rate playback of Speed_Frctor = 1.0 is restored;

[0070] When the playback lags behind, AudFrame_Time < B0_TimeAud, where B0_TimeAud = Clock_Time - AudTolerance_TimeMax + AudDevice_Latency; at this time, configure Speed_Frctor > 1.0 to achieve synchronization by accelerating the playback rate. During the synchronization process, when AudFrame_Time > B1_TimeAud, where B1_TimeAud = Clock_Time - AudTolerance_TimeMin + AudDevice_Latency, it is determined that the synchronization state has been reached again, and the fast playback synchronization process is ended, and the normal playback rate of Speed_Frctor = 1.0 is restored.

[0071] During the synchronization process, according to the actual degree of out-of-synchronization severity, multiple levels of Speed_Frctor can be customized to make the synchronization process progressive, resulting in smoother playback during the synchronization process, less perception by users, and better user experience.

[0072] In a specific application example, the specific implementation steps of the distributed cross-domain audio and video sharing synchronization method of an embodiment of the present invention are as follows:

[0073] In the specific implementation steps, the cross-domain signaling service uses the standard Socket protocol to implement sending and listening.

[0074] The master device in the main playback domain invites the master device in the slave playback domain to perform synchronous sharing, and sends the streaming media playback address and the start playback time to the slave playback domain when inviting.

[0075] After the master device in the slave playback domain accepts the invitation, it initializes the player in the slave playback domain using the streaming media playback address and the start playback time. The player will request the streaming media data at the start playback time point from the streaming media content platform (which can be a streaming media server). At the same time, start the second cross-domain signaling module to listen for signaling messages. When starting playback in the slave playback domain, first create a synchronization reference timeline. The initialization time StartTime of the synchronization reference timeline is calculated using the start playback time StartPlayTime transmitted from the main playback domain, the network transmission delay NetLatency, and the initialization time PrepareTime of the master player in the slave playback domain. The calculation method is: StartTime = StartPlayTime + NetLatency + PrepareTime;

[0076] During the simulcast sharing process, the main playback domain synchronizes the current playback time as the synchronization reference time SyncRefTime to the slave playback domain at a frequency of every 500 milliseconds; during the playback process, the slave playback domain will continue to receive the synchronization reference time SyncRefTime sent by the main playback domain, and the slave playback domain uses the synchronization reference time and network transmission delay to calculate the update time UpdateTime of the synchronization reference time axis, and uses the update time to update the synchronization reference time axis. The update time calculation method is: UpdateTime = SyncRefTime + NetLatency. During the time period when the synchronization reference time is not updated, the synchronization reference time axis will automatically run according to the natural time;

[0077] The main playback domain or the slave playback domain triggers Seek jump SeetTime = SeekPosition, the master end in the domain sends the Seek message and the Seek time point SeekPosition to other domains, and then the master end player in the domain performs the Seek operation on its own end; other domains immediately perform the Seek operation after receiving the Seek message and the Seek time point SeekPosition; after the Seek succeeds, each domain synchronizes the Seek completion message to other domains, and resumes playback when confirming that the Seek of other domains is successful and the domain itself is also successful, and playback starts from the key frame closest to the SeekPosition;

[0078] When the main playback domain or the slave playback domain triggers the Pause operation, the master end in the domain sends the Pause message to other domains. After receiving the Pause message, other domains immediately respond and perform the Pause operation, thereby achieving the Pause effect;

[0079] When the main playback domain or the slave playback domain triggers Buffering, the master end in the domain sends the BufferingStart buffering start message to other domains. After receiving the BufferingStart buffering start message, other domains pause playback and wait for the BufferingEnd buffering end message to resume playback.

[0080] When the main playback domain or the slave playback domain performs a stream switching operation, the master end in the domain that triggers the stream switching sends the new playback address and start time to the master ends in other domains. After receiving the message, all the master ends in the domains immediately stop playback and start new playback using the new address and start time.

[0081] The specific implementation steps of video synchronization during video playback in the playback domain are as follows:

[0082] The acceptable synchronization error threshold VidTolerance_Time is set to twice the video frame display time. In this embodiment, the video frame rate is 25, so the display time of each frame is 40 milliseconds, and the acceptable synchronization error VidTolerance_Time = 40*2 = 80 milliseconds. That is, this embodiment can control the synchronization error within 2 video frames, that is, within 80 milliseconds;

[0083] After receiving the video data from the playback domain, the video data is continuously sent to the decoder for decoding. The decoder outputs the decoded video frame data. The playback control module calls the synchronization control module to judge the synchronization status of the decoded video frame data. The judgment uses parameters such as Tolerance_Time, synchronization reference time axis time Clock_Time, and network transmission delay NetLatency.

[0084] If it is determined to be synchronized, it is directly sent to the device for video rendering;

[0085] If the playback domain synchronization is judged to be 300 milliseconds ahead, it is necessary to synchronize through slow playback. When playing at normal speed, the display time of each frame is 40 milliseconds. If slow playback is needed at this time, adjust the display time of each frame to 60 milliseconds to achieve the slow playback effect. When the synchronization judgment result returns to the synchronized state, restore the display time of each frame to 40 milliseconds, so as to resume normal playback;

[0086] If the playback domain synchronization is judged to be 300 milliseconds behind, it is necessary to perform a catch-up synchronization process through fast playback. When playing at a normal rate, the display time of each frame is 40 milliseconds. If fast playback is required at this time, the display time of each frame is adjusted to 20 milliseconds to achieve the fast playback effect. When the synchronization judgment result returns to the synchronous state, the display time of each frame is restored to 40 milliseconds, thereby resuming normal playback;

[0087] The specific implementation steps of audio synchronization during video playback are as follows:

[0088] Set the acceptable synchronization error threshold to AudTolerance_TimeMax = 100 milliseconds, AudTolerance_TimeMin = 50 milliseconds; the hardware audio delay of the slave device is 80 milliseconds, and the audio speed processing algorithm uses the time domain compression algorithm;

[0089] After receiving the audio data from the playback domain, the audio data is continuously sent to the decoder for decoding. The decoder outputs the decoded audio frame data. The playback control module calls the synchronization control module to judge the synchronization status of the decoded audio frame data. The judgment uses parameters such as AudTolerance_TimeMax, synchronization reference time axis time Clock_Time, network transmission delay NetLatency and device delay AudDevice_Latency.

[0090] If it is judged to be synchronized, it will be directly sent to the device for sound playback;

[0091] If the playback domain is ahead by 200 milliseconds, the slow playback synchronization logic is started; Speed_Frctor=0.5, Speed_Frctor=0.8 are set; first, Speed_Frctor=0.5 times the slow playback speed is set, when the playback ahead time difference is 100 milliseconds, Speed_Frctor=0.8 times the slow playback speed is set, when the playback ahead time difference is less than Tolerance_TimeMin, it is judged that the synchronization process is completed, enter the synchronization state and set Speed_Frctor=1.0 to play at normal speed;

[0092] If the playback domain is behind by 200 milliseconds, the fast playback synchronization logic is started; Speed_Frctor=1.2, Speed_Frctor=1.5 are set to two levels, first set Speed_Frctor=1.5 times the speed of fast playback, when the playback lag time difference is 150 milliseconds, set Speed_Frctor=1.2 times the speed of fast playback, when the playback advance time difference is less than Tolerance_TimeMin, it is judged that the synchronization process is completed, enter the synchronization state and set Speed_Frctor=1.0 to play at normal speed;

[0093] The master-slave playback domain is closed and playback ends.

[0094] Through the distributed cross-domain audio and video sharing synchronization method set up above, it is possible to achieve millisecond-level audio synchronization and video frame-level synchronization of the same media content played between multiple independent domain space devices in a heterogeneous network; it does not use any third-party time system, but directly uses the player's playback timestamp, so there is no need to rely on a dedicated time server. The audio synchronization processing uses a multiple-speed processing algorithm to ensure the continuity of the audio, and can achieve smooth and imperceptible synchronization by setting multiple levels of multiple values. The master-slave playback domain performs two-way state synchronization, which can well handle the impact of network jitter on the synchronization effect of the master-slave playback domain.

[0095] It should be understood that the exemplary embodiments described herein are illustrative rather than restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A distributed cross-domain audio and video sharing synchronization system, characterized in that: It includes multiple cross-domain playback domains, the playback domain that initiates audio and video sharing is the main playback domain, the playback domain that accepts the invitation is the slave playback domain, a device in each playback domain is the master device in the domain, the master device in the domain belongs to the first level of the distributed hierarchy, and other devices in each playback domain are slave devices in the domain, the slave devices in the domain belong to the second level of the distributed hierarchy. The main playback domain and the slave playback domain are used to transmit signaling messages through a cross-domain signaling service. The main playback domain master device includes a first data acquisition and analysis module, a first playback control module and a first cross-domain signaling module; the slave playback domain master device includes a second data acquisition and analysis module, a second playback control module, a second cross-domain signaling module and a second synchronization control module.

2. The distributed cross-domain audio and video sharing synchronization system according to claim 1, characterized in that: The first data acquisition and analysis module and the second data acquisition and analysis module are used to request audio and video data, and to analyze the format of audio and video data and to separate audio and video data; the first playback control module and the second playback control module are used to decode and play audio and video data; The first cross-domain signaling module and the second cross-domain signaling module are used to send signaling to the cross-domain signaling service and receive signaling from the cross-domain signaling service, and the cross-domain signaling service is used to transmit signaling messages between master-end devices in the cross-domain domain; The second synchronization control module is used for playback synchronization calculation between master-end devices in a cross-domain domain, and provides reference information for synchronization control of the second playback control module.

3. A distributed cross-domain audio and video sharing synchronization method, characterized in that: Using the distributed cross-domain audio and video sharing system as claimed in claim 1 or 2, the audio and video data comes from a streaming media content platform, comprising the following steps: After the master device in the main playback domain obtains the streaming media playback address of the audio and video data, the address is set to the first data acquisition and parsing module and the second data acquisition and parsing module. The first data acquisition and parsing module and the second data acquisition and parsing module access the streaming media content platform according to the streaming media address and download the streaming media data. At the same time, the basic audio and video information of the downloaded streaming media data is parsed and separated from the audio and video. The first playback control module and the second playback control module request the parsed audio data and video data from the data parsing and acquiring module, send the acquired audio data to the audio decoder for decoding, and send the video data to the video decoder for decoding; after the first playback control module and the second playback control module acquire the decoded audio and video, they synchronously control the audio and video, send the audio to play, and send the video to render; The master device in the main playback domain initiates a simulcast sharing invitation to the master device in the slave playback domain through the cross-domain signaling service. The request carries the playback address and start time of the current streaming media video. The start time is the current playback time of the master device in the main playback domain when the invitation is sent. When the master device in the slave playback domain accepts the invitation, the master device in the slave playback domain initializes the player using the playback address and start time, and requests streaming media data starting from the start time from the media content platform. During the simulcast sharing process, the master device in the main playback domain synchronizes the current playback time as the synchronization reference time SyncRefTime to the master device in the slave playback domain at a certain frequency; When the master device in the slave playback domain successfully starts playing, a synchronization reference timeline is created; the synchronization reference timeline initialization time StartTime is calculated using the start time StartPlayTime transmitted from the main playback domain, the network transmission delay NetLatency and the player initialization time PrepareTime of the master device in the slave playback domain, and the calculation method is: StartTime = StartPlayTime + NetLatency + PrepareTime; during the playback process, the synchronization reference time SyncRefTime sent by the main playback domain is continuously received from the slave playback domain, and the master device in the slave playback domain uses the synchronization reference time and the network transmission delay to calculate the update time UpdateTime of the synchronization reference timeline, and uses the update time to update the synchronization reference timeline, and the update time calculation method is: UpdateTime = SyncRefTime + NetLatency. During the time period when the synchronization reference time is not updated, the synchronization reference timeline automatically runs according to the natural time.

4. The distributed cross-domain audio and video sharing synchronization method according to claim 3, characterized in that: The master playback domain and the slave playback domain perform playback operations and state synchronization through cross-domain signaling services. The synchronization signaling performs two-way communication. The playback operation can be triggered by the master playback domain or by the slave playback domain.

5. The distributed cross-domain audio and video sharing synchronization method according to claim 4, characterized in that: When the main playback domain or the slave playback domain triggers a Seek jump, the master end in the domain sends the Seek message and the Seek time point to other domains, and then the master end player in the domain performs the Seek operation on its own end; other domains immediately execute the Seek operation after receiving the Seek message and the Seek time point; after each domain succeeds in Seek, it synchronizes the Seek completion message to other domains, and resumes playback after confirming that the Seek of other domains is successful and its own Seek is also successful.

6. The distributed cross-domain audio and video sharing synchronization method according to claim 4, characterized in that: When the main playback domain or the slave playback domain triggers the Pause operation, the master end in the domain sends the Pause message to other domains. After receiving the Pause message, other domains immediately respond and perform the Pause operation to achieve the pause effect.

7. The distributed cross-domain audio and video sharing synchronization method according to claim 4, characterized in that: When the main playback domain or the slave playback domain triggers Buffering, the master end in the domain sends the BufferingStart buffering start message to other domains. After receiving the BufferingStart buffering start message, other domains pause playback and wait to receive the BufferingEnd buffering end message before resuming playback.

8. The distributed cross-domain audio and video sharing synchronization method according to claim 4, characterized in that: When the main playback domain or the slave playback domain performs a stream switching operation, the master end in the domain that triggers the stream switching sends the new playback address and start time to the master ends in other domains. After receiving the message, all the master ends in the domains immediately stop playback and start new playback using the new address and start time.

9. The distributed cross-domain audio and video sharing synchronization method according to any one of claims 3 to 8, characterized in that: When cross-domain audio and video are shared simultaneously, the main playback domain provides a synchronization reference time, and the secondary playback domain refers to the synchronization reference time provided by the main playback domain to synchronize audio and video.

10. The distributed cross-domain audio and video sharing synchronization method according to claim 9, characterized in that: After receiving the video data from the playback domain, it is sent to the decoder for decoding. After successful decoding, the current decoded frame is synchronized: After obtaining the first-frame video decoding data from the playback domain, if the synchronous reference timeline has not been started, wait for the synchronous reference timeline to start; if the synchronous reference timeline has already been started, enter the synchronous processing logic. A_TimeVid = Clock_Time + VidTolerance_Time. When VidFrame_Time > A_TimeVid, it indicates that the current video frame is ahead of the video being played in the main playback domain. At this time, the playback rate needs to be slowed down until resynchronization. B_TimeVid = Clock_Time - VidTolerance_Time. When VidFrame_Time < B_TimeVid, it indicates that the current video frame is behind the video being played in the main playback domain. At this time, the playback rate needs to be increased until resynchronization. When B_TimeVid < VidFrame_Time < A_TimeVid, that is, VidFrame_Time is between B_TimeVid and A_TimeVid, it indicates that the current video frame is within the allowable synchronization error range of the video frame being played in the main playback domain and is in a synchronous state. The video is played at the normal playback frame rate. Among them, the video frame timestamp is VidFrame_Time, the synchronous reference timeline time is Clock_Time, the acceptable synchronization error threshold for the video is VidTolerance_Time, the acceptable synchronization error threshold for audio and video can be set according to actual requirements, and the network transmission delay is Network_Latency.

11. The distributed cross-domain audio and video sharing synchronization method according to claim 9, characterized in that: After receiving the audio data from the playback domain, it is sent to the decoder for decoding. After successful decoding, synchronous processing is performed on the current decoded frame. The detailed process of audio synchronization processing is as follows: After obtaining the first-frame audio decoding data from the playback domain, if the synchronous reference timeline has not been started, wait for the synchronous reference timeline to start; if the synchronous reference timeline has already been started, enter the synchronous processing logic. A_TimeAud = Clock_Time + AudTolerance_Time + AudDevice_Latency. When AudFrame_Time > A_TimeAud, it indicates that the current audio frame is ahead of the audio being played in the main playback domain. At this time, the playback rate needs to be slowed down until resynchronization. B_TimeAud = Clock_Time - AudTolerance_Time + AudDevice_Latency. When AudFrame_Time < B_TimeAud, it indicates that the current audio frame is behind the audio being played in the main playback domain. At this time, the playback rate needs to be increased until resynchronization. When B_TimeAud < AudFrame_Time < A_TimeAud, that is, AudFrame_Time is between B_TimeAud and A_TimeAud, it indicates that the current audio frame and the audio frame being played in the main playback domain are within the allowed synchronization error range and are in a synchronized state. The audio can be played at the normal playback rate; Among them, the audio frame timestamp is AudFrame_Time, the time on the synchronization reference timeline is Clock_Time, the acceptable synchronization error threshold for audio is AudTolerance_Time. The acceptable synchronization error thresholds for audio and video can be set according to actual requirements. The network transmission delay is Network_Latency, and the audio delay time of the device hardware is AudDevice_Latency. The audio delay of the device hardware refers to the time consumed from the input of the audio decoded data to the sound being played out from the device sound card.

Images