Cross-domain audio and video sharing synchronization method

Abstract

The invention discloses a cross-domain audio and video sharing synchronization method, which comprises the following steps that: after main end equipment in a main playing domain acquires a local media file storage position, the storage position is set to a first data acquisition and analysis module; the first data acquisition and analysis module reads media data from a local media file to analyze audio and video basic information; the first playing control module requests the analyzed audio data and video data from the first data analysis acquisition module, sends the acquired audio data to a first audio decoder for decoding, and sends the video data to a first video decoder for decoding; after acquiring the decoded audio and video, the first playing control module synchronously controls the audio and video; the master end in the master playing domain initiates a simulcasting sharing invitation to the master end in the slave playing domain through a cross-domain signaling service; and after the master end in the slave playing domain receives the invitation, the master end player in the slave playing domain starts a second data acquisition and analysis module to monitor and receive the audio and video data forwarded by the master playing domain.

Description

Technical Field

[0001] The present invention belongs to the technical field of audio and video playback, and in particular relates to a cross-domain audio and video sharing synchronization method. Background Art

[0002] With the development of technology, the application scenarios of multimedia are changing with each passing day. From the original TV screen, to mobile terminals, to outdoor large screens, and then to the multi-screen integration of the Internet of Everything. With the advancement of network communication technology, emerging application scenarios such as home theater and cockpit entertainment have a technical basis for realization. This multi-screen integration in an independent space has a natural demand for multi-screen simultaneous broadcasting in the field of video playback. For example, in a smart car cockpit, multiple screen devices in different locations can share audio and video content such as movies, TV series, variety shows, and music, which can enhance the riding experience. Furthermore, audio and video content can also be shared and synchronized between multiple independent multi-screen integration spaces across domains, such as watching movies together between cars and watching movies together between home theaters.

[0003] Most media content contains both video and audio. To share media content between multiple spatial devices across domains, it is necessary to share and synchronize video and audio data. One common screen sharing method in existing technologies is screen projection, which is two-screen sharing between a small screen and a large screen. For example, a mobile terminal shares video content to a smart TV. This method can only achieve one-to-one screen projection, and cannot achieve sharing between multiple screen devices or cross-domain sharing. Summary of the invention

[0004] In view of the above problems, the technical problem to be solved by the present invention is to provide a cross-domain audio and video sharing synchronization method for realizing shared playback of audio and video content in a cross-domain multi-screen fusion scenario based on local videos in the main playback domain.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] The present invention provides a cross-domain audio and video sharing synchronization method, which is applied to a cross-domain audio and video sharing system. The cross-domain audio and video sharing system includes multiple cross-domain playback domains, the playback domain that initiates audio and video sharing is used as the main playback domain, the playback domain that accepts the invitation is used as the slave playback domain, a device in each playback domain is used as a master device in the domain, the master device in the domain belongs to the first level of the distributed hierarchy, and other devices in each playback domain are used as slave devices in the domain, the slave devices in the domain belong to the second level of the distributed hierarchy, and the master playback domain and the slave playback domain are transmitted through a cross-domain signaling service. The main playback domain master device includes a first data acquisition and analysis module, a first playback control module and a first cross-domain signaling module; the slave playback domain master device includes a second data acquisition and analysis module, a second playback control module, a second cross-domain signaling module and a second synchronization control module, the cross-domain signaling service is used to transmit signaling messages between master devices in the cross-domain domain, and the cross-domain media forwarding service is used for the master device in the main playback domain to send the audio and video data being played to the master device in the slave playback domain, and the audio and video data are derived from the local media files in the main playback domain, including the following steps:

[0007] After the master device in the main playback domain obtains the storage location of the local media file, it sets the storage location to the first data acquisition and analysis module, and the first data acquisition and analysis module reads the media data from the local media file to analyze the basic information of the audio and video, and at the same time parses and separates the audio and video from the media data;

[0008] The first playback control module requests the parsed audio data and video data from the first data parsing and obtaining module, sends the obtained audio data to the first audio decoder for decoding, and sends the video data to the first video decoder for decoding; after the first playback control module obtains the decoded audio and video, it performs synchronization control on the audio and video, and finally sends the audio to be played and the video to be rendered;

[0009] The master end in the main playback domain initiates a simulcast sharing invitation to the master end in the slave playback domain through the cross-domain signaling service. The request carries the start time StartPlayTime, which is the current playback time of the master end in the main playback domain when the invitation is sent. At the same time, the data forwarding function of the first data parsing and obtaining module is started to send the video data being played to the cross-domain media forwarding service;

[0010] When the host in the playback domain accepts the invitation, the host in the playback domain initializes the player using the start time, and at the same time, the host player in the playback domain starts the second data acquisition and analysis module to monitor and receive the audio and video data forwarded by the main playback domain;

[0011] During the simulcast sharing process, the master end in the main playback domain synchronizes the current playback time as the synchronization reference time SyncRefTime to the master end device in the slave playback domain at a certain frequency;

[0012] When the main end device in the playback domain successfully starts playing, a synchronization reference timeline is created; the synchronization reference timeline initialization time StartTime is calculated using the start time StartPlayTime transmitted from the main playback domain, the network transmission delay NetLatency and the time taken to initialize the main end player in the playback domain PrepareTime, and the calculation method is: StartTime = StartPlayTime + NetLatency + PrepareTime; during the playback process, the synchronization reference time SyncRefTime sent by the main playback domain is continuously received from the playback domain, and the update time UpdateTime of the synchronization reference timeline is calculated from the main end in the playback domain using the synchronization reference time and the network transmission delay, and the synchronization reference timeline is updated using the update time, and the update time is calculated as: UpdateTime = SyncRefTime + NetLatency. During the time period when the synchronization reference time is not updated, the synchronization reference timeline automatically runs according to the natural time.

[0013] In a possible implementation, the main playback domain and the slave playback domain perform playback operations and state synchronization through a cross-domain signaling service, and the synchronization signaling communicates bidirectionally. The playback operation can be triggered by the main playback domain or by the playback domain.

[0014] In one possible implementation, when a Seek jump is triggered in the main playback domain or the slave playback domain, the master end in the domain sends the Seek message and the Seek time point to other domains; the Seek operation of the actual video data is performed by the main playback domain, which first clears the cache of unplayed data, and then reads the video data at the corresponding position of the media file according to the Seek time point; the Seek operations in other slave playback domains will first clear the cache of unplayed data, and then wait to receive the audio and video data after the Seek in the main playback domain, and continue playing using the audio and video data after the Seek in the main playback domain.

[0015] In a possible implementation, when the main playback domain or the slave playback domain triggers a Pause operation, the master end in the domain sends a Pause message to other domains, and other domains immediately respond to and perform a Pause operation after receiving the Pause message, thereby achieving a pause effect.

[0016] In a possible implementation, when a stream switching operation is performed in the primary playback domain or the secondary playback domain, the domain that triggers the stream switching will send the stream switching message to all other domains; the actual stream switching operation is performed by the primary playback domain. The primary playback domain first stops playing the current stream, and then retrieves new local media file data for playback; after receiving the stream switching message, the secondary playback domain first resets the player, and then waits to receive the audio and video data after the stream switching in the primary playback domain, and uses the audio and video data after the stream switching in the primary playback domain for playback.

[0017] In a possible implementation, when sharing cross-domain audio and video simultaneous playback, the primary playback domain provides a synchronization reference time, and the secondary playback domain synchronizes audio and video with reference to the synchronization reference time provided by the primary playback domain.

[0018] In a possible implementation, after receiving the video data, the secondary playback domain sends it to the decoder for decoding. After successful decoding, synchronization processing is performed on the current decoded frame:

[0019] After the secondary playback domain obtains the first-frame video decoding data, if the synchronization reference time axis has not been started, it waits for the synchronization reference time axis to start. If the synchronization reference time axis has already been started, it enters the synchronization processing logic;

[0020] A_TimeVid = Clock_Time + VidTolerance_Time. When VidFrame_Time > A_TimeVid, it indicates that the current video frame is ahead of the video being played in the primary playback domain. At this time, the playback rate needs to be slowed down until resynchronization;

[0021] B_TimeVid = Clock_Time - VidTolerance_Time. When VidFrame_Time < B_TimeVid, it indicates that the current video frame is behind the video being played in the primary playback domain. At this time, the playback rate needs to be increased until resynchronization;

[0022] When B_TimeVid < VidFrame_Time < A_TimeVid, that is, VidFrame_Time is between B_TimeVid and A_TimeVid, it indicates that the current video frame is within the allowable synchronization error range of the video frame being played in the primary playback domain, and is in a synchronized state. The video is played at the normal playback frame rate;

[0023] Among them, the video frame timestamp is VidFrame_Time, the synchronization reference time axis time is Clock_Time, the acceptable synchronization error threshold for video is VidTolerance_Time, the acceptable synchronization error threshold for audio and video can be set according to actual requirements, and the network transmission delay is Network_Latency.

[0024] In a possible implementation, after receiving audio data from the playback domain, it is sent to a decoder for decoding. After successful decoding, synchronization processing is performed on the current decoded frame. The detailed process of audio synchronization is as follows:

[0025] After obtaining the first-frame audio decoding data from the playback domain, if the synchronization reference timeline has not been started, wait for the synchronization reference timeline to start. If the synchronization reference timeline has already started, enter the synchronization processing logic;

[0026] A_TimeAud = Clock_Time + AudTolerance_Time + AudDevice_Latency. When AudFrame_Time > A_TimeAud, it indicates that the current audio frame is ahead of the audio being played in the main playback domain. At this time, the playback rate needs to be slowed down until resynchronization;

[0027] B_TimeAud = Clock_Time - AudTolerance_Time + AudDevice_Latency. When AudFrame_Time < B_TimeAud, it indicates that the current audio frame is behind the audio being played in the main playback domain. At this time, the playback rate needs to be increased until resynchronization;

[0028] When B_TimeAud < AudFrame_Time < A_TimeAud, that is, AudFrame_Time is between B_TimeAud and A_TimeAud, it indicates that the current audio frame is within the allowable synchronization error range of the audio frame being played in the main playback domain and is in a synchronized state. The audio can be played at the normal playback rate;

[0029] Among them, the audio frame timestamp is AudFrame_Time, the synchronization reference timeline time is Clock_Time, the acceptable synchronization error threshold for audio is AudTolerance_Time. The acceptable synchronization error threshold for audio and video can be set according to actual needs. The network transmission delay is Network_Latency, and the device hardware audio delay time is AudDevice_Latency. The device hardware audio delay refers to the time consumed from the input of audio decoding data to the sound output from the device sound card.

[0030] The present invention has the following beneficial effects:

[0031] (1) Without using any third-party time system, directly using the playback time (PTS) built into the video data, so that the unity of the time system can be ensured without relying on a dedicated time server;

[0032] (2) The master and slave playback domains perform two-way state synchronization, which can effectively handle the impact of network jitter on the synchronization effect of the master and slave playback domains;

[0033] (3) Through the cross-domain media forwarding service, the local media files of the main playback domain can be shared simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of a cross-domain audio and video sharing synchronization method according to an embodiment of the present invention;

[0035] Figure 2 The present invention is a schematic structural diagram of a cross-domain audio and video sharing synchronization system in a cross-domain audio and video sharing synchronization method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Reference Figure 1 , which is a flowchart of a cross-domain audio and video sharing synchronization method according to an embodiment of the present invention, is applied to a cross-domain audio and video sharing system, such as Figure 2 As shown, the cross-domain audio and video sharing system includes multiple cross-domain playback domains, the playback domain that initiates audio and video sharing is used as the main playback domain, the playback domain that accepts the invitation is used as the slave playback domain, and a device in each playback domain is used as a master device in the domain, and the master device in the domain belongs to the first level of the distributed hierarchy. Other devices in each playback domain are slave devices in the domain, and the slave devices in the domain belong to the second level of the distributed hierarchy. The main playback domain and the slave playback domain are transmitted through a cross-domain signaling service to transmit signaling messages. The main playback domain master device includes a first data acquisition and analysis module, a first playback control module, and a first cross-domain signaling module; the slave playback domain master device includes a second data acquisition and analysis module, a second playback control module, a second cross-domain signaling module, and a second synchronization control module. The cross-domain signaling service is used to transmit signaling messages between master devices in the cross-domain domain. The cross-domain media forwarding service is used for the master device in the main playback domain to send the video data being played to the master device in the slave playback domain. The audio and video data are derived from the local media files in the main playback domain, including the following steps:

[0038] S101, after the master device in the main playback domain obtains the storage location of the local media file, the storage location is set to the first data acquisition and analysis module, and the first data acquisition and analysis module reads media data from the local media file to analyze basic audio and video information, and simultaneously parses and separates audio and video from the media data;

[0039] S102, the first playback control module requests the parsed audio data and video data from the first data parsing and obtaining module, sends the obtained audio data to the first audio decoder for decoding, and sends the video data to the first video decoder for decoding; after the first playback control module obtains the decoded audio and video, it performs synchronization control on the audio and video, and finally sends the audio to be played and sends the video to be rendered;

[0040] S103, the master end in the main playback domain initiates a simulcast sharing invitation to the master end in the slave playback domain through the cross-domain signaling service, the request carries the start time StartPlayTime, the start time is the current playback time of the master end in the main playback domain when the invitation is sent, and at the same time, the data forwarding function of the first data parsing and obtaining module is started to send the video data being played to the cross-domain media forwarding service;

[0041] S104, when the master end in the playback domain accepts the invitation, the master end in the playback domain initializes the player using the start time, and at the same time, the master end player in the playback domain starts a second data acquisition and analysis module to monitor and receive the audio and video data forwarded from the main playback domain;

[0042] S105, during the simulcast sharing process, the master end in the main playback domain synchronizes the current playback time as the synchronization reference time SyncRefTime to the master end device in the slave playback domain at a certain frequency;

[0043] S106, when the master device in the playback domain successfully starts playing, a synchronization reference time axis is created; the synchronization reference time axis initialization time StartTime is calculated using the start time StartPlayTime transmitted from the main playback domain, the network transmission delay NetLatency and the time taken to initialize the master player in the playback domain PrepareTime, and the calculation method is: StartTime = StartPlayTime + NetLatency + PrepareTime; during the playback process, the synchronization reference time SyncRefTime sent from the main playback domain is continuously received from the playback domain, and the master end in the playback domain uses the synchronization reference time and the network transmission delay to calculate the update time UpdateTime of the synchronization reference time axis, and the update time is used to update the synchronization reference time axis, and the update time calculation method is: UpdateTime = SyncRefTime + NetLatency, and during the time period when the synchronization reference time is not updated, the synchronization reference time axis automatically runs according to the natural time.

[0044] In one embodiment of the present invention, the operation synchronization counter state synchronization logic of the master playback domain and the slave playback domain is as follows:

[0045] The main playback domain and the slave playback domain perform playback operations and status synchronization through cross-domain signaling services. The synchronization signaling communicates bidirectionally, and the playback operation can be triggered by the main playback domain or by the playback domain.

[0046] When the main playback domain or the slave playback domain triggers a Seek jump, the master end in the domain sends the Seek message and the Seek time point to other domains; the Seek operation of the actual video data is performed by the main playback domain, which first clears the cache of unplayed data, and then reads the video data at the corresponding position of the media file according to the Seek time point; the Seek operation of other slave playback domains will first clear the cache of unplayed data, and then wait to receive the audio and video data after the Seek of the main playback domain, and continue to play using the audio and video data after the Seek of the main playback domain.

[0047] When the main playback domain or the slave playback domain triggers the Pause operation, the master end in the domain sends the Pause message to other domains. After receiving the Pause message, other domains immediately respond and perform the Pause operation, thereby achieving the pause effect.

[0048] If the main playback domain or the slave playback domain performs a stream switching operation, the domain that triggers the stream switching will send a stream switching message to all other domains; the actual stream switching operation is performed by the main playback domain, which first stops the playback of the current stream, and then re-acquires new local media file data for playback; after receiving the stream switching message, the slave playback domain first resets the player, and then waits to receive the audio and video data after the main playback domain switches the stream, and uses the audio and video data after the main playback domain switches the stream for playback.

[0049] In one embodiment of the present invention, when cross-domain audio and video are simultaneously broadcast and shared, the main playback domain provides a synchronization reference time, and the slave playback domain performs audio and video synchronization with reference to the synchronization reference time provided by the main playback domain.

[0050] Furthermore, in one embodiment of the present invention, after receiving the video data from the playback domain, the video data is sent to the decoder for decoding. After the decoding is successful, the video frame synchronization process of the current decoded frame is performed as follows:

[0051] After getting the first frame of video decoding data from the playback domain, if the synchronization reference time axis has not been started, wait for the synchronization reference time axis to be started. If the synchronization reference time axis has been started, enter the synchronization processing logic;

[0052] A_TimeVid = Clock_Time + VidTolerance_Time. When VidFrame_Time > A_TimeVid, it indicates that the current video frame is ahead of the video being played in the main playback domain. At this time, the playback rate needs to be slowed down until resynchronization occurs.

[0053] B_TimeVid = Clock_Time - VidTolerance_Time. When VidFrame_Time < B_TimeVid, it indicates that the current video frame is behind the video being played in the main playback domain. At this time, the playback rate needs to be increased until resynchronization occurs.

[0054] When B_TimeVid < VidFrame_Time < A_TimeVid, that is, VidFrame_Time is between B_TimeVid and A_TimeVid, it indicates that the current video frame is within the allowable synchronization error range of the video frame being played in the main playback domain and is in a synchronized state. The video is played at the normal playback frame rate.

[0055] Among them, VidFrame_Time is the video frame timestamp, Clock_Time is the time on the synchronization reference timeline, and VidTolerance_Time is the allowable synchronization error threshold for the video. The allowable synchronization error threshold for the video can be set according to actual requirements.

[0056] Furthermore, in an embodiment of the present invention, after receiving audio data from the playback domain, it is sent to the decoder for decoding. After successful decoding, synchronization processing is performed on the current decoded frame. The detailed process of audio synchronization processing is as follows:

[0057] After obtaining the first frame of audio decoded data from the playback domain, if the synchronization reference timeline has not been started, wait for the synchronization reference timeline to start. If the synchronization reference timeline has already started, enter the synchronization processing logic.

[0058] A_TimeAud = Clock_Time + AudTolerance_Time + AudDevice_Latency. When AudFrame_Time > A_TimeAud, it indicates that the current audio frame is ahead of the audio being played in the main playback domain. At this time, the playback rate needs to be slowed down until resynchronization occurs.

[0059] B_TimeAud = Clock_Time - AudTolerance_Time + AudDevice_Latency. When AudFrame_Time < B_TimeAud, it indicates that the current audio frame is behind the audio being played in the main playback domain. At this time, the playback rate needs to be increased until resynchronization occurs.

[0060] When B_TimeAud < AudFrame_Time < A_TimeAud, that is, AudFrame_Time is between B_TimeAud and A_TimeAud, it indicates that the current audio frame and the audio frame being played in the main playback domain are within the allowable synchronization error range and are in a synchronized state. The audio can be played at the normal playback rate;

[0061] Among them, the audio frame timestamp is AudFrame_Time, the time on the synchronization reference timeline is Clock_Time, the acceptable synchronization error threshold for audio is AudTolerance_Time, and the acceptable synchronization error threshold for audio and video can be set according to actual requirements. The network transmission delay is Network_Latency, and the audio delay time of the device hardware is AudDevice_Latency. The audio delay of the device hardware refers to the time consumed from the input of the audio decoded data to the sound being played from the device sound card.

[0062] For the processing of audio playing ahead and playing behind, the general processing method is to wait when it is ahead and drop frames to catch up when it is behind. However, the characteristic of audio is its continuity. Waiting and dropping frames to catch up will damage the continuity of the audio, making the audio sound intermittent and not smooth. In order to achieve the synchronization effect while maintaining the continuity of the audio, in an embodiment of the present invention, the following strategy is adopted to implement the audio synchronization processing in the case of playing ahead and behind:

[0063] First, refine the audio synchronization error threshold. The audio synchronization error threshold is subdivided into AudTolerance_TimeMax and AudTolerance_TimeMin, where AudTolerance_TimeMax > AudTolerance_TimeMin. AudTolerance_TimeMax is used to determine whether it is in a synchronized state; AudTolerance_TimeMin is used to determine whether synchronization has been re - achieved during the synchronization process to end the synchronization process. The advantage of this processing is to prevent the critical situation of frequent synchronization and desynchronization state transitions in the case of only one threshold; this threshold is not fixed and can be configured according to the actual situation, as long as AudTolerance_TimeMax > AudTolerance_TimeMin is ensured. The audio synchronization processing introduces a synchronization judgment threshold interval [AudTolerance_TimeMin, AudTolerance_TimeMax]. AudTolerance_TimeMax is used to determine whether it is desynchronized, and AudTolerance_TimeMin is used to determine whether the synchronization process is completed, which can eliminate the frequent synchronization and desynchronization state transition phenomena at the critical position of synchronization judgment.

[0064] Secondly, for the processing of audio playback being ahead or behind, considering maintaining the continuity of the audio, an audio speed - change processing algorithm is adopted. While ensuring the sound speed - change, the sampling rate, fundamental frequency, and formants of the voice do not change, so as to achieve the effect of changing speed without changing pitch. In addition, for the synchronization processing, multiple different levels of speed - factor values can be defined according to the severity of desynchronization to achieve a progressive processing during the synchronization process, making the audio change relatively subtly during the processing and reducing the user's perception; the speed - factor value is defined as Speed_Factor.

[0065] When the playback is ahead, AudFrame_Time > A0_TimeAud, and A0_TimeAud = Clock_Time + AudTolerance_TimeMax + AudDevice_Latency; at this time, configure Speed_Frctor < 1.0, and achieve synchronization by slowing down the playback rate. During the synchronization process, when AudFrame_Time < A1_TimeAud, and A1_TimeAud = Clock_Time + AudTolerance_TimeMin + AudDevice_Latency, it is determined that synchronization has been re - achieved, end the slow - play synchronization process, and resume normal - rate playback with Speed_Frctor = 1.0;

[0066] When the playback lags behind, AudFrame_Time < B0_TimeAud, where B0_TimeAud = Clock_Time - AudTolerance_TimeMax + AudDevice_Latency; at this time, configure Speed_Frctor > 1.0 to achieve synchronization by accelerating the playback rate. During the synchronization process, when AudFrame_Time > B1_TimeAud, where B1_TimeAud = Clock_Time - AudTolerance_TimeMin + AudDevice_Latency, it is determined that synchronization has been re - achieved, and the fast - playback synchronization process ends, and the normal playback rate of Speed_Frctor = 1.0 is restored.

[0067] During the synchronization process, according to the actual severity of the desynchronization, multiple levels of Speed_Frctor can be customized to make the synchronization process progressive, resulting in smoother playback during the synchronization process, less perceptible to the user, and better user experience.

[0068] In a specific application example, the specific implementation steps of the cross - domain audio - video synchronous sharing method of an embodiment of the present invention are as follows:

[0069] In the specific implementation steps, the cross - domain signaling service uses the standard Socket protocol to implement sending and listening.

[0070] The master - end device in the main playback domain invites the master - end device in the slave playback domain for synchronous sharing, and sends the streaming media playback address and the start - playback time to the slave playback domain when inviting.

[0071] After the master - end device in the slave playback domain accepts the invitation, it initializes the player in the slave playback domain using the cross - domain media forwarding service address and the start - playback time. The player will request the streaming media data at the start - playback time point from the cross - domain media forwarding service. At the same time, it starts the second cross - domain signaling module to listen for signaling messages. When starting playback in the slave playback domain, a synchronous reference timeline needs to be created first. The initialization time StartTime of the synchronous reference timeline is calculated using the start - playback time StartPlayTime transmitted from the main playback domain, the network transmission delay NetLatency, and the initialization time PrepareTime of the master - end player in the slave playback domain. The calculation method is: StartTime = StartPlayTime + NetLatency + PrepareTime.

[0072] During the simulcast sharing process, the main playback domain synchronizes the current playback time as the synchronization reference time SyncRefTime to the slave playback domain at a frequency of every 1000 milliseconds; during the playback process, the slave playback domain will continue to receive the synchronization reference time SyncRefTime sent by the main playback domain, and the slave playback domain uses the synchronization reference time and network transmission delay to calculate the update time UpdateTime of the synchronization reference time axis, and uses the update time to update the synchronization reference time axis. The update time calculation method is: UpdateTime = SyncRefTime + NetLatency. During the time period when the synchronization reference time is not updated, the synchronization reference time axis will automatically run according to the natural time;

[0073] When the main playback domain or the slave playback domain triggers a Seek jump, the master end in the domain sends the Seek message and the Seek time point to other domains; the Seek operation of the actual video data is performed by the main playback domain, which first clears the cache of unplayed data, and then reads the video data at the corresponding position of the media file according to the Seek time point; the Seek operation of other slave playback domains will first clear the cache of unplayed data, and then wait to receive the audio and video data after the Seek of the main playback domain, and continue to play using the audio and video data after the Seek of the main playback domain;

[0074] When the main playback domain or the slave playback domain triggers the Pause operation, the master end in the domain sends the Pause message to other domains. After receiving the Pause message, other domains immediately respond and perform the Pause operation, thereby achieving the Pause effect;

[0075] When the main playback domain or the slave playback domain triggers Buffering, the master end in the domain sends the BufferingStart buffering start message to other domains. After receiving the BufferingStart buffering start message, other domains pause playback and wait for the BufferingEnd buffering end message to resume playback.

[0076] When the main playback domain or the slave playback domain performs a stream switching operation, the domain that triggers the stream switching will send a stream switching message to all other domains; the actual stream switching operation is performed by the main playback domain, which first stops the playback of the current stream, and then re-acquires new local media file data for playback; after receiving the stream switching message, the slave playback domain first resets the player, and then waits to receive the audio and video data after the main playback domain switches the stream, and uses the audio and video data after the main playback domain switches the stream for playback.

[0077] The specific implementation steps of video synchronization during video playback in the playback domain are as follows:

[0078] The acceptable synchronization error threshold VidTolerance_Time is set to twice the video frame display time. In this embodiment, the video frame rate is 30, so the display time of each frame is 33 milliseconds, and the acceptable synchronization error VidTolerance_Time = 33*2 = 66 milliseconds. That is, this embodiment can control the synchronization error within 2 video frames, that is, within 66 milliseconds;

[0079] After receiving the video data from the playback domain, the video data is continuously sent to the decoder for decoding. The decoder outputs the decoded video frame data. The playback control module calls the synchronization control module to judge the synchronization status of the decoded video frame data. The judgment uses parameters such as Tolerance_Time, synchronization reference time axis time Clock_Time, and network transmission delay NetLatency.

[0080] If it is determined to be synchronized, it is directly sent to the device for video rendering;

[0081] If the playback domain synchronization is judged to be 200 milliseconds ahead, it is necessary to synchronize through slow playback. When playing at normal speed, the display time of each frame is 33 milliseconds. If slow playback is needed at this time, adjust the display time of each frame to 50 milliseconds to achieve the slow playback effect. When the synchronization judgment result returns to the synchronized state, restore the display time of each frame to 33 milliseconds, thereby resuming normal playback;

[0082] If the playback domain synchronization is judged to be 200 milliseconds behind, it is necessary to perform a catch-up synchronization process through fast playback. When playing at a normal rate, the display time of each frame is 33 milliseconds. If fast playback is required, the display time of each frame is adjusted to 17 milliseconds to achieve the fast playback effect. When the synchronization judgment result returns to the synchronous state, the display time of each frame is restored to 33 milliseconds, thereby resuming normal playback;

[0083] The specific implementation steps of audio synchronization during video playback are as follows:

[0084] Set the acceptable synchronization error threshold to AudTolerance_TimeMax = 100 milliseconds, AudTolerance_TimeMin = 50 milliseconds; the hardware audio delay of the slave device is 80 milliseconds, and the audio speed processing algorithm uses the time domain compression algorithm;

[0085] After receiving the audio data from the playback domain, the audio data is continuously sent to the decoder for decoding. The decoder outputs the decoded audio frame data. The playback control module calls the synchronization control module to judge the synchronization status of the decoded audio frame data. The judgment uses parameters such as AudTolerance_TimeMax, synchronization reference time axis time Clock_Time, network transmission delay NetLatency and device delay AudDevice_Latency.

[0086] If it is judged to be synchronized, it will be directly sent to the device for sound playback;

[0087] If it is 250 milliseconds ahead of the playback domain, the slow playback synchronization logic is started; Speed_Frctor=0.5, Speed_Frctor=0.75 are set; first, Speed_Frctor=0.5 times the slow playback speed is set, when the playback ahead time difference is 100 milliseconds, Speed_Frctor=0.75 times the slow playback speed is set, when the playback ahead time difference is less than Tolerance_TimeMin, it is judged that the synchronization process is completed, enter the synchronization state and set Speed_Frctor=1.0 to play at normal speed;

[0088] If the playback domain is behind by 250 milliseconds, the fast playback synchronization logic is started; Speed_Frctor=1.25, Speed_Frctor=1.5 are set to two levels. First, Speed_Frctor=1.5 times the speed of fast playback is set. When the playback lag time difference is 150 milliseconds, Speed_Frctor=1.25 times the speed of fast playback is set. When the playback advance time difference is less than Tolerance_TimeMin, it is judged that the synchronization process is completed, and the synchronization state is entered and Speed_Frctor=1.0 is set to play at normal speed;

[0089] The master-slave playback domain is closed and playback ends.

[0090] Through the cross-domain audio and video sharing synchronization method set above, it is possible to achieve millisecond-level audio synchronization and video frame-level synchronization of the same media content played between multiple independent domain space devices in a heterogeneous network; it does not use any third-party time system, but directly uses the player's playback timestamp, so there is no need to rely on a dedicated time server. The audio synchronization processing uses a multiple-speed processing algorithm to ensure the continuity of the audio, and can achieve smooth and imperceptible synchronization by setting multiple levels of multiple values. The master-slave playback domain performs two-way state synchronization, which can well handle the impact of network jitter on the synchronization effect of the master-slave playback domain.

[0091] It should be understood that the exemplary embodiments described herein are illustrative rather than restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A cross-domain audio and video sharing synchronization method, characterized in that: The invention is applied to a cross-domain audio and video sharing system, wherein the cross-domain audio and video sharing system includes multiple cross-domain playback domains, the playback domain that initiates audio and video sharing is the main playback domain, the playback domain that accepts the invitation is the slave playback domain, a device in each playback domain is the master device in the domain, the master device in the domain belongs to the first level of the distributed hierarchy, and the other devices in each playback domain are slaves in the domain, the slave devices in the domain belong to the second level of the distributed hierarchy, and the main playback domain and the slave playback domain are transmitted through a cross-domain signaling service to transmit signaling messages, the main playback domain master device includes a first data acquisition and analysis module, a first playback control module and a first cross-domain signaling module; the slave playback domain master device includes a second data acquisition and analysis module, a second playback control module, a second cross-domain signaling module and a second synchronization control module, the cross-domain signaling service is used to transmit signaling messages between the master devices in the cross-domain domain, and the cross-domain media forwarding service is used for the master device in the main playback domain to send the audio and video data being played to the master device in the slave playback domain, and the audio and video data are derived from the local media files in the main playback domain, and the invention includes the following steps: After the master device in the main playback domain obtains the storage location of the local media file, it sets the storage location to the first data acquisition and analysis module, and the first data acquisition and analysis module reads the media data from the local media file to analyze the basic information of the audio and video, and at the same time parses and separates the audio and video from the media data; The first playback control module requests the parsed audio data and video data from the first data parsing and obtaining module, sends the obtained audio data to the first audio decoder for decoding, and sends the video data to the first video decoder for decoding; after the first playback control module obtains the decoded audio and video, it performs synchronization control on the audio and video, and finally sends the audio to be played and the video to be rendered; The master end in the main playback domain initiates a simulcast sharing invitation to the master end in the slave playback domain through the cross-domain signaling service. The request carries the start time StartPlayTime, which is the current playback time of the master end in the main playback domain when the invitation is sent. At the same time, the data forwarding function of the first data parsing and obtaining module is started to send the video data being played to the cross-domain media forwarding service; When the host in the playback domain accepts the invitation, the host in the playback domain initializes the player using the start time, and at the same time, the host player in the playback domain starts the second data acquisition and analysis module to monitor and receive the audio and video data forwarded by the main playback domain; During the simulcast sharing process, the master end in the main playback domain synchronizes the current playback time as the synchronization reference time SyncRefTime to the master end device in the slave playback domain at a certain frequency; When the main end device in the playback domain successfully starts playing, a synchronization reference timeline is created; the synchronization reference timeline initialization time StartTime is calculated using the start time StartPlayTime transmitted from the main playback domain, the network transmission delay NetLatency and the time taken to initialize the main end player in the playback domain PrepareTime, and the calculation method is: StartTime = StartPlayTime + NetLatency + PrepareTime; during the playback process, the synchronization reference time SyncRefTime sent by the main playback domain is continuously received from the playback domain, and the update time UpdateTime of the synchronization reference timeline is calculated from the main end in the playback domain using the synchronization reference time and the network transmission delay, and the synchronization reference timeline is updated using the update time, and the update time is calculated as: UpdateTime = SyncRefTime + NetLatency. During the time period when the synchronization reference time is not updated, the synchronization reference timeline automatically runs according to the natural time.

2. The cross-domain audio and video sharing synchronization method according to claim 1, characterized in that: The master playback domain and the slave playback domain perform playback operations and state synchronization through cross-domain signaling services. The synchronization signaling communicates bidirectionally, and the playback operation can be triggered by the master playback domain or by the slave playback domain.

3. The cross-domain audio and video sharing synchronization method according to claim 1, characterized in that: When the main playback domain or the slave playback domain triggers a Seek jump, the master end in the domain sends the Seek message and the Seek time point to other domains; the Seek operation of the actual video data is performed by the main playback domain, which first clears the cache of unplayed data, and then reads the video data at the corresponding position of the media file according to the Seek time point; the Seek operation of other slave playback domains will first clear the cache of unplayed data, and then wait to receive the audio and video data after the Seek of the main playback domain, and continue to play using the audio and video data after the Seek of the main playback domain.

4. The cross-domain audio and video sharing synchronization method according to claim 1, characterized in that: When the main playback domain or the slave playback domain triggers the Pause operation, the master end in the domain sends the Pause message to other domains. After receiving the Pause message, other domains immediately respond and perform the Pause operation, thereby achieving the pause effect.

5. The cross-domain audio and video sharing synchronization method according to claim 1, characterized in that: If the main playback domain or the slave playback domain performs a stream switching operation, the domain that triggers the stream switching sends a stream switching message to all other domains. The actual stream switching operation is performed by the main playback domain, which first stops the playback of the current stream and then re-acquires new local media file data for playback. After receiving the stream switching message from the playback domain, the player is reset first, and then waits to receive the audio and video data after the main playback domain switches the stream, and uses the audio and video data after the main playback domain switches the stream for playback.

6. The cross-domain audio and video sharing synchronization method according to any one of claims 1 to 5, characterized in that: When cross-domain audio and video are shared simultaneously, the main playback domain provides a synchronization reference time, and the secondary playback domain refers to the synchronization reference time provided by the main playback domain to synchronize audio and video.

7. The cross-domain audio and video sharing synchronization method according to claim 6, characterized in that: After receiving the video data from the playback domain, it is sent to the decoder for decoding. After successful decoding, the current decoded frame is synchronized: After getting the first frame of video decoding data from the playback domain, if the synchronization reference time axis has not been started, wait for the synchronization reference time axis to be started. If the synchronization reference time axis has been started, enter the synchronization processing logic; A_TimeVid = Clock_Time + VidTolerance_Time. When VidFrame_Time > A_TimeVid, it indicates that the current video frame is ahead of the video being played in the main playback domain. At this time, the playback rate needs to be slowed down until resynchronization occurs. B_TimeVid = Clock_Time - VidTolerance_Time. When VidFrame_Time < B_TimeVid, it indicates that the current video frame is behind the video being played in the main playback domain. At this time, the playback rate needs to be increased until resynchronization occurs. When B_TimeVid < VidFrame_Time < A_TimeVid, that is, VidFrame_Time is between B_TimeVid and A_TimeVid, it indicates that the current video frame is within the allowable synchronization error range of the video frame being played in the main playback domain and is in a synchronized state. The video is played at the normal playback frame rate. Among them, the video frame timestamp is VidFrame_Time, the synchronization reference time axis time is Clock_Time, the acceptable synchronization error threshold for the video is VidTolerance_Time, the acceptable synchronization error threshold for audio and video can be set according to actual requirements, and the network transmission delay is Network_Latency.

8. The cross-domain audio and video sharing synchronization method according to claim 6, characterized in that: After receiving the audio data from the playback domain, it is sent to the decoder for decoding. After successful decoding, synchronization processing is performed on the current decoded frame. The detailed process of audio synchronization processing is as follows: After obtaining the first frame of audio decoded data from the playback domain, if the synchronization reference time axis has not been started, wait until the synchronization reference time axis starts. If the synchronization reference time axis has already started, enter the synchronization processing logic. A_TimeAud = Clock_Time + AudTolerance_Time + AudDevice_Latency. When AudFrame_Time > A_TimeAud, it indicates that the current audio frame is ahead of the audio being played in the main playback domain. At this time, the playback rate needs to be slowed down until resynchronization occurs. B_TimeAud = Clock_Time - AudTolerance_Time + AudDevice_Latency. When AudFrame_Time < B_TimeAud, it indicates that the current audio frame is behind the audio being played in the main playback domain. At this time, the playback rate needs to be increased until resynchronization occurs. When B_TimeAud < AudFrame_Time < A_TimeAud, that is, AudFrame_Time is between B_TimeAud and A_TimeAud, it indicates that the current audio frame is within the allowable synchronization error range of the audio frame being played in the main playback domain and is in a synchronized state. The audio can be played at the normal playback rate. The audio frame timestamp is AudFrame_Time, the synchronization reference timeline time is Clock_Time, the acceptable synchronization error threshold of audio is AudTolerance_Time, the acceptable synchronization error threshold of audio and video can be set according to actual needs, the network transmission delay is Network_Latency, and the device hardware audio delay time is AudDevice_Latency, where the device hardware audio delay refers to the time consumed from the audio decoding data input device to the sound being broadcast from the device sound card.

Images