A method, system, device and storage medium for synchronizing multiple video streams

Abstract

This invention discloses a method, system, apparatus, and storage medium for synchronizing multiple video streams. The method includes: acquiring an index file and media segment files sent by a streaming media server; wherein the index file includes reference time offset information of the first frame image of each media segment file, and the media segment files include the network latency time of the current video stream from the streaming end to the streaming media server; determining the first media segment file for initial playback of each video stream, and the reference time offset value of the first frame image of the first media segment file; determining the reference time offset value of the current frame of each video stream according to a preset time interval; and performing synchronization operations on the normally playing video streams based on the reference time offset value of the current frame of each video stream; the synchronization operations include refreshing, fast forwarding, or pausing. This invention enables the synchronization of multiple video streams in live streaming scenarios, and each video stream can be controlled independently, making it widely applicable in the field of video processing technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, and in particular to a multi-path video synchronization method, system, device and storage medium. BACKGROUND

[0002] As a mature, stable and efficient audio and video transmission protocol, HLS (HTTP Live Streaming) has a wide range of application fields and superior performance characteristics, and plays an increasingly important role in digital entertainment and information exchange, providing better services and experiences. HLS can be applied to various scenarios such as video live streaming, on-demand, online education, remote conferencing, etc., providing clear and smooth audio-visual effects for users, and also facilitating the publication and management of content providers.

[0003] In the related art, there are two modes for multi-path HLS video stream synchronization. Traditional mode 1: multi-path HLS video stream synchronization is achieved by merging multiple video streams into one video stream on a streaming media server. This approach makes it impossible for users to individually operate certain video streams, such as enlarging, reducing or changing the page layout of certain video streams, and significantly increases the transcoding pressure on the streaming media server. Traditional mode 2: the playback progress of video files that have already been generated and ensure that the first frame images of multiple videos have the same source time, such as MP4 and other on-demand files, cannot meet the needs of various live streaming scenarios, including HLS live streaming. SUMMARY

[0004] Therefore, the embodiments of the present application provide a multi-path video synchronization method, system, device and storage medium, which can synchronize multiple video streams in a live streaming scenario and individually control each video stream.

[0005] In one aspect, the embodiments of the present application provide a multi-path video synchronization method applied to a playback end, comprising the following steps:

[0006] Obtaining an index file and a media segment file sent by a streaming media server; wherein the index file includes reference time offset value information of the first frame image of each media segment file, and the media segment file includes the network delay time of the current video stream from the streaming end to the streaming media server;

[0007] Determining the first media segment file for initial playback of each video stream according to the current time position of the current video stream playback, the relative time relationship between the first frame image and the last frame image of each media segment file, and obtaining the reference time offset value of the first frame image of the first media segment file for initial playback of each video stream from the index file.

[0008] According to a preset time interval, the reference time offset value of the current frame of each video stream is determined based on the reference time offset value of the first frame of the first media segment file of each video stream during initial playback, the current time of each video stream, and the time of the first frame of the first media segment file of each video stream during initial playback.

[0009] According to the preset time interval, the maximum reference time offset, minimum reference time offset, and average reference time offset are calculated based on the reference time offset of the current frame of each video stream. The video stream being played normally is synchronized based on the difference between the maximum reference time offset and the minimum reference time offset, or the video stream being played normally is synchronized based on the relationship between the average reference time offset and the reference time offset of the current frame of each video stream. The synchronization operation includes refreshing, fast forwarding, or pausing.

[0010] Optionally, determining the first media segment file for initial playback of each video stream based on the current time position of the current video stream playback and the relative time relationship between the first and last frames of each media segment file specifically includes:

[0011] The duration of each media segment file in each video stream is determined by the relative temporal relationship between the first and last frames of each media segment file in each video stream.

[0012] The duration of each media segment file in each video stream is accumulated in chronological order to obtain the accumulated time of each media segment file in each video stream.

[0013] The current time position of each current video stream is matched with the accumulated time of each video stream to determine the first media segment file for the initial playback of each video stream.

[0014] Optionally, determining the reference time offset value of the current frame of each video stream based on the reference time offset value of the first frame of the first media segment file initially played by each video stream, the current time of each video stream, and the time of the first frame of the first media segment file initially played by each video stream specifically includes:

[0015] Calculate the difference between the current time of each video stream and the time of the first frame of the first media segment file that was initially played for each video stream;

[0016] The reference time offset value of the current frame of each video stream is determined based on the difference and the reference time offset value of the first frame of the first media segment file initially played for each video stream.

[0017] Optionally, the step of synchronizing the normally playing video stream based on the difference between the maximum and minimum reference time offset values, or synchronizing the normally playing video stream based on the relationship between the average reference time offset value and the reference time offset value of the current frame of each video stream, specifically includes:

[0018] For a normally playing video stream, if the difference between the maximum reference time offset value and the minimum reference time offset value is greater than a preset threshold, all video streams are refreshed.

[0019] If the difference between the maximum reference time offset value and the minimum reference time offset value is less than the preset threshold, and the average reference time offset value is greater than the reference time offset value of the current frame, the video stream is fast-forwarded for a first preset time period.

[0020] If the difference between the maximum reference time offset and the minimum reference time offset is less than the preset threshold, and the average reference time offset is less than the reference time offset of the current frame, the video stream is paused for a second preset time period.

[0021] Optionally, the method further includes:

[0022] For video streams that are not playing normally, pause the synchronization operation until they resume normal playback; the abnormal playback includes any one of the following states: paused, video loading, or video loading failure.

[0023] On the other hand, embodiments of the present invention provide a multi-channel video synchronization method, applied to a streaming end, including:

[0024] Calculate the network latency time from the streaming end to the streaming media server for the current video stream;

[0025] The video stream and network latency are sent to the streaming media server so that the streaming media server generates an index file and media segment files, wherein the index file includes reference time offset information of the first frame of each media segment file.

[0026] Optionally, the network latency is calculated using the following method:

[0027] A first request is sent to the network latency query interface of the streaming media server, feedback information is received from the network latency query interface, and the first moment of sending the first request and the fourth moment of receiving the feedback information are recorded; the feedback information includes the second moment when the first request enters the network latency query interface and the third moment when the first request leaves the network latency query interface;

[0028] Calculate the network latency based on the first time point, the fourth time point, the second time point, and the third time point.

[0029] Optionally, the method further includes:

[0030] Calculate network latency at least 3 times;

[0031] Remove the maximum and minimum values ​​from at least three network latency times to obtain the remaining network latency time;

[0032] Update the network latency time to the average of the remaining network latency time.

[0033] On the other hand, embodiments of the present invention provide a multi-channel video synchronization method, applied to a streaming media server, comprising:

[0034] Receive the video stream sent by the streaming client and the network latency;

[0035] An index file and media segment files are generated based on the video stream; wherein, the index file includes the reference time offset information of the first frame of each media segment file, and the media segment file includes the network latency time of the current video stream being pushed to the streaming media server.

[0036] Optionally, the reference time offset information is calculated using the following method:

[0037] Send a second request to the query interface of the time base server;

[0038] Receive the reference time offset value information returned by the query interface of the time reference server.

[0039] On the other hand, embodiments of the present invention provide a multi-channel video synchronization system applied to a playback end, comprising:

[0040] The first module is used to obtain the index file and media segment file sent by the streaming media server; wherein, the index file includes the reference time offset value information of the first frame image of each media segment file, and the media segment file includes the network latency time of the current video stream being pushed from the streaming end to the streaming media server;

[0041] The second module is used to determine the first media segment file for initial playback of each video stream based on the current time position of the current video stream playback and the relative time relationship between the first frame image and the last frame image of each media segment file, and to obtain the reference time offset value of the first frame image of the first media segment file for initial playback of each video stream from the index file.

[0042] The third module is used to determine the reference time offset value of the current frame of each video stream according to a preset time interval, based on the reference time offset value of the first frame of the first media segment file of each video stream during initial playback, the current time of each video stream, and the time of the first frame of the first media segment file of each video stream during initial playback.

[0043] The fourth module is used to calculate the maximum reference time offset, minimum reference time offset, and average reference time offset value according to the reference time offset value of the current frame of each video stream at the preset time interval, and to perform synchronization operation on the normally playing video stream according to the difference between the maximum reference time offset value and the minimum reference time offset value, or to perform synchronization operation on the normally playing video stream according to the relationship between the average reference time offset value and the reference time offset value of the current frame of each video stream; the synchronization operation includes refreshing, fast forwarding, or pausing.

[0044] On the other hand, embodiments of the present invention provide a multi-channel video synchronization system applied to a streaming end, comprising:

[0045] The fifth module is used to calculate the network latency time of the current video stream from the streaming end to the streaming media server;

[0046] The sixth module is used to send the video stream and network latency time to the streaming media server so that the streaming media server can generate index files and media segment files.

[0047] On the other hand, embodiments of the present invention provide a multi-channel video synchronization system applied to a streaming media server, comprising:

[0048] The seventh module is used to receive the video stream and network latency sent by the streaming end;

[0049] The eighth module is used to generate an index file and media segment files based on the video stream; wherein the index file includes reference time offset information of the first frame image of each media segment file, and the media segment file includes the network latency time of the current video stream being pushed to the streaming media server.

[0050] On the other hand, embodiments of the present invention provide a multi-channel video synchronization device, comprising:

[0051] At least one processor;

[0052] At least one memory for storing at least one program;

[0053] When the at least one program is executed by the at least one processor, the at least one processor implements the video synchronization method described above for any one of the streaming end, streaming media server, or playback end.

[0054] On the other hand, embodiments of the present invention provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the video synchronization method described above for any one of the streaming end, streaming media server, or playback end.

[0055] On the other hand, embodiments of the present invention provide a multi-channel video synchronization system, including a streaming media server and a streaming push end and a playback end connected to the streaming media server; wherein,

[0056] The streaming terminal is used to execute the video synchronization method described above.

[0057] The streaming media server is used to execute the video synchronization method of the streaming media server;

[0058] The playback terminal is used to execute the video synchronization method of the playback terminal.

[0059] Implementing this embodiment of the invention has the following beneficial effects: First, the network latency time from the streaming end to the streaming media server is calculated by the streaming end, and this network latency time is sent to the streaming media server, taking into account the video timing error caused by the network latency from the streaming end to the streaming media server. Next, the media server generates an index file and media segment files for the video stream, and sends these files to the playback end. The index file includes the reference time offset value information of the first frame of each media segment file, and the media segment files include the network latency time from the current video stream to the streaming media server. The reference time offset value information takes into account the video timing error caused by system time errors between different streaming media servers, making it applicable to multiple different streaming media servers. Next, after receiving the index file and media segment file, the playback end determines the reference time offset value of the first frame of the first media segment file initially played by each video stream, and then determines the reference time offset value of the current frame of each video stream. Based on the reference time offset value of the current frame of each video stream, the normally playing video stream is synchronized. By obtaining the reference time offset value of the first frame of the first media segment file initially played by each video stream, the reference time offset value of the current frame of each video stream is determined. The calculation method is simple and does not require parsing the media segment file data packets. The synchronization operation of the normally playing video stream is performed by using the reference time offset value of the current frame of each video stream. The synchronization effect is good, and the time accuracy can reach the millisecond level. It can realize the synchronization of multiple video streams in live streaming scenarios, and each video stream can be controlled independently. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the structure of a multi-channel video synchronization system provided in an embodiment of the present invention;

[0061] Figure 2 This is a flowchart illustrating the steps of a multi-channel video synchronization method applied to a playback terminal, as provided in an embodiment of the present invention.

[0062] Figure 3 This is a flowchart illustrating the steps for determining the first media segment file for initial playback of a video stream, as provided in an embodiment of the present invention.

[0063] Figure 4 This is a flowchart illustrating the steps for determining the reference time offset value of the current frame of a video stream according to an embodiment of the present invention.

[0064] Figure 5 This is a flowchart illustrating the steps for synchronizing multiple video streams according to an embodiment of the present invention.

[0065] Figure 6 This is a flowchart illustrating the steps of a multi-channel video synchronization method applied to a streaming end, as provided in an embodiment of the present invention.

[0066] Figure 7 This is a flowchart illustrating the steps of a multi-channel video synchronization method applied to a streaming media server, as provided in an embodiment of the present invention.

[0067] Figure 8 This is a structural block diagram of a multi-channel video synchronization system applied to a playback terminal, provided by an embodiment of the present invention;

[0068] Figure 9 This is a structural block diagram of a multi-channel video synchronization system applied to a streaming end, provided by an embodiment of the present invention;

[0069] Figure 10 This is a structural block diagram of a multi-channel video synchronization system applied to a streaming media server, provided by an embodiment of the present invention.

[0070] Figure 11 This is a structural block diagram of a video synchronization system device provided in an embodiment of the present invention. Detailed Implementation

[0071] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0072] It is understood that the terms "first," "second," etc., used in the embodiments of the present invention may be used to describe various concepts herein, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another.

[0073] In the embodiments of the present invention, the terms "at least one", "multiple", "each", "any", etc., are used. "At least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiple, and "any" refers to any one of the multiple.

[0074] In related technologies, there are two modes for multi-channel HLS streaming media synchronization. One mode cannot operate independently on each video stream, while the other mode is subject to many limitations and cannot be applied to live streaming scenarios. The main reasons for HLS streaming media playback latency are: 1. Network latency from the streaming server to the streaming media server; 2. CDN distribution latency, playback terminal pull-down latency, decoding and rendering latency, etc.

[0075] Based on the above, embodiments of the present invention provide a multi-channel video synchronization method, system, apparatus, and storage medium. First, the network latency time from the streaming end to the streaming media server is calculated by the streaming end, and this network latency time is sent to the streaming media server, taking into account the video timing error caused by the network latency. Next, the media server generates an index file and media segment files for the video stream, and sends these files to the playback end. The index file includes the reference time offset value information of the first frame of each media segment file, and the media segment files include the network latency time from the current video stream to the streaming media server. The reference time offset value information takes into account the video timing error caused by system time errors between different streaming media servers, making it applicable to multiple streaming media servers. The same streaming media server is used. Then, after receiving the index file and media segment file, the playback end determines the reference time offset value of the first frame of the first media segment file of the initial playback of each video stream, and then determines the reference time offset value of the current frame of each video stream. Based on the reference time offset value of the current frame of each video stream, the normally playing video stream is synchronized. By obtaining the reference time offset value of the first frame of the first media segment file of the initial playback of each video stream, the reference time offset value of the current frame of each video stream is determined. The calculation method is simple and does not require parsing the media segment file data packets. By using the reference time offset value of the current frame of each video stream to perform synchronization operation on the normally playing video stream, it is possible to synchronize multiple video streams in a live broadcast scenario. The synchronization effect is good, the time accuracy can reach the millisecond level, and each video stream can be controlled independently.

[0076] See Figure 1 , Figure 1This is a schematic diagram illustrating the implementation environment of a multi-channel video synchronization method provided in an embodiment of the present invention. The multi-channel video synchronization method is applied to a multi-channel video synchronization system, which includes a streaming end 110, a streaming media server 120, and a playback end 130. The streaming end 110 calculates the network latency of the current video stream and sends the captured video stream and the network latency of the current video stream to the streaming media server 120. The streaming media server 120 creates an index file and media segment files from the received video streams and sends the index file and media segment files to the playback end 130. After receiving the index file and media segment files, the playback end 130 simultaneously plays the multiple video streams synchronously.

[0077] It should be noted that the streaming terminal 110 is connected to the streaming media server 120 via wired or wireless means, and the streaming media server 120 is connected to the playback terminal 130 via wired or wireless means. The streaming media server 120 can also be a cluster of multiple streaming media servers, and the number of streaming media servers in the cluster is determined according to the actual application requirements. The streaming terminal 110 or playback terminal 130 involved in this application embodiment includes, but is not limited to, smartphones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc.

[0078] See Figure 2 This invention provides a method for synchronizing multiple video streams, applied to a playback device, comprising the following steps:

[0079] S110. Obtain the index file and media segment file sent by the streaming media server; wherein, the index file includes the reference time offset value information of the first frame image of each media segment file, and the media segment file includes the network latency time of the current video stream being pushed from the streaming end to the streaming media server.

[0080] The playback terminal receives the index file and media segment files sent by the streaming media server. In this embodiment, the index file includes, but is not limited to, an M3U8 file, and the media segment files include, but are not limited to, TS files. The position of the reference time offset information of the first frame of the media segment file in the index file is determined according to the actual application. The position of the network latency time of the current video stream from the streaming terminal to the streaming media server in the media segment file is determined according to the actual application, and this embodiment does not impose specific limitations.

[0081] S120. Determine the first media segment file for initial playback of each video stream based on the current time position of the current video stream playback and the relative time relationship between the first and last frames of each media segment file, and obtain the reference time offset value of the first frame of the first media segment file for initial playback of each video stream from the index file.

[0082] When playing video on an HLS player, an M3U8 file includes multiple TS files. The player does not necessarily start playing from the first TS file in the TS file list of the M3U8 file. Therefore, it is necessary to analyze the relationship between the current position of the video playback and the relative time of the first frame and the relative time of the last frame of each TS slice to determine which TS file is played first for each stream on the current player. Then, the reference time offset value (in milliseconds) corresponding to the first frame of that video stream is obtained.

[0083] S130. According to a preset time interval, determine the reference time offset value of the current frame of each video stream based on the reference time offset value of the first frame of the first media segment file of each video stream during initial playback, the current time of each video stream, and the time of the first frame of the first media segment file of each video stream during initial playback.

[0084] It should be noted that the preset time interval refers to the time interval for video synchronization. The preset time interval is determined according to the actual application, and this embodiment does not impose specific restrictions. For example, the preset time interval is set to 1 second.

[0085] The playback end calculates the base time offset of the current frame for each video stream by taking into account the base time offset of the first frame of the TS file played at the beginning of each video stream, the current video stream playback time, and the base time offset of the latest frame played in the current TS file when this frame was generated. The obtained base time already takes into account the streaming latency of the push end, making it more accurate. The current video stream playback time can be directly obtained using native JavaScript in the webpage's H5 interface, and the base time offset of the latest frame played in the current TS file can be obtained from the filename of the current TS file. It should be noted that if the time units are inconsistent, unit conversion is required. For example, the base time offset of the first frame of the initially played TS file is in seconds, while the current video stream playback time is in milliseconds; therefore, milliseconds and seconds need to be converted during the calculation.

[0086] S140. According to a preset time interval, calculate the maximum reference time offset, minimum reference time offset, and average reference time offset based on the reference time offset of the current frame of each video stream. Perform synchronization operation on the normally playing video stream based on the difference between the maximum and minimum reference time offset, or perform synchronization operation on the normally playing video stream based on the relationship between the average reference time offset and the reference time offset of the current frame of each video stream. The synchronization operation includes refreshing, fast forwarding, or pausing.

[0087] Similarly, the preset time interval refers to the time interval for video synchronization. The preset time interval is determined according to the actual application. This embodiment does not impose specific restrictions. For example, the preset time interval is set to 1 second.

[0088] Specifically, the playback end first determines characteristic parameters such as the maximum reference time offset, minimum reference time offset, and average reference time offset based on the reference time offset value of the current frame of each video stream. Then, it performs synchronization operation on the normally playing video stream based on these characteristic parameters, or performs synchronization operation on the normally playing video stream based on the relationship between these characteristic parameters and the reference time offset value of the current frame of each video stream.

[0089] Alternatively, the multi-channel video synchronization method on the playback end also includes:

[0090] S150. For video streams that are not playing normally, pause the synchronization operation until they resume normal playback. Not playing normally includes any of the following states: paused, video loading, or video loading failure.

[0091] It should be noted that the video streams participating in the synchronization do not include those that are paused, loading, or have failed to load. These three types of video streams can participate in video synchronization once they are playing normally. Pausing the synchronization operation for video streams that are not playing normally reduces unnecessary computation and alleviates the computational burden.

[0092] Optionally, see Figure 3 Based on the current time position of the current video stream playback and the relative time relationship between the first and last frames of each media segment file, the first media segment file for initial playback of each video stream is determined, specifically including:

[0093] S121. Determine the duration of each media segment file in each video stream by the relative time relationship between the first and last frames of each media segment file in each video stream.

[0094] S122. The duration of each media segment file in each video stream is accumulated in chronological order to obtain the accumulated time of each media segment file in each video stream.

[0095] S123. Match the current time position of each current video stream with the cumulative time of each video stream to determine the first media segment file for the initial playback of each video stream.

[0096] Specifically, the duration of each media segment file is obtained by subtracting the playback time of the first frame from the playback time of the last frame. The durations of each media segment file in each video stream are accumulated according to the playback time order of the media segments to obtain the accumulated time of each media segment file in each video stream. If the current time position of the current video stream is within a certain accumulated time range, the media segment file corresponding to that accumulated time range is taken as the first media segment file.

[0097] Specifically, the function to retrieve the first TS file played from each video stream is as follows:

[0098] Fb=fb(Tsc x ,Tsd1,Tsf1,Tsd2,Tsf2,Tsd3,Tsf3……Tsd n ,Tsf n )

[0099] Where Fb is the filename of the first TS file played in the current stream, fb is the function to get the filename of the first TS file played in the current stream, and Tsc x Let currentTime be the current time (the current initial position of video playback) when the x-th player starts playing, Tsd1 be the total duration of the first TS file, and Tsf1 be the filename of the first TS file. n Tsf represents the total duration of the nth TS file. n Let Tsc be the filename of the nth TS file; if Tsc ∈ [0, Tsd1), then return Tsf1; if Tsc ∈ [0, Tsd1), then return Tsf1. Then return Tsf n .

[0100] In one specific embodiment, the HLS playback terminal records the initial time of each HLS video stream:

[0101] When each HLS video stream starts playing, the currentTime of each player is recorded (the currentTime of the initial video playback, which is generated by the player itself). For example, there are 3 video streams, the first video stream is denoted as Tsc1, the second video stream as Tsc2, and the third video stream as Tsc3.

[0102] Analyze the M3U8 file information of the currently playing HLS video: When each HLS video starts playing, the first TS file to be played is obtained by analyzing the corresponding M3U8 file information (playback does not necessarily start from the first TS file).

[0103] In a specific embodiment, the content of the shili.m3u8 file is as follows:

[0104] #EXTM3U

[0105] #EXT-X-VERSION:3

[0106] #EXT-X-MEDIA-SEQUENCE:4176

[0107] #EXT-X-TARGETDURATION:10

[0108] #EXTINF:9.493,1234

[0109] 73e162672dd62b880192796a1816b998_20.ts

[0110] #EXTINF:8.322,10727

[0111] d5b8473b826887491449319f277c7c06_40.ts

[0112] #EXTINF:2.633,19049

[0113] e88dd1fd8aa1b880e16618de42e25230_125.ts

[0114] This M3U8 file contains information about three TS files. The information for the first TS file is as follows:

[0115] #EXTINF:9.493,1234

[0116] 73e162672dd62b880192796a1816b998_20.ts

[0117] The first line contains two values, separated by a comma. The first value indicates the current TS file duration of 9.493 seconds. This means that if the first frame of the current TS file has a duration of 0 seconds, the last frame will have a duration of 9.493 seconds. The second value (here, 1234) indicates the base time offset (in milliseconds) corresponding to the first frame of the current video file.

[0118] The second line contains the video file name of the current TS file, which the player uses to download the TS video file. (The 20 in the file name here represents the network latency (in milliseconds) from the current video stream's push end to the streaming media server.

[0119] The information in the second TS file is as follows:

[0120] #EXTINF:8.322,10727

[0121] d5b8473b826887491449319f277c7c06_40.ts

[0122] The first line contains two values, separated by a comma. The first value indicates the current TS file duration of 8.322 seconds. If the time of the first frame of the current TS file is 9.493 seconds (based on the first frame of the first TS file in the current M3U8 file), then the last frame is 9.493 + 8.322 = 17.815 seconds. The second value (10727 in this case) indicates the reference time offset (in milliseconds) corresponding to the first frame of the current video file.

[0123] The second line contains the video file name of the current TS file. The player downloads the TS video file using this name (40 in the file name here indicates the network latency (in milliseconds) of the current video stream being pushed from the streaming end to the streaming media server.

[0124] The information in the third TS file is as follows:

[0125] #EXTINF:2.633,19049

[0126] e88dd1fd8aa1b880e16618de42e25230_125.ts

[0127] The first line contains two values, separated by a comma. The first value indicates the current TS file duration: 2.633 seconds. This means the first frame of the current TS file is 17.815 seconds (based on the first frame of the first TS file in the current M3U8 file). Therefore, the last frame is 9.493 + 8.322 + 2.633 = 20.448 seconds. The second value (here, 19049) represents the base time offset (in milliseconds) corresponding to the first frame of the current video file. The second line contains the video file name of the current TS file, which the player uses to download the TS video file. The "125" in the filename indicates the network latency (in milliseconds) between the current streaming end and the streaming media server.

[0128] When each HLS video starts playing, the player determines which of the three TS files the current player's Tsc1 value falls within during its start time. This allows the player to determine which TS file is being played first in the current stream. For example, if 0 ≤ Tsc1 < 9.493, it means the player is initially playing the first TS file; if 17.815 ≤ Tsc1 < 20.448, it means the player is initially playing the third TS file.

[0129] Optionally, see Figure 4The reference time offset value of the current frame of each video stream is determined based on the reference time offset value of the first frame of the first media segment file initially played for each video stream, the current time of each video stream, and the time of the first frame of the first media segment file initially played for each video stream. Specifically, this includes:

[0130] S131. Calculate the difference between the current time of each video stream and the time of the first frame of the first media segment file that is initially played for each video stream.

[0131] S132. Determine the reference time offset value of the current frame of each video stream based on the difference and the reference time offset value of the first frame of the first media segment file of each video stream during initial playback.

[0132] In one specific embodiment, the formula for calculating the specific reference time offset (measured in streaming media server time) for each stream playback is as follows:

[0133] T x =Tss x +(Tc x -Tsc x )×1000

[0134] Where Tx represents the reference time offset of the latest frame played in the Xth video stream, and Tss x Tc represents the streaming media server's base time offset value, in milliseconds, corresponding to the first frame of the first TS file that begins playing from the Xth video stream. x Tsc represents the latest current time value of the Xth video stream, in seconds; x The currentTime value represents the current time in seconds when the first frame of the first TS file of the Xth video stream begins playing.

[0135] For example, to obtain the current time (currentTime) and playback duration of each HLS stream, the current time of each HLS stream is as follows: the current time of the first video stream is denoted as Tc1, the current time of the second video stream is denoted as Tc2, and the current time of the third video stream is denoted as Tc3; the current playback duration of each HLS stream is as follows: the current duration of the first video stream is denoted as Tt1 = Tc1 - Tsc1, the current duration of the second video stream is denoted as Tt2 = Tc2 - Tsc2, and the current duration of the third video stream is denoted as Tt3 = Tc3 - Tsc3; the reference time offset value of the latest frame of the first TS file initially played for each stream is as follows: the reference time offset value of the first frame of the first TS file initially played for the first video stream is denoted as Tss1, the reference time offset value of the first frame of the first TS file initially played for the second video stream is denoted as Tss2, and the reference time offset value of the first frame of the first TS file initially played for the third video stream is denoted as Tss3. The reference time offset value of the current frame played by each HLS stream is calculated as follows: The reference time offset value of the latest frame played by the first video stream: T1=Tss1+(Tc1-Tsc1)×1000, the reference time offset value of the latest frame played by the second video stream: T2=Tss2+(Tc2-Tsc2)×1000, the reference time offset value of the latest frame played by the third video stream: T3=Tss3+(Tc3-Tsc3)×1000.

[0136] Optionally, see Figure 5 Synchronization is performed on the normally playing video stream based on the difference between the maximum and minimum reference time offset values, or based on the relationship between the average reference time offset value and the reference time offset value of the current frame of each video stream. Specifically, this includes:

[0137] S141. For a normally playing video stream, if the difference between the maximum reference time offset and the minimum reference time offset is greater than a preset threshold, refresh each video stream.

[0138] S142. If the difference between the maximum reference time offset and the minimum reference time offset is less than a preset threshold, and the average reference time offset is greater than the reference time offset of the current frame, the video stream is fast-forwarded for the first preset time period.

[0139] S143. If the difference between the maximum reference time offset and the minimum reference time offset is less than a preset threshold, and the average reference time offset is less than the reference time offset of the current frame, the video stream is paused for a second preset time period.

[0140] It should be noted that the preset threshold is determined based on the actual application, and this embodiment does not impose specific restrictions. For example, the preset threshold is set to 80 seconds. The first preset time period or the second preset time period is determined based on the magnitude of the average reference time offset value and the reference time offset value of the current frame.

[0141] Specifically, by comparing the specific reference time offset T of each video stream playing to the latest frame. x ; Calculate the maximum T for each video stream x Let it be denoted as Tmax, the minimum T for each flow. x Let Tmax be denoted as Tmin. If Tmax - Tmin is greater than a certain threshold (e.g., 100 seconds), a synchronization operation is performed. The synchronization operation has two schemes. First, it is determined whether synchronization scheme 1 is met: if Tmax - Tmin is too large, exceeding a set threshold (e.g., 3000 milliseconds), then synchronization is achieved by refreshing all participating video streams through the player; otherwise, synchronization scheme 2 is used. Synchronization scheme 2: The average value of the specific reference time offset of each stream to the latest frame is calculated and denoted as Tpj. Then, the Tmax - Tmin of each video stream is... x The value is compared with Tpj; the slower stream is fast-forwarded to Tpj-T. x Milliseconds, fast-playing video streams are paused via T. x- Playback begins only after the duration of Tpj, achieving synchronized playback across all streams. When Tmax-Tmin is too large, exceeding a set threshold, synchronization scheme 2 is ineffective, causing frequent video reloading; therefore, synchronization scheme 1 is used.

[0142] See Figure 6 This invention provides a multi-channel video synchronization method applied to a streaming end, comprising:

[0143] S210. Calculate the network latency time from the streaming end to the streaming media server for the current video stream;

[0144] S220. The video stream and network latency are sent to the streaming media server so that the streaming media server generates an index file and media segment files, wherein the index file includes reference time offset information of the first frame image of each media segment file.

[0145] Specifically, the streaming end first calculates the network latency time from the current video stream to the streaming media server, and then sends the multiple video streams and the network latency time to the streaming media server; after receiving the multiple video streams and the network latency time, the streaming media server generates an index file and media segment files.

[0146] Optionally, network latency is calculated using the following method:

[0147] S211. Send a first request to the network latency query interface of the streaming media server, receive feedback information from the network latency query interface, and record the first moment of sending the first request and the fourth moment of receiving the feedback information; the feedback information includes the second moment when the first request enters the network latency query interface and the third moment when the first request leaves the network latency query interface.

[0148] S212. Calculate the network delay time based on the first time point, the fourth time point, the second time point, and the third time point.

[0149] Specifically, the streaming client sends a request to the streaming media server's network latency query interface and receives feedback information. Then, it calculates the network latency based on the feedback information, the sending time, and the receiving time. For example, it calculates the number of milliseconds (denoted as T) of the network latency for each stream from the streaming client to the streaming media server. TLYS ), and make a record. T TLYS The calculation method involves sending a request from the streaming client to the network latency query interface provided by the streaming media server, with the streaming client recording the request sending time T. TLFS (in milliseconds) and the time T for receiving the return value TLFH (in milliseconds), the data returned by the network latency query interface includes the time T when the request entered the interface. TLJR (in milliseconds), and the time T for the request to leave the interface. TLLK (Milliseconds). The formula for calculating the network latency from the streaming client to the streaming media server is: T TLYS =(T TLFH -T TLFS -(T TLLK -T TLJR )) / 2.

[0150] Optionally, the method further includes:

[0151] S2121. Calculate the network latency at least 3 times;

[0152] S2122. Remove the maximum and minimum values ​​from at least three network latency times to obtain the remaining network latency time;

[0153] S2123. Update the network latency time to the average of the remaining network latency time.

[0154] To improve the accuracy of network latency in the push client's push to the streaming media server, T TLYS The test needs to be performed 3 or more times. Then, the maximum and minimum values ​​are removed, and the average of the remaining values ​​is denoted as T. TLYSPJFor example, if the network latency is calculated four times: 1s, 0.5s, 0.8s, and 0.9s, and the maximum value of 1s and the minimum value of 0.5s are removed, the average network latency is (0.8+0.9) / 2 = 0.85s, and the final calculated network latency is 0.85s.

[0155] See Figure 7 This invention provides a multi-channel video synchronization method applied to a streaming media server, comprising:

[0156] S310, Receive the video stream and network latency sent by the streaming end;

[0157] S320. Generate an index file and media segment files based on the video stream; wherein, the index file includes the reference time offset information of the first frame image of each media segment file, and the media segment file includes the network latency time of the current video stream being pushed to the streaming media server.

[0158] Specifically, the streaming media server receives the video stream and network latency from the streaming client, and then generates an index file and media segment files based on the video stream and network latency. It should be noted that the storage location of the reference time offset and network latency information is determined according to the actual application; this embodiment does not impose specific limitations.

[0159] In one specific embodiment, when the streaming media server slices the video stream (generating TS files), the reference time offset value information corresponding to the first frame of each slice is written to the end of the #EXTINF duration value of the current slice in the M3U8 file, separated by commas; at the same time, the network latency T in milliseconds for pushing the current video stream to the streaming media server is also written. TLYS The value is contained in the name of the current TS file, separated from other filename parts by a symbol. For example, in the TS file name xxxx_20.ts, 20 represents the network latency in milliseconds for the current video stream to be pushed to the streaming media server. An example M3U8 file is shown below:

[0160] The contents of the shili.m3u8 file are:

[0161] #EXTM3U

[0162] #EXT-X-VERSION:3

[0163] #EXT-X-MEDIA-SEQUENCE:4176

[0164] #EXT-X-TARGETDURATION:10

[0165] #EXTINF:9.493,1234

[0166] 73e162672dd62b880192796a1816b998_20.ts

[0167] #EXTINF:8.322,10727

[0168] d5b8473b826887491449319f277c7c06_40.ts

[0169] #EXTINF:2.633,19049

[0170] e88dd1fd8aa1b880e16618de42e25230_125.ts

[0171] Optionally, the reference time offset information is calculated using the following method:

[0172] S321. Send a second request to the query interface of the time base server;

[0173] S322. Receive the reference time offset value information fed back by the query interface of the time reference server.

[0174] Specifically, the time base server provides a unified reference time offset value acquisition service for the streaming media server cluster. First, it maintains consistency with a certain standard time through NTP service or a Chrony time synchronization server. Then, the time base server provides a reference time offset value interface service to provide a consistent reference time offset value acquisition service for the current streaming media server cluster. The reference time offset is the number of milliseconds that have elapsed since the current streaming media server cluster was started. This reduces the amount of data transmitted and unifies the time of the streaming media server cluster. By providing a unified reference time offset value query service, the problem of time inconsistencies between different streaming media servers can be avoided.

[0175] See Figure 8 This invention provides a multi-channel video synchronization system for use in a playback device, comprising:

[0176] The first module is used to obtain the index file and media segment file sent by the streaming media server; wherein, the index file includes the reference time offset value information of the first frame image of each media segment file, and the media segment file includes the network latency time of the current video stream from the streaming end to the streaming media server;

[0177] The second module is used to determine the first media segment file for initial playback of each video stream based on the current time position of the current video stream playback and the relative time relationship between the first and last frames of each media segment file, and to obtain the reference time offset value of the first frame of the first media segment file for initial playback of each video stream from the index file.

[0178] The third module is used to determine the reference time offset value of the current frame of each video stream according to a preset time interval, based on the reference time offset value of the first frame of the first media segment file of each video stream during initial playback, the current time of each video stream, and the time of the first frame of the first media segment file of each video stream during initial playback.

[0179] The fourth module is used to calculate the maximum, minimum, and average reference time offset values ​​based on the reference time offset value of the current frame of each video stream at preset time intervals. It performs synchronization operations on the normally playing video stream based on the difference between the maximum and minimum reference time offset values, or based on the relationship between the average reference time offset value and the reference time offset value of the current frame of each video stream. The synchronization operations include refreshing, fast forwarding, or pausing.

[0180] See Figure 9 This invention provides a multi-channel video synchronization system for use in a streaming terminal, comprising:

[0181] The fifth module is used to calculate the network latency time of the current video stream from the streaming end to the streaming media server;

[0182] The sixth module is used to send the video stream and network latency time to the streaming media server so that the streaming media server can generate index files and media segment files.

[0183] See Figure 10 This invention provides a multi-channel video synchronization system applied to a streaming media server, comprising:

[0184] The seventh module is used to receive the video stream and network latency sent by the streaming end;

[0185] The eighth module is used to generate an index file and media segment files based on the video stream. The index file includes the reference time offset information of the first frame of each media segment file, and the media segment files include the network latency time of the current video stream being pushed to the streaming media server.

[0186] See Figure 11 This invention provides a multi-channel video synchronization device, comprising:

[0187] At least one processor;

[0188] At least one memory for storing at least one program;

[0189] When at least one program is executed by at least one processor, the at least one processor implements the video synchronization method described above for any one of the streaming end, streaming media server, or playback end.

[0190] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. The memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include remote memory located remotely relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0191] Furthermore, this application also discloses a computer program product or computer program stored in a computer-readable storage medium. A processor of a computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the described method. Similarly, the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0192] This invention also provides a computer-readable storage medium storing a processor-executable program that, when executed by a processor, implements the above-described method.

[0193] It is understood that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0194] See Figure 1 This invention provides a multi-channel video synchronization system, including a streaming media server and a streaming push end and a playback end connected to the streaming media server; wherein,

[0195] The streaming end is used to execute the video synchronization method described above.

[0196] A streaming media server, used to execute the video synchronization method of the streaming media server;

[0197] The playback end is used to execute the video synchronization method on the playback end.

[0198] Specifically, for the streaming end, it is mainly implemented through a video shooting device, and it may specifically include at least one camera and a transmission component; the streaming media server may also be a streaming media server cluster, and the streaming media server includes a processor and memory; for the playback end, it is mainly implemented through a playback device, and it may specifically include at least a player and a processor.

[0199] It is evident that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0200] In a specific live shopping application scenario, before pushing the video stream, the streaming client calculates the network latency of the current video stream. Then, the streaming client sends multiple video streams from different aspects of the same product and the network latency to the streaming media server cluster. After receiving the multiple video streams of the product, one or more streaming media servers in the streaming media server cluster obtain the reference time offset information from the time base server. Then, based on the multiple video streams, the reference time offset information, and the network latency, they generate an index file and a media segment file, and send the index file and media segment file to the streaming client. After receiving the index file and media segment file, the streaming client first determines the first media segment file for initial playback of each video stream, as well as the reference time offset value of the first frame of the first media segment file. Then, according to a preset time interval, it determines the reference time offset value of the current frame of each video stream, and performs synchronization operations such as refreshing, fast forwarding, or pausing on the normally playing video stream based on the reference time offset value of the current frame of each video stream.

[0201] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method of synchronizing multiple video paths, the method comprising: Applied to a playing terminal, comprising: obtaining an index file and media segment files sent by a streaming media server, wherein the index file comprises reference time offset value information of a first frame image of each of the media segment files, and the media segment files comprise network delay time of a current video stream pushed from a pushing terminal to the streaming media server; determining a first media segment file for initial playing of each video stream according to a current time position of the current video stream playing, and relative time relationship between the first frame image and a last frame image of each of the media segment files, and obtaining reference time offset value of the first frame image of the first media segment file for initial playing of each video stream from the index file; determining reference time offset value of a current frame of each video stream according to the reference time offset value of the first frame image of the first media segment file for initial playing of each video stream, current time of each video stream, and time of the first frame image of the first media segment file for initial playing of each video stream at a preset time interval; calculating maximum reference time offset value, minimum reference time offset value and average reference time offset value according to the reference time offset value of the current frame of each video stream at the preset time interval, and performing synchronization operation on normally played video streams according to difference between the maximum reference time offset value and the minimum reference time offset value, or according to relationship between the average reference time offset value and the reference time offset value of the current frame of each video stream; the synchronization operation comprises refreshing, fast forwarding or pausing.

2. The synchronization method of claim 1, wherein, The determining of the first media segment file for initial playing of each video stream according to the current time position of the current video stream playing, and the relative time relationship between the first frame image and the last frame image of each of the media segment files specifically comprises: determining time length of each media segment file in each video stream according to relative time relationship between the first frame image and the last frame image of each media segment file in each video stream respectively; adding the time length of each media segment file in each video stream in time sequence to obtain accumulated time of each media segment file in each video stream; matching the current time position of the current video stream playing with the accumulated time of each video stream to determine the first media segment file for initial playing of each video stream.

3. The synchronization method of claim 1, wherein, The determining of the reference time offset value of the current frame of each video stream according to the reference time offset value of the first frame image of the first media segment file for initial playing of each video stream, current time of each video stream, and time of the first frame image of the first media segment file for initial playing of each video stream specifically comprises: calculating difference between the current time of each video stream and the time of the first frame image of the first media segment file for initial playing of each video stream; determining the reference time offset value of the current frame of each video stream according to the difference and the reference time offset value of the first frame image of the first media segment file for initial playing of each video stream.

4. The synchronization method of claim 1, wherein, The synchronization operation of the normally played video stream according to the difference between the maximum reference time offset value and the minimum reference time offset value, or the synchronization operation of the normally played video stream according to the relationship between the average reference time offset value and the reference time offset value of the current frame of each video stream, specifically comprises: For the normally played video stream, if the difference between the maximum reference time offset value and the minimum reference time offset value is greater than a preset threshold, refreshing each video stream; If the difference between the maximum reference time offset value and the minimum reference time offset value is less than the preset threshold, and the average reference time offset value is greater than the reference time offset value of the current frame, fast forwarding the video stream by a first preset time period; If the difference between the maximum reference time offset value and the minimum reference time offset value is less than the preset threshold, and the average reference time offset value is less than the reference time offset value of the current frame, pausing the video stream by a second preset time period.

5. The synchronization method according to any of claims 1-4, characterized by, The method further comprises: For the video stream not normally played, pausing the synchronization operation until normally played again; the not normally played includes any one of a pause state, a video loading state or a video loading failure state.

6. A multiplexed video synchronization apparatus, characterized by, Comprise: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-5.

7. A computer readable storage medium having stored therein a program that is executable by a processor, characterized in that, The processor executable program when executed by the processor is used to execute the method of any one of claims 1-5.

8. A multiple video synchronization system, characterized by, Comprise a streaming media server, a push stream end connected with the streaming media server and a playing end; wherein, The push stream end is used to calculate the network delay time of the current video stream from the push stream end to the streaming media server; send the video stream and the network delay time to the streaming media server, so that the streaming media server generates an index file and a media segment file, wherein the index file comprises the reference time offset value information of the first frame image of each media segment file; The streaming media server is used to receive the video stream and the network delay time sent by the push stream end, and generate an index file and a media segment file according to the video stream; wherein the index file comprises the reference time offset value information of the first frame image of each media segment file, and the media segment file comprises the network delay time of the current video stream pushed to the streaming media server; The playing end is used to execute the method of any one of claims 1-5.

Images