Video synchronous playback method, system and device and storage medium

Abstract

The invention provides a video synchronous playback method, system and device, and a storage medium. The method comprises the following steps: receiving video frames from a plurality of media resources and buffering the video frames; for each media resource, determining a corresponding current clock frequency; wherein in one statistical period, the average clock frequency in the statistical period is calculated based on the total receiving time and the RTP timestamp difference value, and the current clock frequency is determined according to the average clock frequency; for the buffered video frame, calculating an absolute timestamp based on the RTCP sender report and the corresponding current clock frequency; and encapsulating the buffered video frame to obtain an encapsulated video file of the plurality of media resources, sending the encapsulated video file and an absolute timestamp list of the video frame to a client, and after the client is configured to receive the encapsulated video file of the plurality of media resources, synchronously playing back the video frame in the encapsulated video file according to the absolute timestamp list. By adopting the method and the device, various types of media resource video data can be well compatible.

Description

Background Art

[0002] Synchronous playback of video surveillance recordings is a common function in security solutions. Usually, users can select a certain time point to start synchronous forward or reverse playback (playback), so that multiple media resources can be viewed at the same time. This method can observe the details of an event from different angles, help to accurately analyze and judge the cause and process of the event, provide multi-angle evidence for law enforcement, and ensure the integrity and reliability of the evidence.

[0003] In the process of synchronous video playback, how to obtain the absolute timestamp of each frame is currently the biggest challenge. In the existing solution, the clock rate of the bitstream can be obtained through the SDP (Session Description Protocol) message carried in the signaling. The SDP message contains the format of "a=rtpmap:<payload type><encoding name> / <clock rate> [ / <encoding parameters> ]”). Then, from the sender report of RTCP (Real-time Transport Control Protocol), an RTP (Real-time Transport Protocol) timestamp and the corresponding NTP (Network Time Protocol) timestamp are obtained. Finally, the frame absolute timestamp of each frame is calculated by the following formula:

[0004]

[0005] Among them, RTCP report refers to the sender report of RTCP. However, there are three problems with the existing solution. The first problem is that the existing solution relies on the clock frequency in the SDP message, but some old devices or some brand devices do not use the clock frequency in the SDP message to stamp the RTP timestamp. For example, the clock frequency of the video is usually 90000, but the actual clock frequency of these devices may be 25, 30, 50, 60, 1000, etc., and even the clock frequency per second may fluctuate. The second problem is that some manufacturers connect through SDK (Software Development Kit, Software Development Kit), and without SDP messages, they cannot obtain the accurate clock frequency. The third problem is that some devices may have inaccurate clocks, which are manifested as fast or slow clocks. For example, the actual time has passed 3 seconds, and the device's clock may have passed 5 seconds or 1 second, which results in inaccurate clock frequency. Due to the above three problems, the traditional solution cannot effectively and accurately obtain the absolute timestamp of the frame. Summary of the invention

[0006] In response to the problems in the prior art, the purpose of the present application is to provide a video synchronous playback method, system, device and storage medium, which can accurately calculate the absolute timestamp used for synchronous playback for media resources whose clock frequency cannot be obtained from SDP messages, thereby being well compatible with various types of media resource video data.

[0007] The present invention provides a method for synchronously replaying a video, comprising the following steps:

[0008] Receive and buffer video frames from multiple media sources;

[0009] For each media resource, determine the current clock frequency corresponding to the media resource; wherein, within a statistical period, the average clock frequency within the statistical period is calculated based on the total receiving time within the statistical period and the RTP timestamp difference, and the current clock frequency is determined according to the average clock frequency within the statistical period;

[0010] For the buffered video frame, calculating an absolute timestamp based on the RTCP sender report and the corresponding current clock frequency;

[0011] The encapsulated buffered video frames are encapsulated to obtain encapsulated video files of multiple media resources, and the encapsulated video files and an absolute timestamp list of the video frames therein are sent to a client. The client is configured to synchronously play back the video frames in the encapsulated video files according to the absolute timestamp list after receiving the encapsulated video files of multiple media resources.

[0012] By adopting the video synchronous playback method of the present application, after obtaining the video frames of multiple media resources, the average clock frequency within a statistical period is calculated based on the total receiving time and the RTP timestamp difference within the statistical period, and then the current clock frequency is determined without relying on SDP messages. For media resources that cannot obtain the clock frequency from the SDP message, the accurate clock frequency can also be obtained, and then the absolute timestamp used for synchronous playback can be accurately calculated, so that it can be well compatible with various types of media resource video data. After sending the encapsulated video file and the absolute timestamp list to the client, the client can synchronously play back the videos of multiple media resources according to the absolute timestamp list, which improves compatibility and adaptability and realizes high-precision synchronous playback of multiple media resource videos.

[0013] In some embodiments, determining the current clock frequency according to the average clock frequency in the statistical period includes the following steps:

[0014] Determine whether the difference between the average clock frequency in the statistical period and the current clock frequency is greater than a preset difference threshold;

[0015] If yes, the current clock frequency is updated using the average clock frequency within the statistical period;

[0016] If not, keep the current clock frequency unchanged.

[0017] In some embodiments, the time length of the statistical period is the time length of one or more picture groups, and the first frame of the statistical period corresponds to the first frame of a picture group, or the time length of the statistical period is equal to the time length of the period reported by the RTCP sender.

[0018] In some embodiments, calculating an absolute timestamp based on an RTCP sender report and a corresponding current clock frequency comprises the following steps:

[0019] determining whether the buffered video frame corresponds to an absolute time synchronization checkpoint;

[0020] If yes, the absolute timestamp of the video frame is calculated based on the RTCP sender report and the corresponding current clock frequency.

[0021] In some embodiments, the client is a web client;

[0022] The encapsulation of the buffered video frames to obtain encapsulated video files of multiple media resources comprises the following steps:

[0023] The buffered video frames are encapsulated into multiple fragments to obtain encapsulated video files corresponding to each of the media resources, wherein each of the encapsulated video files includes multiple fragments, each fragment includes one or more picture groups, and each fragment adopts the FMP4 format that complies with the ISO BMFF specification.

[0024] In some embodiments, the client is configured to synchronously play back the video frames in the encapsulated video files after receiving the encapsulated video files of multiple media resources using the following steps:

[0025] According to the absolute timestamp list, determine the first frame with the smallest absolute timestamp, use the encapsulated video file to which the first frame belongs as the reference file, and use the smallest absolute timestamp as the reference time;

[0026] Calculate the time offset of the first frame of other packaged video files except the reference file relative to the reference time;

[0027] Input each segment in the encapsulated video file into a pre-created media source buffer object, create a video element object for each encapsulated video file, and associate the video element object with the media source buffer object;

[0028] Initialize the current time attribute of the video element object of the reference file, and make the video element object of the reference file enter the playing state, and the video element objects of other encapsulated video files enter the pause state;

[0029] When the reference file is played to a time point corresponding to a time offset of other encapsulated video files, the current time attribute of the video element objects of other encapsulated video files is initialized, and the video element objects of other encapsulated video files enter a playing state.

[0030] In some embodiments, after the client causes the video element objects of other encapsulated video files to enter the play state, the client further includes controlling the synchronous playback of the encapsulated video files of multiple media resources by using the following steps:

[0031] Determine the original frame interval time according to the current speed;

[0032] Determine an event interval time of a synchronous playback timer according to the original frame interval time;

[0033] At every event interval, the current time attribute of the video element object of the currently playing encapsulated video file is increased by the event interval.

[0034] The embodiment of the present application further provides a video synchronous playback system, which is applied to the video synchronous playback method. The system includes a video-on-demand server and a client. The video-on-demand server includes:

[0035] A video receiving module, used for receiving and buffering video frames from multiple media resources;

[0036] A clock frequency calculation module is used to determine the current clock frequency corresponding to each media resource for each media resource; wherein, within a statistical period, the average clock frequency within the statistical period is calculated based on the total receiving time within the statistical period and the RTP timestamp difference, and the current clock frequency is determined according to the average clock frequency within the statistical period;

[0037] An absolute time calculation module, configured to calculate an absolute timestamp for the buffered video frame based on an RTCP sender report and a corresponding current clock frequency;

[0038] An encapsulation and sending module, used for encapsulating the buffered video frames to obtain encapsulated video files of multiple media resources, and sending the encapsulated video files and a list of absolute timestamps of the video frames therein to a client;

[0039] The client is configured to synchronously play back the video frames in the encapsulated video files according to the absolute timestamp list after receiving the encapsulated video files of multiple media resources.

[0040] By adopting the video synchronization playback system provided by the present application, each functional module can implement the steps of the above-mentioned video synchronization playback method, thereby obtaining the beneficial effects of the above-mentioned video synchronization playback method.

[0041] The embodiment of the present application also provides a video synchronous playback device, including:

[0042] processor;

[0043] a memory storing executable instructions of the processor;

[0044] Wherein, the processor is configured to perform the steps of the video synchronous playback method by executing the executable instructions.

[0045] By adopting the video synchronous playback device provided by the present application, the processor executes the video synchronous playback method when executing the executable instructions, thereby obtaining the beneficial effects of the above-mentioned video synchronous playback method.

[0046] An embodiment of the present application also provides a computer-readable storage medium for storing a program, which, when executed by a processor, implements the steps of the video synchronous playback method.

[0047] By adopting the computer-readable storage medium provided by the present application, the program stored therein implements the steps of the video synchronous playback method when being executed, thereby obtaining the beneficial effects of the above-mentioned video synchronous playback method. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings.

[0049] Figure 1 is a flow chart of a video synchronous playback method according to an embodiment of the present application;

[0050] Figure 2 This is a flow chart of calculating the average clock frequency within a statistical period according to an embodiment of the present application;

[0051] Figure 3 This is a flowchart of client synchronous playback in one embodiment of the present application;

[0052] Figure 4 is a schematic diagram of a video synchronous playback system according to an embodiment of the present application;

[0053] Figure 5 It is a structural schematic diagram of a video synchronous playback device according to an embodiment of the present application;

[0054] Figure 6 It is a schematic diagram of the structure of a computer storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their repeated description will be omitted.

[0056] like Figure 1 As shown, in one embodiment, the present application provides a video synchronous playback method. The method is implemented using a one-on-demand server and includes the following steps:

[0057] S100: receiving video frames from multiple media resources and buffering them;

[0058] Here, each media resource is provided by a different media device, such as a camera, a video recording platform, etc. Each media resource includes video data obtained from a media device;

[0059] The video frames are transmitted between each media device and the on-demand server through the RTP protocol; after the connection between the media device and the on-demand server is established, the on-demand server continuously obtains video frames from the media resource in the form of a code stream; the on-demand server is used to interact with the client and receive the client's synchronous playback request, which includes information such as the media resource URI (Uniform Resource Identifier) ​​and the start time of the media resource to be played back. The on-demand server establishes a connection with the corresponding media device according to the synchronous playback request and obtains the video frame code stream that meets the start time requirement in the synchronous playback request;

[0060] When the on-demand server obtains the video frame and buffers it in the server, it records the receiving time and RTP timestamp of each frame for calculation in the subsequent step S200; the video frame is buffered here for subsequent encapsulation of the video frame;

[0061] S200: For each media resource, determine the current clock frequency corresponding to the media resource; wherein, within a statistical period, calculate the average clock frequency within the statistical period based on the total receiving time within the statistical period and the RTP timestamp difference, and determine the current clock frequency according to the average clock frequency within the statistical period;

[0062] Here, each media resource calculates the current clock frequency separately, and does not need to rely on SDP messages; the statistical period refers to the time range for the average clock frequency statistics. The on-demand server uses the statistical period as the statistical unit to continuously count the average clock frequency in each statistical period. The statistical period has a preset period length. At the end of each statistical period (after receiving the last video frame in the statistical period), the total receiving time and RTP timestamp difference corresponding to all video frames in the statistical period are counted, and then the average clock frequency in the statistical period is calculated. This is an averaging process of the clock frequency in the statistical period, which can eliminate the problem of inaccurate clock frequency caused by a single clock frequency fluctuation. After the average clock frequency of the statistical period is counted, a new video frame is received as the first video frame of the next statistical period; the statistical period of each media resource is timed separately, and the statistical result is the average clock frequency in each statistical period of each media resource;

[0063] Here, the total receiving time in each statistical period corresponds to the difference between the receiving time of the last frame in the statistical period and the receiving time of the first frame. The receiving time refers to the time when the on-demand server receives the video frame. The RTP timestamp difference in each statistical period corresponds to the difference between the RTP timestamp of the last frame in the statistical period and the RTP timestamp of the first frame. When the media device sends a video frame to the on-demand server, each video frame is sent using an RTP packet. The RTP packet carries the RTP timestamp of the current frame. The RTP timestamp in the RTP packet is used to calculate the RTP timestamp difference in this step.

[0064] S300: For the buffered video frame, calculate an absolute timestamp based on the RTCP sender report and the corresponding current clock frequency;

[0065] The current timestamp here is the current timestamp of the media resource corresponding to the video frame; the RTCP sender report includes RTP timestamp and NTP timestamp. NTP timestamp is an inherent attribute of the video frame and is generated when recording. RTP timestamp is a relative time and may change. The time difference between the RTP time of the current frame minus the RTP time of the previous frame and the time difference between the NTP timestamp of the current frame minus the NTP time of the previous frame are multiples. The multiple relationship is related to the clock frequency; each video frame has RTP timestamp and NTP Timestamp: When the media device sends a video frame to the on-demand server, each video frame is sent using an RTP packet. The RTP packet carries the RTP timestamp of the current frame, which is used to calculate the RTP timestamp difference within the statistical period in the above step S200. The RTP packet does not carry the NTP timestamp. The RTCP sender report selects the RTP timestamp and NTP timestamp of a certain frame to send. According to the relationship between the difference between the RTP timestamps and the NTP timestamp difference between different frames, the absolute time of each frame is calculated. The optional calculation method is further described below.

[0066] S400: Encapsulate the buffered video frames to obtain encapsulated video files of multiple media resources, and send the encapsulated video files and a list of absolute timestamps of the video frames therein to the client. The client is configured to synchronously play back the video frames in the encapsulated video files according to the list of absolute timestamps after receiving the encapsulated video files of the multiple media resources.

[0067] Here, the absolute timestamp list includes the absolute timestamps of each video frame in the encapsulated video file calculated by step S300. The absolute timestamp list includes the absolute timestamp corresponding to the frame number of each video frame in the encapsulated video file. The absolute timestamp of each frame can be obtained by searching the list. Synchronous playback refers to the synchronous playback of multiple media resources based on absolute timestamps on the client; multiple media resources can be displayed on the same screen of the client at the same time by splicing the screen, each media resource occupies an image area, and the encapsulated video file of the media resource will continue to play in the image area to which it belongs in chronological order, and the absolute time between the multiple media resources is aligned with each other. For example, the user can view the surveillance videos of multiple monitoring devices at the same time on the client at the same time, so that the details of the same event can be observed from different angles.

[0068] By adopting the video synchronous playback method of the present application, after obtaining the video frames of multiple media resources through step S100, the average clock frequency within a statistical period is calculated according to the total receiving time and the RTP timestamp difference within the statistical period through step S200, and then the current clock frequency is determined without relying on the SDP message. For media resources whose clock frequency cannot be obtained from the SDP message (such as video streams that are not RTP timestamped according to the clock frequency in the SDP message and video streams that lack clock frequency information), the accurate clock frequency can also be obtained, and then the absolute timestamp used for synchronous playback is accurately calculated through step S300, so that it can be well compatible with various types of media resource video data. After sending the encapsulated video file and the absolute timestamp list to the client through step S400, the client can synchronously play back the videos of multiple media resources according to the absolute timestamp list, thereby improving compatibility and adaptability and realizing high-precision synchronous playback of multiple media resource videos.

[0069] Before step S200, the time length of the statistical cycle is also set. A longer statistical cycle can provide a more accurate clock frequency estimation result, and can smooth out short-term fluctuations and errors, but there may be a delay in clock frequency update. A shorter statistical cycle can reduce delays, but may result in reduced accuracy of clock frequency estimation. Therefore, the time length of the statistical cycle is related to the sensitivity of clock frequency adjustment. The longer the statistical cycle, the weaker the sensitivity. It is necessary to weigh the length of the statistical cycle according to the specific application scenario and performance requirements.

[0070] In this embodiment, the time length of the statistical period can be selected as the time length of one or more picture groups, and the first frame of the statistical period corresponds to the first frame of a picture group, so that the statistical period is consistent with the rhythm of encapsulation according to the picture group during subsequent encapsulation. In another alternative implementation mode I, the statistical period can also be set to be consistent with the period reported by the RTCP sender, that is, the time length of the statistical period is equal to the period reported by the RTCP sender, so that it is consistent when the absolute timestamp is calculated according to the RTCP sender report and the time frequency. However, the present application is not limited to this. In other alternative implementation modes, in order to reduce the number of buffered video frames when calculating the clock frequency, the statistical time can also be set to be less than the time length of a picture group. Here, the picture group (GOP) is a collection of continuous pictures in a coded video stream, and each coded video stream is composed of continuous GOPs. GOP includes I frames (key frames) and non-key frames, and key frames can be decoded independently, while non-key frames need to rely on other frames for decoding. In video coding, the GOP structure specifies the arrangement order within and between frames, that is, the image arrangement between I frames and I frames. The first image of a GOP must be an I frame, which ensures that each GOP does not need to refer to other images when decoding, can be decoded independently, and allows fast searching through the video to assist random access of the video.

[0071] like Figure 2 As shown, in step S200, the average clock frequency in the statistical period is calculated based on the total receiving time and the RTP timestamp difference in the statistical period, including the following steps:

[0072] S210: Calculate the total receiving time of the statistical period based on the receiving time of the last frame and the receiving time of the first frame in the statistical period; that is, subtract the receiving time of the first frame from the receiving time of the last frame to obtain the accurate total receiving time in the statistical period;

[0073] S220: Calculate the RTP timestamp difference of the statistical period based on the RTP timestamp of the last frame and the RTP timestamp of the first frame in the statistical period;

[0074] That is, the RTP timestamp of the last frame is subtracted from the RTP timestamp of the first frame to obtain the RTP timestamp difference within the statistical period;

[0075] S230: Calculate the average clock frequency in the statistical period according to the total receiving time and RTP timestamp difference in the statistical period;

[0076] The RTP timestamp obtained in step S220 is divided by the total receiving time obtained in step S210 to obtain the average clock frequency in the statistical period.

[0077] In this embodiment, in step S200, determining the current clock frequency according to the average clock frequency in the statistical period includes the following steps:

[0078] Determine whether the difference between the average clock frequency in the statistical period and the current clock frequency is greater than a preset difference threshold;

[0079] If yes, the current clock frequency is updated using the average clock frequency within the statistical period;

[0080] If not, keep the current clock frequency unchanged.

[0081] Therefore, only when the calculated average clock frequency within the statistical period fluctuates greatly relative to the current clock frequency, in order to avoid a large deviation in the absolute timestamp calculation, the average clock frequency within the statistical period is used to update the current clock frequency. Otherwise, the current clock frequency is not updated, thereby avoiding frequent adjustments to the clock frequency.

[0082] This is only one way to update the current clock frequency. In another implementation, it can also be set that after the average clock frequency is calculated in each statistical period, the current clock frequency is updated, which also falls within the protection scope of this application.

[0083] In step S300, the absolute timestamp is calculated based on the RTCP sender report and the corresponding current clock frequency, including using the following formula to calculate the absolute timestamp:

[0084]

[0085] Among them, RTCP report refers to RTCP sender report. The frame RTP timestamp is the RTP timestamp in the RTP packet of the frame for which the absolute timestamp needs to be calculated, and the RTP timestamp and NTP timestamp of the RTCP report refer to a pair of RTP timestamps and NTP timestamps contained in the RTCP sender report (corresponding to the same video frame, which is in the same statistical period as the frame for which the absolute timestamp needs to be calculated). Furthermore, if there are multiple RTCP sender reports within the statistical period, the RTCP sender report whose sending time is closest to the receiving time of the current frame for which the absolute timestamp needs to be calculated is selected. During the reception of the picture group by the on-demand server, the RTCP sender report is received from the media resource. NTP refers to the Network Time Protocol, which is used to synchronize device clocks in a computer network to ensure that all devices have a consistent time reference. RTCP sender report (RTCP Sender Report) is a type of RTCP message used for statistical information sent by the sender report, including RTP timestamps and NTP timestamps.

[0086] In this embodiment, it can be simplified to only calculate the absolute timestamps of some video frames, thereby reducing the amount of calculation of the absolute timestamps of the video frames. In step S300, the absolute timestamp is calculated based on the RTCP sender report and the corresponding current clock frequency, including the following steps:

[0087] determining whether the buffered video frame corresponds to an absolute time synchronization checkpoint;

[0088] The absolute time synchronization checkpoint may be a time point required in a synchronous playback request sent by the client. For example, the client may specify the first frame of each picture group as a checkpoint in the request, or the first four frames in each picture group as a checkpoint, etc., and an absolute timestamp needs to be calculated for each checkpoint;

[0089] If yes, the absolute timestamp of the video frame is calculated based on the RTCP sender report and the corresponding current clock frequency.

[0090] The absolute time synchronization checkpoint is set to the first frame of each picture group, for example, that is, the absolute timestamp is calculated only for the first frame of each picture group, but the present application is not limited thereto. The setting interval of the absolute time synchronization checkpoint can also be consistent with the period of the RTCP sender report, or the interval time length can be adjusted as needed.

[0091] In the prior art, some video synchronization playback methods are implemented based on the client-server architecture (CS architecture). This architecture has difficulties in cross-business integration and multi-version compatibility, so a more flexible and efficient browser-server architecture (BS architecture) is needed. However, the existing BS architecture also has some defects. For example, the pseudo-BS architecture based on Web plug-in technology (such as ActiveX, NPAPI, PPAPI, etc.), its audio and video media core functions are essentially still implemented in local mode, which has obvious defects in plug-in installation experience, version compatibility and security, especially since the current mainstream browsers no longer support this technology.

[0092] HTML5 (a language description method for building Web content) introduces technologies and APIs (application programming interfaces) such as video tags, WebRTC (web RTC), MSE (media source extensions), WebAssembly (web components, a binary instruction set based on a stack-based virtual machine), and WebCodecs (an interface for low-level manipulation of audio and video on a web page). Although these technologies have improved browser support for audio and video applications, there is still a lack of interfaces and solutions for synchronous playback. If HTML5 is used to implement the synchronous playback function, there are still many difficulties. For example, the transmission, decoding, and display of video tags and WebRTC are all encapsulated by the browser, and the application layer cannot be flexibly controlled; WASM has poor decoding performance and cannot meet the requirements of many scenarios; although WebCodecs can control the frame data before and after decoding, it lacks compatibility; although MSE can independently control the transmission method, the decoding and rendering process is encapsulated by the browser, and the decoded frame data cannot be directly operated.

[0093] In this embodiment, the client is a web client. In step S400, encapsulating the buffered video frames to obtain encapsulated video files of multiple media resources includes the following steps:

[0094] The buffered video frames are encapsulated into multiple slices to obtain encapsulated video files corresponding to each media resource, wherein each encapsulated video file includes multiple slices of the media resource, each slice includes one or more picture groups, and each slice adopts the FMP4 format that complies with the ISO BMFF specification. That is, for each media resource, an encapsulated video file is obtained, and the encapsulated video file includes multiple FMP4 slices divided in units of picture groups.

[0095] When encapsulating fragments, the time base (time base, as the unit of PTS) of the fragment is equal to 1 divided by the current clock frequency (the current clock frequency of the corresponding media resource obtained in the above step S200), and the PTS (display timestamp) value of the video frame in the fragment uses the RTP timestamp.

[0096] The current time attribute of each frame used by the client when playing the fragment is actually obtained by subtracting the PTS timestamp of the first frame from the PTS timestamp of the current frame and then dividing it by the current clock frequency, that is, the relative time of the video frame to the first frame of the fragment.

[0097] The present invention realizes the low-latency playback requirement of the Web by adopting HTML5's MSE (Media Source Extension) extended video source technology and combining it with a segment transmission scheme based on GOP. Since MSE requires the input segments to conform to the format of the ISO BMFF byte stream, the segments are encapsulated in a format that conforms to the ISO BMFF specification. In order to maximize cross-browser compatibility, the FMP4 format that conforms to the ISO BMFF specification is specifically adopted. A media segment in the FMP4 format consists of moof, mdata, and optional styp and sidx top-level boxes. The initialization segment Initialization Segmen consists of ftyp and moov top-level boxes. MSE is an HTML extension technical specification developed by the W3C organization and has been basically supported by mainstream browsers. It allows JavaScript to dynamically construct <audio>and <video>MSE defines a MediaSource object, which can be used as the media data source of the HTML video element HTMLMediaElement.

[0098] In step S400, the encapsulated video file and the absolute timestamp list of the video frames therein are sent to the client, including: transmitting each FMP4 segment in the encapsulated video file and the absolute timestamp list of the video frames therein to the web client. Here, the absolute timestamp list may include an absolute timestamp list for each encapsulated video file, or an absolute timestamp list may be generated for one or more segments.

[0099] In some embodiments, the web client may obtain the reverse order segments from the on-demand server piece by piece using HTTP Get. In other embodiments, the web client may also establish a websocket (a network communication protocol, a protocol for full-duplex communication on a single TCP connection provided by HTML5) long connection with the on-demand server, and the on-demand server may push the segments to the web client piece by piece.

[0100] In this embodiment, the web client performs synchronous playback according to the absolute timestamp list. In the client synchronous playback, three objects, the media source buffer SourceBuffer object, the media source mediaSource object and the HTML video element HTMLMediaElement object, are used to play the media content. <video>or <audio>The data of the tag provides functions such as play, pause, jump, etc. to control media playback. The media source object is an interface that allows JavaScript to dynamically provide media data to the HTML video element object. Through the media source object, one or more media source buffer objects can be created and associated with the HTML video element. The media source buffer is a buffer used to store and manage media data fragments. Through the media source object, the media source buffer object can add media data fragments (such as video and audio) to the buffer and pass them to the HTML video element object for playback on demand. Simply put, the media source object manages media data through the media source buffer component and passes this data to the HTML video element object for playback. The HTML video element object is responsible for the actual media playback, while the media source object and the media source buffer object are responsible for dynamically loading and managing media data.

[0101] The Web client inputs the segments in the ISO BMFF byte stream format to the pre-created MSE SourceBuffer object. At this time, the segments are called Media Segments in MSE. Before inputting the first Media Segment, an Initialization Segment (including audio and video media parameters) must be input to initialize MSE. Both the Initialization Segment and the Media Segment are encapsulated in ISO BMFF format.

[0102] like Figure 3 As shown, after receiving the packaged video files of multiple media resources, the client is configured to synchronously play back the video frames in the packaged video files using the following steps:

[0103] S510: Determine the first frame with the smallest absolute timestamp according to the absolute timestamp list, use the encapsulated video file to which the first frame belongs as a reference file, and use the smallest absolute timestamp as a reference time;

[0104] For example, if videos from three cameras need to be played back synchronously on the client, it is equivalent to having packaged video files of three devices to be transmitted to the client. The packaged video files of the three devices are file A, file B, and file C respectively. The first fragments of the packaged video files of the three devices are fragment A1, fragment B1, and fragment C1 respectively. The absolute time list corresponding to each file includes the absolute time of each video frame in the file; according to the absolute time list of the three files, the fragment with the smallest absolute time of the first frame is searched. Assuming that the absolute time of the first frame of fragment A1 is the smallest, the packaged video file A to which fragment A1 belongs is used as the reference file, and the absolute timestamp of its first frame is used as the reference time;

[0105] S520: Calculate the time offset of the first frame of other packaged video files except the reference file relative to the reference time;

[0106] The time offset here is the time offset of the first frame of each non-baseline encapsulated video file compared to the first frame in the base file, which is denoted as Δt below. For example, the time offset of the first frame of other encapsulated video files (file B and file C) relative to the base time is calculated. The time offset of the first frame of file B relative to the base time is, for example, 5 seconds, that is, file B is 5 seconds later than file A. The time offset of the first frame of file C relative to the base time is, for example, 10 seconds, that is, file C is 10 seconds later than file A.

[0107] S530: Input the segments into a pre-created media source buffer object, create a video element object for each encapsulated video file, and associate the video element object with the media source buffer object;

[0108] For example, add an initialization segment, then add each segment in the encapsulated video file A, each segment in the encapsulated video file B, and each segment in the encapsulated video file C to the pre-created media source buffer object of the MSE, create an HTML video element object for each encapsulated video file, and associate the HTML video element object with the media source buffer object;

[0109] S540: Initialize the current time attribute of the video element object of the reference file, and make the video element object of the reference file enter the playing state, and the video element objects of other encapsulated video files enter the pause state;

[0110] The segments in the base file will start playing in sequence, while other packaged video files have not yet started playing;

[0111] For example, the current time attribute of the HTML video element object of file A is initialized to 0, so that the HTML video element object of file A enters the playing state, and the HTML video element objects of files B and C enter the paused state;

[0112] S550: when the reference file is played to the time point corresponding to the time offset of the other encapsulated video files, the current time attribute of the video element objects of the other encapsulated video files is initialized, and the video element objects of the other encapsulated video files enter the playing state;

[0113] For example, set a synchronous playback timer, use the setInteval command or requestAnimationFrame to set the timer, and periodically check the current playback time. When the base file A plays to 5 seconds, start playing file B, first set the current time attribute of file B to 0, and make the HTML video element object of file B enter the playback state. When the base file plays to 10 seconds, start playing file C, first set the current time attribute of file C to 0, and make the HTML video element object of file C enter the playback state.

[0114] In this embodiment, the client realizes synchronous playback by continuously modifying the current time attribute of the HTML video element object. For example, after the client makes the video element objects of other encapsulated video files enter the playback state, it also includes the following steps to control the synchronous playback of the encapsulated video files of multiple media resources:

[0115] Determine the original frame interval time according to the current speed;

[0116] For example, when the current speed is 1, the uniform speed playback mode is used. When calculating the original frame interval, 1000 milliseconds is divided by the frame rate. Assuming the frame rate is 30fps, the original frame interval is 1000 / 30≈33.33 milliseconds. If the speed is not 1, the original frame interval is 1000 milliseconds divided by the frame rate and then divided by the speed. For example, assuming the frame rate is 30fps and the speed is 2x, the original frame interval is 1000 / (30*2)≈16.67 milliseconds.

[0117] Determine the event interval of the synchronous playback timer according to the original frame interval;

[0118] The event interval can be set to be the same as the original frame interval, so that each frame is recalibrated to ensure the synchronization of each segment during playback; or the event interval can be set to a multiple of the original frame interval, which can be set as needed;

[0119] At every event interval, the current time attribute of the video element object of the currently playing encapsulated video file is increased by the event interval;

[0120] For example, the event interval of the timing event of the synchronous playback timer is set to control the increase time of the current time attribute. When the event interval is 0.033 seconds, the current time attribute of the video element object of the currently playing encapsulated video file is increased by 0.033 seconds every 0.033 seconds.

[0121] After each segment is played, the next segment of the encapsulated video file corresponding to the segment is loaded and played. If the subsequent segments in the encapsulated video file of a media resource are not received in time, the playback of the encapsulated video files of all media resources is paused until the subsequent segments in the encapsulated video files of all media resources are ready, and then the playback is resumed.

[0122] The client can also jump to play according to the random drag and drop operation of the user. Specifically, according to the time node input by the user, the current time attribute of the HTML video element object of the reference file is set, and the current time attribute of the HTML video element object of other encapsulated video files is calculated by subtracting the time offset calculated in the above step S520 from the current time attribute of the HTML video element object of the reference file.

[0123] In this embodiment, due to the timer accuracy problem, cumulative errors may be introduced, and the execution of the timer may be delayed when the CPU load is high. At the same time, time-consuming operations will also affect the execution time of the timer, thereby affecting the playback synchronization between multiple media resources. The client further uses current time calibration to ensure the synchronization of the playback of encapsulated video files of multiple media resources. The current time calibration is performed based on a specific calibration period, which can be performed each time the current time attribute of other non-benchmark encapsulated video files is modified, or other calibration periods can be set, for example, other non-benchmark encapsulated video files are performed after each continuous playback of n (n>1) video frames. Each time the current time calibration is performed, for other non-benchmark encapsulated video files, the current time attribute t11 of the current playback frame of the benchmark file is calculated with the current time attribute t12 of the other encapsulated video files. The current time attribute difference Δt1 is compared with the time offset Δt obtained in step S520, and the deviation value is calculated as |Δt1-Δt|. If the deviation between the current time attribute difference and the time offset |Δt1-Δt| exceeds the first deviation threshold, it is necessary to accelerate, decelerate or pause the other non-benchmark encapsulated video files to ensure synchronization. For example, when the current time attribute difference Δt1 between benchmark file A and file B is greater than the initial time offset Δt between benchmark file A and file B, and the deviation |Δt1-Δt| between the two exceeds the first deviation threshold, the playback of file B is somewhat delayed, and the playback of file B is appropriately accelerated; if the current time attribute difference Δt1 between benchmark file A and file B is less than the initial time offset Δt between benchmark file A and file B, and the deviation |Δt1-Δt| between the two exceeds the first deviation threshold, the playback of file B is somewhat too fast, and the playback of file B needs to be decelerated or paused. Setting the first deviation threshold can avoid repeated and frequent adjustments to the playback speed of other encapsulated video files, which affects the smoothness of video playback.

[0124] In this embodiment, the client also uses absolute time calibration to ensure the synchronization of the playback of the packaged video files of multiple media resources. The absolute time calibration is performed in a specific calibration period, for example, the absolute time calibration is performed when the first frame of a segment of each reference file is played, or the calibration period is set to calibrate once for multiple segments, and the calibration period of the absolute time calibration can also be consistent with the sending period of the RTCP sender report. Each execution of the absolute time calibration includes: obtaining the video frame currently being played of the encapsulated video file of each media resource, determining the frame number of the currently played video frame in the encapsulated video file according to the current time attribute of the currently played video frame (that is, determining which frame of the currently played video frame in the encapsulated video file is), finding the absolute timestamp of the currently played video frame in the absolute timestamp list according to the frame number, and comparing whether the absolute timestamps of the frames currently being played by multiple media resources are consistent. If they are consistent, it means that the encapsulated video files of multiple media resources are played synchronously and the time is aligned with each other. If they are inconsistent and the deviation exceeds the second deviation threshold, it is necessary to adjust the playback speed of the non-benchmark encapsulated video file. For example, if the absolute time of the frame currently played by file B is later than the absolute time of the frame currently played by benchmark file A, and the deviation between the two exceeds the second deviation threshold, it means that file B is played slowly and file B needs to be accelerated. If the absolute time of the frame currently played by file B is earlier than the absolute time of the frame currently played by benchmark file A, and the deviation between the two exceeds the second deviation threshold, it means that file B is played too fast and file B needs to be slowed down or paused. Setting the second deviation threshold can avoid repeatedly and frequently adjusting the playback speed of other encapsulated video files, thereby affecting the smoothness of video playback.

[0125] The above two calibration methods of current time calibration and absolute time calibration can be selected as needed. For example, only one of the current time calibration or the absolute time calibration can be selected as needed, or the current time calibration and the absolute time calibration can be used at the same time, and the calibration frequency of the current time calibration is higher than the calibration frequency of the absolute time calibration, etc., all of which are within the protection scope of this application.

[0126] Therefore, by adopting the control of synchronous playback by the client in the video synchronous playback method, based on the media source extension technology of HTML5, combined with the fragmented transmission of picture groups, the low-latency playback requirements of web are achieved, making the business integration flexible and efficient, and the experience is simple and convenient. In addition, the playback control of HTML video element objects is driven by clock frequency, relative time, absolute time, various calibration strategies, etc., to achieve the synchronous playback function based on HTML5. In this synchronous playback method, there is no need to transcode and transfer, and no need to add additional computing resources and storage resources. Video data can be optionally transmitted in a streaming manner, which occupies less cache and memory, and has a fast image processing response.

[0127] like Figure 4 As shown, the embodiment of the present application further provides a video synchronous playback system, which is applied to the above-mentioned video synchronous playback method. The system includes a video-on-demand server M100 and a client M200. The video-on-demand server M100 includes:

[0128] The video receiving module M110 is used to receive and buffer video frames from multiple media resources;

[0129] The clock frequency calculation module M120 is used to determine the current clock frequency corresponding to each media resource for each media resource; wherein, within a statistical period, the average clock frequency within the statistical period is calculated based on the total receiving time within the statistical period and the RTP timestamp difference, and the current clock frequency is determined according to the average clock frequency within the statistical period;

[0130] An absolute time calculation module M130, for calculating an absolute timestamp for the buffered video frame based on an RTCP sender report and a corresponding current clock frequency;

[0131] The encapsulation and sending module M140 is used to encapsulate the buffered video frames to obtain encapsulated video files of multiple media resources, and send the encapsulated video files and the absolute timestamp list of the video frames therein to the client;

[0132] The client M200 is configured to synchronously play back the video frames in the encapsulated video files according to the absolute timestamp list after receiving the encapsulated video files of multiple media resources.

[0133] By adopting the video synchronous playback system of the present application, after the on-demand server M100 adopts the video receiving module M110 to obtain video frames of multiple media resources, it adopts the clock frequency calculation module M120 to calculate the average clock frequency within a statistical period based on the total receiving time and the RTP timestamp difference within the statistical period, and then determines the current clock frequency without relying on the SDP message. For media resources whose clock frequency cannot be obtained from the SDP message (such as video streams that are not RTP timestamped according to the clock frequency in the SDP message and video streams that lack clock frequency information), it is also possible to obtain accurate clock frequencies, and then adopt the absolute time calculation module M130 to accurately calculate the absolute timestamp used for synchronous playback, so that it can be well compatible with various types of media resource video data. After the encapsulated video file and the absolute timestamp list are sent to the client by the encapsulation and sending module M140, the client can synchronously play back the videos of multiple media resources according to the absolute timestamp list, thereby improving compatibility and adaptability and achieving high-precision synchronous playback of multiple media resource videos.

[0134] The following uses the architecture of the video synchronous playback system as an example to introduce the specific implementation method of synchronous playback. Among them, the client is a web client.

[0135] The Web client initiates a synchronous playback request to the VOD server, wherein the playback request at least includes information such as a media resource URI (Uniform Resource Identifier, a string used to identify a certain Internet resource name), a start time of the media resource to be played back, and the like.

[0136] Corresponding to the above step S100, after receiving the request to start playback, the on-demand server requests the camera or the lower-level recording platform to play the video, and buffers a certain amount of frame data to execute the above step S200 to obtain the current clock frequency. The on-demand server will receive the video frame RTP packet and RTCP sending report from the camera and the recording platform, and buffer the video frame. S200 is continuously performed in the background for each statistical period. When the difference between the calculated average clock frequency in the statistical period and the current clock frequency is greater than the preset difference threshold, the current clock frequency is updated, otherwise the current clock frequency remains unchanged.

[0137] According to step S300, the absolute timestamp of the video frame is calculated. Specifically, only the absolute timestamp of the checkpoint may be calculated, or the absolute timestamp of all the video frames may be calculated.

[0138] According to step S400, the FMP4 slicer is used to encapsulate the slices to obtain the encapsulated video files corresponding to each media resource. The PTS and DTS (decoding timestamp) of the frame use the timestamp of the RTP packet (PTS and DTS are usually equal in security scenarios because there are no B frames). The time base is equal to 1 divided by the current clock frequency. The encapsulation process of the FMP4 slicer is as follows: the on-demand server reads the SPS and PPS parameter sets of the media resources and other encoding and decoding parameter information to initialize the slicer, and sets the slice length of the slicer to the time length of one or more picture groups, and then inputs the read video frame data into the FMP4 slicer frame by frame. When the slice length is met, the slicer will output an FMP4 slice. The generated FMP4 format slice is called a Media Segment slice, which is composed of three top-level boxes: sidx, moof, and mdata. When the first segment is output, an initialization segment Initialization Segment is also output first. The initialization segment contains the encoding and decoding parameter information such as the SPS and PPS parameter sets of the aforementioned media resources, and is composed of two top-level boxes, ftyp and moov.

[0139] According to step S400, the encapsulated video files and the absolute time list of multiple media resources are transmitted to the web client. For better management, auxiliary information such as the segment number and the segment duration can be added.

[0140] According to steps S510 to S550, the client synchronously plays back the packaged video files of multiple media resources.

[0141] An embodiment of the present application also provides a video synchronization playback device, comprising a processor; a memory, in which executable instructions of the processor are stored; wherein the processor is configured to perform the steps of the video synchronization playback method by executing the executable instructions.

[0142] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system".

[0143] Refer to the following Figure 5 The electronic device 600 according to this embodiment of the present application is described. Figure 5 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0144] like Figure 5 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0145] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various exemplary embodiments of the present application described in the above-mentioned video synchronous playback method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0146] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0147] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0148] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0149] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0150] By adopting the video synchronous playback device provided by the present application, the processor executes the video synchronous playback method when executing the executable instructions, thereby obtaining the beneficial effects of the above-mentioned video synchronous playback method.

[0151] The embodiment of the present application also provides a computer-readable storage medium for storing a program, and when the program is executed by a processor, the steps of the video synchronous playback method are implemented. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present application described in the above-mentioned video synchronous playback method section of this specification.

[0152] refer to Figure 6 As shown, a program product 800 for implementing the above method according to an embodiment of the present application is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0153] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0154] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0155] Program codes for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or cluster. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0156] By adopting the computer-readable storage medium provided by the present application, the program stored therein implements the steps of the video synchronous playback method when being executed, thereby obtaining the beneficial effects of the above-mentioned video synchronous playback method.

[0157] The above content is a further detailed description of the present application in combination with specific preferred implementation methods, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. For ordinary technicians in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the concept of the present application, which should be deemed to fall within the scope of protection of the present application.< / audio> < / video> < / video> < / audio>

Claims

1. A video synchronous playback method, characterized in that: The steps include: Receive and buffer video frames from multiple media sources; For each media resource, determine the current clock frequency corresponding to the media resource; wherein, within a statistical period, the average clock frequency within the statistical period is calculated based on the total receiving time within the statistical period and the RTP timestamp difference, and the current clock frequency is determined according to the average clock frequency within the statistical period; For the buffered video frame, calculating an absolute timestamp based on the RTCP sender report and the corresponding current clock frequency; The encapsulated buffered video frames are encapsulated to obtain encapsulated video files of multiple media resources, and the encapsulated video files and an absolute timestamp list of the video frames therein are sent to a client. The client is configured to synchronously play back the video frames in the encapsulated video files according to the absolute timestamp list after receiving the encapsulated video files of multiple media resources.

2. The video synchronous playback method according to claim 1, characterized in that: Determining the current clock frequency according to the average clock frequency in the statistical period includes the following steps: Determine whether the difference between the average clock frequency in the statistical period and the current clock frequency is greater than a preset difference threshold; If yes, the current clock frequency is updated using the average clock frequency within the statistical period; If not, keep the current clock frequency unchanged.

3. The video synchronous playback method according to claim 1, characterized in that: The time length of the statistical period is the time length of one or more picture groups, and the first frame of the statistical period corresponds to the first frame of a picture group, or the time length of the statistical period is equal to the time length of the period reported by the RTCP sender.

4. The video synchronous playback method according to claim 1, characterized in that: Calculate the absolute timestamp based on the RTCP sender report and the corresponding current clock frequency, including the following steps: determining whether the buffered video frame corresponds to an absolute time synchronization checkpoint; If yes, the absolute timestamp of the video frame is calculated based on the RTCP sender report and the corresponding current clock frequency.

5. The video synchronous playback method according to claim 1, characterized in that: The client is a web client; The encapsulation of the buffered video frames to obtain encapsulated video files of multiple media resources comprises the following steps: The buffered video frames are encapsulated into multiple fragments to obtain encapsulated video files corresponding to each of the media resources, wherein each of the encapsulated video files includes multiple fragments, each fragment includes one or more picture groups, and each fragment adopts the FMP4 format that complies with the ISO BMFF specification.

6. The video synchronous playback method according to claim 5, characterized in that: The client is configured to synchronously play back the video frames in the encapsulated video file by using the following steps after receiving the encapsulated video file of multiple media resources: According to the absolute timestamp list, determine the first frame with the smallest absolute timestamp, use the encapsulated video file to which the first frame belongs as the reference file, and use the smallest absolute timestamp as the reference time; Calculate the time offset of the first frame of other packaged video files except the reference file relative to the reference time; Input each segment in the encapsulated video file into a pre-created media source buffer object, create a video element object for each encapsulated video file, and associate the video element object with the media source buffer object; Initialize the current time attribute of the video element object of the reference file, and make the video element object of the reference file enter the playing state, and the video element objects of other encapsulated video files enter the pause state; When the reference file is played to a time point corresponding to a time offset of other encapsulated video files, the current time attribute of the video element objects of other encapsulated video files is initialized, and the video element objects of other encapsulated video files enter a playing state.

7. The video synchronous playback method according to claim 6, characterized in that: After the client makes the video element objects of other encapsulated video files enter the playing state, the following steps are also used to control the synchronous playback of the encapsulated video files of multiple media resources: Determine the original frame interval time according to the current speed; Determine an event interval time of a synchronous playback timer according to the original frame interval time; At every event interval, the current time attribute of the video element object of the currently playing encapsulated video file is increased by the event interval.

8. A video synchronous playback system, characterized in that: The video synchronous playback method applied to any one of claims 1 to 7, wherein the system comprises a video-on-demand server and a client, and the video-on-demand server comprises: A video receiving module, used for receiving and buffering video frames from multiple media resources; A clock frequency calculation module is used to determine the current clock frequency corresponding to each media resource for each media resource; wherein, within a statistical period, the average clock frequency within the statistical period is calculated based on the total receiving time within the statistical period and the RTP timestamp difference, and the current clock frequency is determined according to the average clock frequency within the statistical period; An absolute time calculation module, configured to calculate an absolute timestamp for the buffered video frame based on an RTCP sender report and a corresponding current clock frequency; An encapsulation and sending module, used for encapsulating the buffered video frames to obtain encapsulated video files of multiple media resources, and sending the encapsulated video files and a list of absolute timestamps of the video frames therein to a client; The client is configured to synchronously play back the video frames in the encapsulated video files according to the absolute timestamp list after receiving the encapsulated video files of multiple media resources.

9. A video synchronous playback device, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to execute the steps of the video synchronous playback method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the video synchronous playback method described in any one of claims 1 to 7 are implemented.

Images