Method and apparatus for processing multi-path video stream, storage medium and electronic device

Abstract

The application discloses a processing method and device of multi-path video stream, a storage medium and an electronic device. The method comprises the following steps: acquiring multi-path video stream for real-time playing of a live picture; performing encoding processing on the multi-path video stream, and adding a global timestamp to a data storage unit sequence corresponding to each frame of image after encoding; determining a plurality of transmission delay times of a plurality of frames of images with the same global timestamp in the multi-path video stream based on the global timestamp and a local time when each frame of image after encoding is transmitted to a playing buffer; and controlling the multi-path live picture corresponding to the multi-path video stream to be synchronously played based on the plurality of transmission delay times. The application solves the technical problem that the frames of the multi-path video stream cannot be synchronously played at the playing end.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method and apparatus for processing multiple video streams, a storage medium, and an electronic device. Background Technology

[0002] In current multi-view video stream synchronization processing, it is usually based on the management and playback of independent video streams, such as each video stream being transmitted and decoded based on its own timestamp. This approach only focuses on the synchronization of audio and video within a single video stream, but cannot ensure the synchronization of video from multiple video streams at the playback end.

[0003] In scenarios such as cloud-based live streaming, "bullet time" sports broadcasts, and panoramic monitoring, users need to simultaneously switch between or quickly switch between multiple video streams from different perspectives. Using the traditional multi-stream processing methods described above can lead to "time gaps" and "switching abruptness," impacting user experience. Furthermore, in unstable network environments, each video stream may experience varying degrees of network jitter, further exacerbating display differences between the live streams. This results in a technical problem where frame-level synchronization of multiple video streams is impossible at the playback end.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method and apparatus for processing multiple video streams, a storage medium, and an electronic device to at least solve the technical problem that the images of multiple video streams cannot be synchronized at the frame level at the playback end.

[0006] According to one aspect of the embodiments of this application, a method for processing multiple video streams is provided, comprising: acquiring multiple video streams for real-time playback of live broadcast images; encoding the multiple video streams and adding a global timestamp to the data storage unit sequence corresponding to each encoded frame image; determining multiple transmission delay times for multiple frames with the same global timestamp in the multiple video streams based on the global timestamps and the local time when each encoded frame image is transmitted to the playback buffer; and controlling the synchronous playback of multiple live broadcast images corresponding to the multiple video streams based on the multiple transmission delay times.

[0007] Optionally, the above-mentioned encoding process for multiple video streams and adding a global timestamp to the data storage unit sequence corresponding to each encoded frame image includes: encoding each frame image in the multiple video streams to obtain multiple data storage unit sequences; adding a global timestamp to the target storage unit in each data storage unit sequence, wherein the target storage unit has a specific storage area required to store supplementary enhancement information, and the supplementary enhancement information includes custom image description information.

[0008] Optionally, adding a global timestamp to the target storage unit in each data storage unit sequence includes: constructing a target storage unit for writing the global timestamp; encapsulating the global timestamp into a specific storage area in the target storage unit; and inserting the encapsulated target storage unit before the image data storage unit in each data storage unit sequence to obtain each data storage unit sequence with the global timestamp added, wherein the image data storage unit is used to store the encoded data of the image itself.

[0009] Optionally, the above-mentioned determination of multiple transmission delay times for multiple frames with the same global timestamp in multiple video streams based on the global timestamp and the local time when each encoded frame image is transmitted to the playback buffer includes: in response to the received encoded multiple video streams, parsing the data storage unit sequence corresponding to each frame image to obtain a global timestamp; obtaining the first frame image of the first video segment in each video stream to obtain multiple first frame images, wherein the multiple frame images include multiple first frame images, and the global timestamps of the multiple first frame images are the same; determining multiple transmission delay times based on the multiple global timestamps in the multiple first frame images and the multiple local times when the encoded data corresponding to the multiple first frame images is transmitted to the playback buffer.

[0010] Optionally, the above-mentioned parsing of the data storage unit sequence corresponding to each frame image to obtain a global timestamp includes: traversing the data storage unit sequence and parsing the boundary of each data storage unit to obtain the type of each data storage unit; based on the type, identifying the target storage unit with supplementary enhancement information from the data storage unit sequence; and extracting the global timestamp from the target storage unit.

[0011] Optionally, the above-mentioned determination of multiple transmission delay times based on multiple global timestamps in multiple first-frame images and multiple local times when the encoded data corresponding to the multiple first-frame images is transmitted to the playback buffer includes: sequentially obtaining each first-frame image from multiple first-frame images as the current first-frame image; finding the current local time corresponding to the current first-frame image from multiple local times; determining a first time difference between the current local time and the current global timestamp of the current first-frame image; and determining a second time difference between the first time difference and the delay time constant as the current transmission delay time.

[0012] Optionally, the above-mentioned method of controlling the synchronous playback of multiple live broadcast images corresponding to multiple video streams based on multiple transmission delay times includes: determining the maximum transmission delay time from multiple transmission delay times; determining the sum of the current time point and the maximum transmission delay time as the target playback time; and controlling the synchronous playback of multiple live broadcast images at the target playback time.

[0013] Optionally, the above method further includes: when there is a portion of the transmission delay time that exceeds a time threshold among multiple transmission delay times, adjusting the audio playback speed of a portion of the video stream corresponding to the portion of the transmission delay time; and synchronously adjusting the video playback speed of the portion of the video stream based on the adjustment parameter of the audio playback speed to obtain the adjusted delay time, wherein the adjusted delay time is less than or equal to the maximum transmission delay time.

[0014] According to another aspect of the embodiments of this application, a multi-channel video stream processing apparatus is also provided, comprising: a first acquisition unit, configured to acquire multiple video streams for real-time playback of live broadcast images; an encoding unit, configured to encode the multiple video streams and add a global timestamp to the data storage unit sequence corresponding to each encoded frame image; a first processing unit, configured to determine multiple transmission delay times of multiple frames with the same global timestamp in the multiple video streams based on the global timestamps and the local time when each encoded frame image is transmitted to the playback buffer; and a second processing unit, configured to control the synchronous playback of multiple live broadcast images corresponding to the multiple video streams based on the multiple transmission delay times.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program for executing the above-described multi-video stream processing method when the electronic device is run.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0017] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute the multi-channel video stream processing method through the computer program.

[0018] By employing the embodiments provided in this application, by introducing global timestamps of each video stream at the acquisition end and encapsulating them into the encoded data storage unit, the absolute timestamps used to calibrate the transmission and playback times of each video stream can still be parsed even when there are differences in transmission delays among the video streams. At the playback end, based on the parsed global timestamps and the local time (or system time) at which each video stream is transmitted to the player, the transmission delay time of each video stream is accurately calculated. Based on multiple transmission delay times, the target playback time that enables frame-level synchronization of multiple video streams is determined. In other words, by introducing global timestamps of each video stream at the acquisition end, the transmission and playback timelines of each video stream can be calibrated with sub-microsecond precision, thereby ensuring that video streams from different perspectives can achieve precise frame-level synchronization at the playback end, realizing seamless picture transitions and improving the picture synchronization of multiple video streams in real-time playback scenarios. Attached Figure Description

[0019] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0020] Figure 1 This is a schematic diagram illustrating an application scenario of an optional multi-channel video stream processing method according to an embodiment of this application;

[0021] Figure 2 This is a flowchart of an optional multi-channel video stream processing method according to an embodiment of this application;

[0022] Figure 3 This is an overall architecture diagram of an optional multi-channel video stream processing method according to an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of an optional injection of SEI information according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the control logic of an optional synchronous controller according to an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the structure of an optional multi-channel video stream processing apparatus according to an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] The technical solutions in this application will comply with legal regulations during implementation. When operating according to the technical solutions in the embodiments, the data used will not involve user privacy, ensuring that the operation process is compliant and legal while guaranteeing data security. In addition, when the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use, and processing of related data must comply with the relevant regulations and standards of the relevant countries or regions.

[0030] According to one aspect of the embodiments of this application, a method for processing multiple video streams is provided. As an optional implementation, the above-described method for processing multiple video streams can be applied, but is not limited to, to applications such as... Figure 1 The application scenarios shown are as follows. In, for example... Figure 1 In the application scenario shown, the target terminal 102 can communicate with the server 106 via network 104, but is not limited to this. The server 106 can perform operations on the database 108, such as write or read data operations. The target terminal 102 may include, but is not limited to, a human-computer interaction screen, a processor, and a memory. The human-computer interaction screen may be used to display a multi-view live broadcast image after synchronous rendering of multiple video streams on the target terminal 102. The processor may be used to respond to the human-computer interaction operations, execute corresponding operations, or generate corresponding instructions and send the generated instructions to the server 106. The memory is used to store relevant processing data, such as global timestamps, multi-frame images, and multiple transmission delay times.

[0031] Optionally, in this embodiment, the target terminal can be a terminal configured with a target client, which may include, but is not limited to, at least one of the following: mobile phone (such as Android phone, iOS phone, etc.), laptop computer, tablet computer, PDA, MID (Mobile Internet Devices), PAD, desktop computer, smart TV, etc. The target client may be a video client, instant messaging client, browser client, educational client, etc. The network may include, but is not limited to, wired network and wireless network, wherein the wired network includes: local area network, metropolitan area network and wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that enable wireless communication. The server may be a single server, a server cluster composed of multiple servers, or a cloud server.

[0032] The technical solution presented in this application can be widely applied to multi-view real-time live streaming scenarios, such as "bullet time" live streaming of sports events, multi-view live streaming of concerts, and VR / AR metaverse content. It solves the problem of switching and jumping between multi-view live stream images caused by network transmission jitter and bitstream transmission errors, thus improving the user's viewing experience.

[0033] As described in the above embodiments, when applying traditional multi-stream synchronization processing methods to real-time live streaming scenarios, a technical problem easily arises where multiple live streams cannot be synchronized at the frame level on the playback end. To address this problem, this application proposes a multi-stream processing method. Figure 2 This is a flowchart of a multi-channel video stream processing method according to an embodiment of this application, which includes the following steps S202 to S208.

[0034] It should be noted that the multi-channel video stream processing method shown in steps S202 to S208 can be, but is not limited to, executed by an electronic device. The electronic device can be, but is not limited to, as shown in... Figure 1 The target terminal or server shown.

[0035] Step S202: Obtain multiple video streams for real-time playback of the live broadcast;

[0036] Step S204: Encode the multiple video streams and add a global timestamp to the data storage unit sequence corresponding to each encoded frame.

[0037] Step S206: Based on the global timestamp and the local time when each encoded frame image is transmitted to the playback buffer, determine multiple transmission delay times for multiple frames with the same global timestamp in the multiple video streams.

[0038] Step S208: Based on multiple transmission delay times, control the synchronous playback of multiple live broadcast images corresponding to multiple video streams.

[0039] To facilitate understanding, let's first combine... Figure 3 The overall architecture diagram is shown, and the processing method of the above-mentioned multi-view live broadcast is briefly described using the display of multi-view real-time live broadcast as an example.

[0040] like Figure 3 As shown, the overall system architecture includes, but is not limited to, the processing at the acquisition and playback ends. The acquisition end primarily involves acquiring multiple video streams through different acquisition devices (such as cameras) and injecting global timestamps.

[0041] The aforementioned multiple video streams can be, but are not limited to, multiple video streams of live footage from the same location and the same real scene from different perspectives, or multiple video streams containing live footage of multiple real scenes from different locations. For example, multiple video streams from multiple program recording sites used for simultaneous broadcast.

[0042] Step 1: At the acquisition end, the following processing can be performed, but is not limited to:

[0043] (1) PTP timestamp synchronization: All streaming nodes (encoders / cameras) can, but are not limited to, synchronize with the master clock via the IEEE1588 PTP protocol to ensure that the local system time is highly consistent and the synchronization accuracy reaches the sub-microsecond level.

[0044] (2) SEI Encapsulation: When the video encoder generates each frame, it obtains the PTP timestamp of the current moment (accurate to microseconds), constructs an H.264 / H.265 SEI NAL Unit (i.e., an SEI-type data storage unit), writes the timestamp into the user_data_unregistered payload of the SEI-type data storage unit, and inserts the SEI NAL Unit before the Slice NAL Unit of the corresponding video frame. The Slice NAL Unit is a NAL unit that contains the encoded data of a local area of ​​the image.

[0045] It should be noted that the PTP timestamp (or global timestamp) is injected into the SEI NAL unit in the NAL unit sequence corresponding to each frame of the encoded image. That is, the encoded data of each frame of the image carries the PTP timestamp.

[0046] Step 2: After transmitting each encoded frame of image carrying the PTP timestamp to the browser (i.e., the playback end), the playback end performs SEI parsing and global timestamp extraction, as follows:

[0047] (1) WebRTC Insertable Streams: On the browser side, the WebRTC Insertable Streams API (or Encoded Transform API) is used, which allows JavaScript to directly access the RTCEncodedVideoFrame before the decoder decodes it.

[0048] (2) Parsing logic: Read the binary data of the video frame, traverse the NAL Unit Header to find the SEI type NALU, parse the SEI load to extract the PTP absolute timestamp written by the acquisition end, and append the timestamp to the metadata of the frame object and pass it down.

[0049] Step 3: Processing by the synchronization controller based on "target delay anchoring". Specifically, this includes:

[0050] (1) Establish a virtual global clock: The client maintains a virtual "playback time";

[0051] (2) Straggler Identification: The controller monitors the buffer status of each video stream in real time, calculates the current end-to-end delay of each stream, and identifies the stream with the largest delay (the slowest stream).

[0052] (3) Calculate the target playback delay: Set the global target playback delay, including anti-jitter margin (e.g., 50ms).

[0053] (4) Frame Holding: For video streams with low latency (fast transmission), the playback time is calculated when a frame arrives. If the current time is less than the playback time, the frame is "held" in the application layer's JitterBuffer until the time is up before the frame is released for rendering.

[0054] (5) Micro-speed Catch-up: If one of the video streams suddenly falls behind due to network jitter, a catch-up mechanism is triggered. Audio processing can, but is not limited to, using the HTML5 Audio's preservesPitch=true attribute to set the playback speed to 1.05x~1.1x without changing the pitch; video processing synchronously accelerates playback until the stream delay returns to the normal range, at which point the speed is restored to 1.0x.

[0055] The key processing steps in each of the above steps will be described in detail below with reference to specific embodiments.

[0056] As can be seen from the above steps, in this embodiment, by performing temporal "alignment" processing on multiple frames with the same PTP time in multiple video streams at the acquisition end, the temporal consistency of the multiple video streams at the video frame level in the acquisition or encoding dimension is ensured. For example, suppose that at time point 1, the first frame image of each video stream in the multiple video streams is acquired synchronously, and at time point 2, the second frame image of each video stream is acquired synchronously, and so on. This ensures that multiple frames acquired synchronously at the same time point in the multiple video streams have the same global timestamp.

[0057] In other words, the technical solution of this application mainly addresses the frame-level synchronization challenges faced by WebRTC multi-view video streams at the playback end, especially the "time gaps" and "switching jumps" caused by network jitter and the lack of a time reference. By introducing a global timestamp and fine-grained calculation of transmission latency, these technical challenges are effectively solved. In the traditional WebRTC architecture, because each video stream is transmitted independently and lacks a unified clock synchronization mechanism, the latency differences between video streams are difficult to measure and compensate for accurately. This directly leads to potential time misalignment during playback of multi-view images, severely damaging the user's immersive experience. The embodiments of this application achieve true frame-level synchronization by injecting a high-precision global timestamp during the video encoding stage and employing a "target latency anchoring" strategy at the playback end.

[0058] The core of this application's embodiments lies in the end-to-end video stream frame-level synchronization mechanism. Starting from the acquisition end of multiple cameras, clock synchronization is achieved through PTP, and a SEI with a precise timestamp is added to each frame during video encoding. Then, during the transmission of the video stream to the playback end, the aforementioned timestamp helps the system understand the original capture time of each frame. At the receiving end, by parsing the SEI, the timestamp information is extracted, and combined with the local time of the playback end, the actual transmission delay of each frame is calculated. Finally, based on this delay information, a "target delay anchoring" strategy is used to dynamically adjust different video streams, ensuring that regardless of network conditions, the multi-view images ultimately presented to the user achieve strict logical time alignment, realizing precise frame-level synchronization.

[0059] The aforementioned processing mechanism effectively solves the frame-level synchronization problem of WebRTC multi-view video streams on the playback end, improving the user experience, especially in demanding scenarios such as live sports broadcasts and metaverse content creation, greatly enhancing the smoothness and immersion during multi-view switching. Through the embodiments of this application, various real-time interactive platforms can provide more stable and high-quality multi-view live streaming services, attracting more users and content creators, thereby creating direct and indirect commercial value.

[0060] As an optional example, the above encoding process for multiple video streams, and adding a global timestamp to the data storage unit sequence corresponding to each encoded frame, includes:

[0061] Each frame of the image in the multiple video streams is encoded to obtain a sequence of multiple data storage units;

[0062] A global timestamp is added to the target storage unit in each data storage unit sequence, wherein the target storage unit has a specific storage area required to store supplementary enhancement information, which includes custom image description information.

[0063] In the context of multi-view video live streaming, video stream encoding is not only a key step in reducing data volume and improving transmission efficiency, but also an indispensable part of achieving frame-level synchronization. This embodiment focuses on timestamp injection technology during the video stream encoding process to ensure that video frames carry high-precision timing information.

[0064] The following is combined with Figure 4 The SEI injection structure diagram shown illustrates the specific method of adding a global timestamp to the encoded video frame data.

[0065] (1) Video encoding initialization: When the multi-view video live streaming system is started, the video streams captured by each camera enter the encoder for format conversion to adapt to network transmission. The video encoder here uses H.264 or H.265, two widely supported video compression standards. They not only provide efficient data compression capabilities, but also have rich Supplemental Enhancement Information (SEI) functions, which can be used to carry metadata, such as timestamps.

[0066] SEI is a type of NAL unit defined in video coding standards such as H.264 / AVC and H.265 / HEVC. It is used to embed additional metadata information into the video bitstream, such as the global timestamp or metadata information of the global timestamp in the embodiments of this application. In other words, SEK can provide additional information that is helpful to the decoding process, and this additional information can be user-defined.

[0067] It should be noted that SEI information is not necessary for the video decoding process; the decoder can decode video images normally even without processing the SEI.

[0068] (2) Acquisition and encapsulation of timestamps;

[0069] A global timestamp can be added to the SEINAL cell in the encoded data of each frame of image in, but not limited to, the following ways:

[0070] Construct the target storage unit for writing the global timestamp;

[0071] Encapsulate the global timestamp into a specific storage area within the target storage unit;

[0072] Before inserting the encapsulated target storage unit into the image data storage unit in each data storage unit sequence, a sequence of each data storage unit with a global timestamp is obtained, where the image data storage unit is used to store the encoded data of the image itself.

[0073] Specifically, it includes the following steps:

[0074] S11, Construct SEI NAL Unit, that is, construct the target storage unit in the encoded data of each frame of image;

[0075] During the generation of each keyframe in the video encoding process, the encoder obtains the current precise timestamp (e.g., a 64-bit Unix microsecond timestamp) from the PTP-synchronized clock. This timestamp is then encapsulated into an SEI NAL Unit. By setting a specific payload type (5 for H.264, 39 for H.265) and writing a custom 16-byte UUID identifier into the `user_data_unregistered` payload section, the uniqueness and identifiability of the timestamp are ensured. Finally, the SEI NAL Unit containing the timestamp is inserted before the corresponding video frame Slice NAL Unit, forming a complete video stream data packet.

[0076] It should be noted that for each frame of image, its encoded data (i.e., the sequence of data storage units) contains NAL units that users can define themselves, and users can define the type of the NAL unit as SEI type.

[0077] S12, Inject a global timestamp into the data storage unit sequence;

[0078] At the output of the video encoder, each frame of encoded video data is organized into a series of data storage unit sequences (NAL sequences), each sequence representing all the encoded data of a single video frame. This series of sequences not only contains compressed data of the video content but also embeds crucial SEI information, forming a video stream with timestamps, thus preparing the necessary signals for subsequent transmission and playback synchronization.

[0079] For example, suppose in a panoramic live broadcast, four cameras capture images from different perspectives (e.g., east, south, west, and north). These cameras are all synchronized with the system clock via the PTP protocol. Whenever the encoder finishes processing a frame, such as the 10th frame from the east perspective, it immediately reads the precise timestamp from the PTP clock, for example, 1662995434123456 microseconds. Then, according to the H.264 standard, this timestamp is encapsulated into a SEI NAL Unit. By setting the payload type to 5 and using a custom UUID identifier, the SEI NAL Unit is distinguished from ordinary image data. Finally, the SEI NAL Unit containing the timestamp is integrated into the encoded data of the 10th frame and enters the transmission stage, waiting to be delivered to the playback end over the network.

[0080] S13, insert the SEI NAL Unit with the global timestamp injected into it as follows Figure 4 Before the Slice NAL Unit shown.

[0081] Based on the above analysis, accurate timestamp acquisition and SEI encapsulation are key to achieving the technical solution of this application. By combining the high-precision timestamp provided by PTP with the SEI of the video coding standard, the embodiments of this application not only achieve accurate injection of time information but also ensure the integrity of video data and the accuracy of time stamps during transmission. This method fully utilizes the flexibility of standard video compression protocols, requiring no major modifications to the underlying protocol stack, significantly increasing the technology's accessibility and practicality.

[0082] (3) Generate a sequence of data storage units carrying global timestamps.

[0083] At the output of the video encoder, each frame of encoded video data is organized into a series of data storage unit sequences, each sequence representing all the encoded data of a single video frame. This series of sequences not only contains compressed data of the video content but also embeds crucial SEI information, forming a video stream with timestamps, thus preparing necessary supplementary and enhanced information for subsequent transmission and playback synchronization.

[0084] By cleverly integrating timestamp acquisition and encoding, the video stream is given precise timestamps during the encoding process, providing strong information support for subsequent transmission delay calculations and synchronized playback control. Throughout the multi-view video live streaming process, the addition of global timestamps constitutes one of the core elements for achieving frame-level synchronization, thereby improving the synchronization performance of the video stream at the playback end and enhancing the user's immersive experience. By closely linking timestamps with video encoding data, not only are the limitations of traditional WebRTC transmission mechanisms overcome, but also strong compatibility and scalability are demonstrated.

[0085] As an optional example, the above method, based on the global timestamp and the local time when each encoded frame is transmitted to the playback buffer, determines multiple transmission delay times for multiple frames with the same global timestamp in multiple video streams, including:

[0086] In response to the received encoded multi-channel video stream, the data storage unit sequence corresponding to each frame image is parsed to obtain a global timestamp;

[0087] Obtain the first frame image of the first video segment in each video stream to obtain multiple first frame images. The multiple first frame images include multiple first frame images, and the global timestamps of the multiple first frame images are the same.

[0088] Multiple transmission delay times are determined based on multiple global timestamps in multiple first-frame images and multiple local times when the encoded data corresponding to multiple first-frame images is transmitted to the playback buffer.

[0089] Once the receiving end acquires the WebRTC video stream from multiple viewpoints, its primary task is to parse the video stream and extract the global timestamp used for synchronization control. Each frame in the video stream is encoded into a series of data units, which, in addition to compressed image data, also contain global timestamp information encapsulated through the SEI mechanism. SEI, as a standard video supplemental enhancement information, is designed to carry various non-core video content metadata, including but not limited to custom image description information.

[0090] The specific steps are as follows:

[0091] S21, Video stream parsing and timestamp extraction;

[0092] First, the decoder or media processor at the playback end needs to parse the sequence of data storage units corresponding to each frame. This process involves traversing the video stream, identifying and extracting NAL Units containing SEIs, and specifically finding the specific SEI type used to carry timestamps. If found, the SEI data needs to be parsed to extract the global timestamp closely related to the video frame.

[0093] For example, suppose during a live football broadcast, viewers simultaneously receive video streams from four different perspectives via the WebRTC protocol. On the playback end, the system begins parsing the first frame of each video stream—the first frame of the first video segment from each perspective. By identifying specific identifiers in the NAL Unit Header, it can locate SEI-type NAL Units and further parse the SEI payload to extract the global timestamp encapsulated within.

[0094] S22, Verification of the consistency of the first frame timestamp;

[0095] To determine the initial synchronization state between multiple video streams, it is necessary to compare the global timestamps carried by the first frame of the first video segment in each stream. Theoretically, these first frames should be captured at the same time point, and therefore their timestamps should be identical. By verifying the consistency of the first frame timestamps, it can be confirmed whether the video streams have undergone correct clock synchronization at the source.

[0096] Continuing with the example of the football live stream, let's assume that the encoder encapsulates the same timestamp for the first frame of each viewpoint at the start of the live stream, for example, 1689831756345678 microseconds. The playback end confirms that the timestamps of all the first frames are indeed consistent by parsing the SEI information, which preliminarily verifies the synchronization quality of the multi-view video stream.

[0097] S23, Delay time calculation.

[0098] Based on the extracted global timestamp and the actual reception time (local time) of the video frame in the playback buffer, the transmission delay of each video stream is calculated. This is achieved through simple mathematical operations: subtracting the global timestamp from the local time and adding a certain transmission offset compensation to obtain the true end-to-end transmission delay.

[0099] In this embodiment, the SEI information containing timestamps is first located and parsed by traversing the NAL Units encoded from the video stream. Then, a consistency check is performed to preliminarily confirm the synchronization status of the video stream source. Finally, based on the global timestamp and the local reception time, the precise transmission delay is calculated, providing an important basis for subsequent synchronization control strategies.

[0100] The above approach provides a crucial data preparation stage for achieving frame-level synchronization of WebRTC multi-view video streams. By accurately extracting timestamps and precisely measuring transmission delays, a solid foundation is laid for the synchronization controller on the playback end, ensuring a smooth and seamless multi-view live streaming experience regardless of network conditions.

[0101] As an optional implementation, the above method parses the sequence of data storage units corresponding to each frame of the image to obtain a global timestamp, including:

[0102] Traverse the sequence of data storage units and parse the boundaries of each data storage unit to obtain the type of each data storage unit;

[0103] Based on type, target storage units with supplementary and enhanced information are identified from the sequence of data storage units;

[0104] Extract the global timestamp from the target storage unit.

[0105] Once the video encoder completes the encoding of the multi-view video stream and embeds SEI information containing high-precision timestamps, the video stream enters the transmission phase. Upon arrival at the playback end, parsing the SEI timestamps becomes a crucial step in ensuring frame-level synchronization of the video stream. This involves traversing the sequence of data storage units after the video stream is encoded, parsing the boundaries of each unit to determine its type, and then identifying and extracting the timestamp.

[0106] The specific steps are as follows:

[0107] S31, Traverse each data storage unit (NAL Unit) in the data storage unit sequence.

[0108] After receiving the video stream, the playback device first needs to traverse the sequence of data storage units (NALUnits) generated by the encoder. This sequence contains all the components of the video stream, from video frame data to SEI information, each encapsulated in a separate data storage unit. The purpose of traversal is to examine and parse each unit one by one to identify its type.

[0109] S32, resolve NAL Uint boundaries and identify SEI type NAL units (H.264 Type=6, H.265 Prefix SEIType=39).

[0110] During the traversal, the boundaries and type identifier of each data storage unit are identified by parsing the header information of each data storage unit. For H.264 encoded video streams, the NAL Unit header contains a one-byte NAL Unit Header, which includes a type identifier; while for H.265 encoded format, the header information contains two bytes, also used to indicate the unit type.

[0111] S33, Identification and timestamp extraction of SEI type target storage units.

[0112] SEI Unit Identification: Based on the type identifier parsed during traversal, specific NAL Units (Target Storage Units) carrying SEI information can be quickly identified. This is especially important for SEI NAL Units containing timestamps, which are crucial for achieving frame-level synchronization of the video stream.

[0113] Timestamp Extraction: If the target storage unit is identified, the synchronization controller on the playback end will further parse the SEI payload and extract the global timestamp used for synchronization control. This timestamp is a precise time point obtained through the PTP protocol during the video acquisition stage, and is encapsulated into the SEI NAL Unit by the encoder when generating each frame image, so that the playback end can perform accurate time synchronization.

[0114] The timestamps mentioned above represent the time points when the video frames were generated at the acquisition end. In this way, the playback end accurately obtains time information closely related to the video frames, preparing the necessary parameters for further synchronization control.

[0115] By traversing the data storage unit sequence, identifying the SEI target storage unit type, and extracting SEI information containing timestamps, the system ensures that the video stream at the playback end can accurately grasp the generation time of each frame, providing a reliable timing reference for subsequent frame-level synchronization operations. This process not only demonstrates a deep understanding and flexible application of video coding standards but also innovatively combines PTP high-precision clock synchronization technology with the real-time transmission characteristics of WebRTC, solving common problems such as "time gaps" and "switching jumps" in multi-view video live streaming. Through precise timestamp extraction, the end-to-end latency of the video stream can be calculated in real time, providing a foundation for active maintenance and micro-speed catching up under the target latency anchoring strategy, improving the smoothness of multi-view switching and the overall viewing experience.

[0116] As an optional implementation, the above-mentioned determination of multiple transmission delay times based on multiple global timestamps in multiple first-frame images and multiple local times when the encoded data corresponding to multiple first-frame images is transmitted to the playback buffer includes:

[0117] Each first frame image is sequentially obtained from multiple first frame images and used as the current first frame image;

[0118] Find the current local time corresponding to the current first frame image from multiple local times;

[0119] Determine the first time difference between the current local time and the current global timestamp of the first frame image;

[0120] The second time difference between the first time difference and the delay time constant is determined as the current transmission delay time.

[0121] To accurately calculate the transmission delay of each video stream, the first frame of each stream is acquired and analyzed from multiple video streams. This process requires not only in-depth analysis of the video data but also accurate matching of the video stream's reception and encoding times. Since the technical solution in this application is a processing procedure in a real-time live streaming scenario, in this embodiment, the acquisition time at the acquisition end is directly equated to the encoding time of each frame, ignoring the time difference between the two.

[0122] S41, Selection and traversal of the first frame image;

[0123] After receiving multiple video streams, the synchronization controller on the playback end iterates through each video stream, selecting the first frame of the first video segment as the current object of analysis. This iteration process requires parsing the video streams to identify each video frame, especially the position of the first frame.

[0124] For example, suppose a concert is being streamed live from multiple angles, with four video streams coming from the front of the stage, backstage, the audience area, and the band area. After receiving these video streams, the playback device first parses the first frame of each stream. Taking the video stream from the front of the stage as an example, the synchronization controller locates the first frame and marks it as the current first frame image.

[0125] S42, Local time is matched with timestamp;

[0126] For the selected first frame of the image, the playback device needs to find the local time that recorded its reception time. This local time is usually recorded by the browser or other playback system and reflects the exact time when the video data arrived at the playback device.

[0127] Continuing with the concert livestream example above, once the first frame of the video stream directly in front of the stage is parsed, the synchronization controller queries the records to find the local reception time corresponding to that first frame. Let's assume this time is recorded as 1689831756845678 microseconds.

[0128] S43, calculate transmission delay and determine time base.

[0129] The calculation of the time difference requires calculating the first time difference between the local reception time of the current first frame image and the global timestamp encapsulated in the SEI.

[0130] Determining a transmission delay benchmark: To overcome network fluctuations and ensure the continuous synchronization of the video stream, this embodiment introduces the concept of a delay time constant. After calculating the initial time difference, this time difference is compared and corrected with a preset delay time constant to determine a stable and referable transmission delay standard.

[0131] For example, suppose the preset delay constant is 50ms to compensate for potential network jitter. After calculating the initial time difference (500 microseconds) of the first frame, the playback device compares this time difference with the delay constant to determine the corrected transmission delay. If the initial time difference is greater than the delay constant, the corrected transmission delay will be the initial time difference minus the delay constant; otherwise, the initial time difference will be added to the delay constant to ensure the synchronization stability of the video stream.

[0132] By acquiring each frame individually, matching local time, calculating time differences, and correcting delays, a complete process is provided for accurately calculating the transmission delay of multiple video streams. This process can be limited to analyzing the first frame of each encoded video stream, or it can intelligently compensate for network fluctuations by comparing the local reception time and global timestamp of each frame, ensuring that the video streams maintain stable synchronization at the playback end.

[0133] Compared to traditional latency calculation methods based on estimation or empirical rules, this method improves the accuracy and reliability of latency calculation by directly quantifying and comparing transmission latency, providing technical support for frame-level synchronization in multi-view WebRTC video live streaming. This innovative time difference correction mechanism not only solves the synchronization challenges caused by network jitter but also provides a more accurate time reference for latency control strategies on the playback end, thereby enhancing the user experience during multi-view switching.

[0134] As an optional example, the above-mentioned method of controlling the synchronous playback of multiple live video feeds corresponding to multiple video streams based on multiple transmission delay times includes:

[0135] The maximum transmission delay time is determined from multiple transmission delay times;

[0136] The target playback time is determined by the sum of the current time and the maximum transmission delay time.

[0137] During the target playback time, control multiple live stream feeds to play simultaneously.

[0138] To facilitate understanding, the following will be combined with... Figure 5 The processing flow of the synchronization controller shown describes in detail the synchronous playback process of the above-mentioned multi-channel live broadcast based on multiple transmission delay times.

[0139] S502 receives encoded data from multiple video streams after encoding at the acquisition end;

[0140] S504, parse the NAL Unit sequence corresponding to each frame of image and extract the absolute timestamp T_ptp;

[0141] S506, calculate the local time when each video stream is transmitted to the playback buffer, and based on this, determine the target video stream with the largest transmission delay, and determine the transmission delay time of the target video stream as the maximum transmission delay time T_max_delay;

[0142] In this process, performance.now is used as a high-precision local clock reference, i.e., the virtual playback clock is determined. The transmission delay time (delay) for each video stream is calculated using the formula (1) shown below:

[0143] delay=local_time-sei_timestamp-transmission_offset (1)

[0144] Among them, local_time is the local time when each video stream is transmitted to the playback buffer, sei_timestamp is the acquisition time indicated by the global timestamp injected at the acquisition end, and transmission_offset can be, but is not limited to, a delay time constant, that is, the delay duration caused by network fluctuations.

[0145] The target delay time (which can also be understood as the maximum delay time) is determined from the transmission delay time delay of each of the above video streams: target_delay=max(delay_stream1, delay_stream2...)+jitter_margin, where jitter_margin is the margin to resist network jitter, which is usually a constant, such as 50ms.

[0146] S508 calculates the target playback time (which can also be understood as the target rendering time) for each video stream based on the maximum transmission delay time and the global timestamp.

[0147] Specifically, it is calculated using the following formula (2):

[0148] T_render=T_ptp+T_max_delay+Buffer (2)

[0149] Where T_ptp is the time indicated by the global timestamp carried in the encoded data of each first frame image, T_max_delay is the maximum transmission delay time, and Buffer is a constant.

[0150] S510, determine whether the current system time is greater than or equal to T_render;

[0151] If yes, proceed to step S512; otherwise, proceed to step S514.

[0152] S512, submit rendering;

[0153] That is, the data is submitted to the decoder on the playback end, and the screen is rendered based on the decoded data.

[0154] S514: Store each video stream in the JitterBuffer of the application layer until the target playback time determined by the above method is reached, then release the frame (such as the first frame of each video stream) for rendering.

[0155] For example, assuming there are two video streams, and the transmission delay of the first video stream is calculated to be 1 second and the transmission delay of the second video stream to be 2 seconds, then T_max_delay equals 2. The global timestamp carried by the first frame is calculated, and this time is added to T_max_delay, and then added to the time constant indicated by the buffer to obtain the target playback time. For video streams with smaller delays, when a video frame arrives, its theoretically achievable playback time (i.e., the target playback time) is calculated. If the current time is less than the target playback time, the frame is "held down" in the application-layer JitterBuffer until the target playback time is reached before being released for rendering.

[0156] In addition to determining the maximum transmission delay time, this application embodiment also sets up an optional monitoring process, namely, monitoring whether there is a situation where the calculated transmission delay time of multiple video streams is excessively large. If so, a micro-speed catching-up mechanism is used to catch up:

[0157] If a portion of the transmission delay exceeds a time threshold among multiple transmission delay times, adjust the audio playback speed of the portion of the video stream corresponding to that portion of the transmission delay.

[0158] Based on the adjustment parameters of the audio playback speed, the playback speed of the video frame of a portion of the video stream is adjusted synchronously to obtain the adjusted delay time, wherein the adjusted delay time is less than or equal to the maximum transmission delay time.

[0159] In multi-view live streaming, the transmission latency of different video streams varies due to network conditions. Occasional network jitter may cause a significant increase in the transmission latency of video streams from certain perspectives, exceeding a preset time threshold. Therefore, when a transmission latency exceeding the preset time threshold is detected, a micro-speed-adjustment strategy is employed to dynamically adjust the playback speed of audio and video footage to ensure that all perspectives are played synchronously at the target playback time.

[0160] On the playback end, the synchronization controller continuously monitors the buffer status of each video stream and calculates the current transmission latency based on the local time and the global timestamp of each video frame in the video stream. If the transmission latency of a video stream exceeds a preset time threshold (for example, the threshold is set to 80ms), it indicates that the video stream has encountered a network latency problem, and measures need to be taken immediately to avoid "time gaps" and "switching jumps" affecting the user's viewing experience.

[0161] In response to transmission delays exceeding time thresholds, this application proposes a micro-speed catching-up strategy, which aims to quickly compensate for delay differences between streams without affecting audio and video quality.

[0162] Specifically, when a transmission delay exceeds a certain threshold, a micro-speed catching-up mechanism is activated. This means that when the stream delay exceeds the limit, the playback end immediately initiates the micro-speed catching-up mechanism to adjust the audio playback speed of the video stream severely affected by transmission delay. Using the HTML5 Audio's `preservesPitch=true` attribute, the audio playback speed can be adjusted without changing the pitch; a typical adjustment range is between 1.05x and 1.1x.

[0163] Synchronized adjustment of video and audio speeds: While adjusting the audio playback speed, the synchronization control module also adjusts the video playback speed accordingly to ensure that the synchronization between audio and video is not disrupted. In video processing, a similar time-stretching technique is used to slightly accelerate video playback, but this acceleration is so subtle that viewers will hardly notice any changes in the image.

[0164] The micro-speed adjustment catches up to a time less than or equal to the maximum transmission delay. Specifically, through the aforementioned micro-speed adjustment catching-up strategy, the playback end can effectively correct delays exceeding a time threshold, ensuring that the adjusted delay time (considering the micro-speed adjustment catching-up results for both audio and video) converges to or is less than the maximum transmission delay time. This process ensures that even under unstable network conditions, all video streams can play synchronously at the target playback time, providing a smooth, seamless multi-view experience.

[0165] The target playback time is the sum of the current local time on the playback device and the maximum transmission delay. If the transmission delay of each video stream (including before and after adjustment) meets the target playback time requirement, the synchronization control module will start the synchronous playback of all video streams at this time point, providing users with a unified multi-view synchronized picture.

[0166] For example, in the context of live basketball streaming, the video stream from the referee's perspective underwent a slight speed-up adjustment, with both audio and video playback speeds increased by 1.07x. This adjustment not only ensured that the referee's perspective caught up with the players' perspective before the target playback time but also maintained high-quality audio and video without any skipping or distortion. Ultimately, all perspectives played synchronously at the expected time, providing users with a seamless, immersive viewing experience when switching perspectives or watching simultaneously.

[0167] By detecting and responding to delays exceeding a time threshold in the video stream, and combining this with a micro-speed-adjustment strategy to synchronously adjust the playback speed of audio and video, various problems that may arise in real-time multi-view live streaming are effectively solved. This method not only improves the robustness of the live streaming system but also ensures high-quality audio and video output, demonstrating the innovative approach and practical value of multi-view video stream synchronization control within the WebRTC framework. Through the micro-speed-adjustment strategy, synchronized playback of multi-view videos can be ensured even under poor network conditions.

[0168] To better understand the above technical solution, a preferred embodiment will be used to further explain and illustrate it below.

[0169] S51, PTP clock synchronization system deployment;

[0170] Deploy a PTP Grandmaster Clock on a multi-camera shooting site. All cameras and encoders are connected to the same local area network via an IEEE 1588v2-enabled switch. It is recommended to use a network interface card (NIC) that supports hardware timestamps, enabling sub-microsecond synchronization accuracy.

[0171] S52, SEI timestamp injection;

[0172] A custom UUID (16 bytes) is written into the SEI user_data_unregistered payload as a unique identifier for the timestamp of this invention. The SEI structure is as follows:

[0173] NAL Unit Header (1 byte for H.264,2 bytes for H.265);

[0174] SEI payload type: 5(user_data_unregistered);

[0175] Payload size: 24 bytes (16 bytes UUID+8 bytes timestamp).

[0176] UUID is a custom 16-byte unique identifier, and PTP timestamp is a 64-bit Unix microsecond timestamp.

[0177] When the encoder outputs each frame, the current PTP timestamp is obtained, the SEI NAL Unit is constructed and inserted before the VCLNAL Unit.

[0178] The aforementioned UUID identifier can be used, but is not limited to, associating timestamps across different video streams. Specifically, during video encoding, each video frame is accompanied by a timestamp, which is stored in the SEI in a specific format. The UUID and the timestamp are encapsulated together in the SEI NAL Unit, so that when the playback end parses the SEI, it can accurately match the timestamp with the video frame by matching the UUID, thereby achieving frame-level synchronization of the video stream.

[0179] The above steps S51 and S52 are the processing flow at the acquisition end.

[0180] S53, WebRTC Insertable Streams configuration;

[0181] Create a TransformStream using RTCRtpReceiver.transform or RTCRtpSender.transform, and access RTCEncodedVideoFrame.data (Uint8Array) in the transform callback. This method transmits the encoded video streams to the playback device.

[0182] S54, Execution of the SEI parsing algorithm;

[0183] Traverse the binary data of video frames, parse the NAL Unit boundaries, identify the SEI type NALU (H.264 Type=6, H.265 Prefix SEI Type=39), match the custom UUID, extract the 64-bit PTP timestamp, and append the timestamp to the frame metadata.

[0184] S55, the implementation of a synchronous controller;

[0185] Virtual playback clock: performance.now() is used as a high-precision local clock reference. The calculation of the transmission delay time and the maximum transmission delay time of each video stream can be referred to the descriptions in Formula (1) and Formula (2) above, which will not be repeated here.

[0186] For cases where frames are transmitted to the playback buffer ahead of schedule, the early frames are stored in the buffer (i.e., Frame Holding) and released at the target time using setTimeout or requestAnimationFrame. setTimeout is a timer function used to release the video stream in the buffer at a specified time. requestAnimationFrame can be understood as, but is not limited to, a callback function used to check if the set time has been reached, such as whether the maximum transmission delay time of 2 seconds has been reached.

[0187] Conversely, when the delay exceeds the threshold based on the transmission delay time, set HTMLMediaElement.playbackRate=1.05~1.1 and preservesPitch=true, which triggers micro-speed catching up. For details, please refer to the description in the above embodiments, which will not be repeated here.

[0188] S56, timestamp rollback processing.

[0189] A 64-bit timestamp can represent approximately 580,000 years, and no rollback processing is needed in practical applications. If a compressed 32-bit relative timestamp is used, rollback conditions (the current timestamp is less than the previous frame and the difference exceeds half a period) need to be detected during parsing, and period compensation needs to be performed.

[0190] The above steps S53 to S56 are the processing flow at the playback end.

[0191] In this application embodiment, PTP high-precision timestamps are combined with video encoding SEI for WebRTC multi-stream synchronization for the first time; a "target delay anchoring" control strategy is proposed to achieve frame-level synchronization in pure software and browser; micro-speed catching up is used to replace frame dropping to ensure a lossless user experience; the overall solution does not require modification of network intermediate devices.

[0192] Based on the analysis of the above embodiments, it can be seen that the technical solution of this application can solve, but is not limited to, the following problems:

[0193] (1) Lack of multi-stream synchronization benchmark: In WebRTC, each video stream is transmitted and decoded independently, lacking a unified global clock anchor point, which makes it impossible to accurately align multi-view images.

[0194] (2) Time deviation caused by network jitter: Video streams from different perspectives travel through different network paths, and the difference in arrival time leads to the phenomenon of "time crack".

[0195] (3) Poor switching experience: When users switch between multiple perspectives, time jumps occur, which ruins the viewing experience.

[0196] In other words, the technical solution of this application aims to provide an end-to-end full-link synchronization mechanism to achieve frame-level precise synchronization of multiple WebRTC video streams, eliminate the impact of network jitter, and provide a smooth multi-view switching experience.

[0197] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0198] According to another aspect of the embodiments of this application, as follows is also provided Figure 6 The illustrated multi-video stream processing apparatus includes:

[0199] The first acquisition unit 602 is used to acquire multiple video streams for real-time playback of live broadcast images;

[0200] The encoding unit 604 is used to encode multiple video streams and add a global timestamp to the data storage unit sequence corresponding to each frame of the encoded image.

[0201] The first processing unit 606 is used to determine multiple transmission delay times of multiple frames with the same global timestamp in multiple video streams based on the global timestamp and the local time when each encoded frame image is transmitted to the playback buffer.

[0202] The second processing unit 608 is used to control the synchronous playback of multiple live broadcast images corresponding to multiple video streams based on multiple transmission delay times.

[0203] Optionally, the above-mentioned encoding unit 604 includes:

[0204] The first encoding module is used to encode each frame of the image in the multiple video streams to obtain a sequence of multiple data storage units;

[0205] An add module is used to add a global timestamp to a target storage unit in each data storage unit sequence, wherein the target storage unit has a specific storage area required to store supplementary enhancement information, which includes custom image description information.

[0206] Optionally, the above-mentioned added modules include:

[0207] The construction submodule is used to build the target storage unit for writing global timestamps;

[0208] The encapsulation submodule is used to encapsulate the global timestamp into a specific storage area in the target storage unit;

[0209] The insertion submodule is used to insert the encapsulated target storage unit before the image data storage unit in each data storage unit sequence, resulting in a sequence of each data storage unit with a global timestamp added, where the image data storage unit is used to store the encoded data of the image itself.

[0210] Optionally, the first processing unit 606 includes:

[0211] The parsing module is used to parse the data storage unit sequence corresponding to each frame of the received encoded multi-channel video stream to obtain the global timestamp.

[0212] The first acquisition module is used to acquire the first frame image of the first video segment in each video stream, and obtain multiple first frame images. The multiple frame images include multiple first frame images, and the global timestamps of the multiple first frame images are the same.

[0213] The first processing module is used to determine multiple transmission delay times based on multiple global timestamps in multiple first-frame images and multiple local times when the encoded data corresponding to multiple first-frame images are transmitted to the playback buffer.

[0214] Optionally, the above-mentioned parsing module includes:

[0215] The traversal submodule is used to traverse the sequence of data storage units and parse the boundaries of each data storage unit to obtain the type of each data storage unit;

[0216] The identification submodule is used to identify target storage units with supplementary and enhanced information from the data storage unit sequence based on type.

[0217] The extraction submodule is used to extract the global timestamp from the target storage unit.

[0218] Optionally, the first processing module mentioned above includes:

[0219] The first acquisition submodule is used to sequentially acquire each first frame image from multiple first frame images as the current first frame image;

[0220] The lookup submodule is used to find the current local time corresponding to the current first frame image from multiple local times;

[0221] Determine the first time difference between the current local time and the current global timestamp of the first frame image;

[0222] The first processing submodule is used to determine the second time difference between the first time difference and the delay time constant as the current transmission delay time.

[0223] Optionally, the second processing unit 608 includes:

[0224] The second processing module is used to determine the maximum transmission delay time from multiple transmission delay times;

[0225] The third processing module is used to determine the target playback time by summing the current time and the maximum transmission delay time.

[0226] The control module is used to control the synchronous playback of multiple live streams at the target playback time.

[0227] Optionally, the above-mentioned device further includes:

[0228] The first adjustment unit is used to adjust the audio playback speed of a portion of the video stream corresponding to a portion of the transmission delay time when there is a portion of the transmission delay time that exceeds a time threshold among multiple transmission delay times.

[0229] The second adjustment unit is used to synchronously adjust the playback speed of a portion of the video stream based on the adjustment parameters of the audio playback speed, so as to obtain the adjusted delay time, wherein the adjusted delay time is less than or equal to the maximum transmission delay time.

[0230] It should be noted that the embodiments of the multi-channel video stream processing device described here can refer to the embodiments of the multi-channel video stream processing method described above, and will not be repeated here.

[0231] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described multi-video stream processing method is also provided. This electronic device may be... Figure 1 The target terminal or server is shown. This embodiment uses the electronic device as an example to illustrate the concept. Figure 7 As shown, the electronic device includes a memory 702 and a processor 704. The memory 702 stores a computer program, and the processor 704 is configured to execute the steps in any of the above method embodiments via the computer program.

[0232] Optionally, the aforementioned electronic device may be located in at least one of a plurality of network devices of the computer.

[0233] Optionally, the processor described above can be configured to perform the following steps via a computer program:

[0234] S1, acquire multiple video streams used for real-time playback of the live broadcast;

[0235] S2 encodes multiple video streams and adds a global timestamp to the data storage unit sequence corresponding to each frame of the encoded image.

[0236] S3, based on the global timestamp and the local time when each encoded frame is transmitted to the playback buffer, determine multiple transmission delay times for multiple frames with the same global timestamp in multiple video streams.

[0237] S4 controls the synchronous playback of multiple live video feeds corresponding to multiple video streams based on multiple transmission delay times.

[0238] Alternatively, as those skilled in the art will understand, Figure 7 The structure shown is for illustrative purposes only. Figure 7 This does not limit the structure of the aforementioned electronic devices or electronic equipment. For example, electronic devices or electronic equipment may also include components that are more... Figure 7 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 7 The different configurations shown.

[0239] The memory 702 can be used to store software programs and modules, such as the program instructions / modules corresponding to the multi-channel video stream processing method and apparatus in this embodiment. The processor 704 executes various functional applications and data processing by running the software programs and modules stored in the memory 702, thereby realizing the aforementioned multi-channel video stream processing method. The memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 702 may further include memory remotely located relative to the processor 704, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 702 may be used, but is not limited to, to store multi-channel video streams, global timestamps, and multiple transmission delay times, etc. As an example, such as... Figure 7 As shown, the memory 702 may include, but is not limited to, the first acquisition unit 602, encoding unit 604, first processing unit 606, and second processing unit 608 in the multi-channel video stream processing device. Furthermore, it may include, but is not limited to, other module units in the multi-channel video stream processing device, which will not be elaborated upon in this example.

[0240] Optionally, the transmission device 706 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 706 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 706 is a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0241] In addition, the above-mentioned electronic device also includes: a display 708 for displaying the synchronous live broadcast images corresponding to multiple video streams; and a connection bus 710 for connecting the various module components in the above-mentioned electronic device.

[0242] In other embodiments, the target terminal or server described above can be a node in a distributed system. This distributed system can be a blockchain system, formed by connecting multiple nodes through network communication. The nodes can form a point-to-point network, and any type of computing device, such as a server or target terminal, can become a node in the blockchain system by joining this point-to-point network.

[0243] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the multi-channel video stream processing method provided in various optional implementations of the above-described server verification processing, wherein the computer program is configured to execute the steps in any of the above-described method embodiments at runtime.

[0244] Optionally, in this embodiment, the computer-readable storage medium described above may be configured to store a computer program for performing the following steps:

[0245] S1, acquire multiple video streams used for real-time playback of the live broadcast;

[0246] S2 encodes multiple video streams and adds a global timestamp to the data storage unit sequence corresponding to each frame of the encoded image.

[0247] S3, based on the global timestamp and the local time when each encoded frame is transmitted to the playback buffer, determine multiple transmission delay times for multiple frames with the same global timestamp in multiple video streams.

[0248] S4 controls the synchronous playback of multiple live video feeds corresponding to multiple video streams based on multiple transmission delay times.

[0249] Optionally, in embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0250] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the target terminal. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0251] The sequence numbers of the embodiments in this application are merely for description and do not represent the superiority or inferiority of the embodiments. If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods in the various embodiments of this application.

[0252] In the above embodiments of this application, the descriptions of each embodiment have their own emphasis. Parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed client can be implemented in other ways in the several embodiments provided in this application. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0253] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0254] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing multiple video streams, characterized in that, include: Acquire multiple video streams for real-time playback of live stream content; The multiple video streams are encoded, and a global timestamp is added to the data storage unit sequence corresponding to each frame of the encoded image. Based on the global timestamp and the local time when each encoded frame is transmitted to the playback buffer, multiple transmission delay times of multiple frames with the same global timestamp in multiple video streams are determined. Based on the multiple transmission delay times, the multiple live broadcast images corresponding to the multiple video streams are controlled to be played synchronously.

2. The method according to claim 1, characterized in that, The process of encoding the multiple video streams and adding a global timestamp to the data storage unit sequence corresponding to each encoded frame includes: Each frame of the multi-channel video stream is encoded to obtain a sequence of multiple data storage units; The global timestamp is added to the target storage unit in each data storage unit sequence, wherein the target storage unit has a specific storage area required to store supplementary enhancement information, which includes custom image description information.

3. The method according to claim 2, characterized in that, Adding the global timestamp to the target storage unit in each data storage unit sequence includes: Construct the target storage unit for writing the global timestamp; The global timestamp is encapsulated into the specific storage area within the target storage unit; Before inserting the encapsulated target storage unit into the image data storage unit in each data storage unit sequence, a sequence of each data storage unit with the global timestamp added is obtained, wherein the image data storage unit is used to store the encoded data of the image itself.

4. The method according to claim 1, characterized in that, The determination of multiple transmission delay times for multiple frames with the same global timestamp in multiple video streams, based on the global timestamp and the local time when each encoded frame is transmitted to the playback buffer, includes: In response to the received encoded multi-channel video stream, the data storage unit sequence corresponding to each frame image is parsed to obtain the global timestamp; The first frame image of the first video segment in each video stream is obtained to obtain multiple first frame images, wherein the multiple frame images include the multiple first frame images and the global timestamp of the multiple first frame images is the same; The multiple transmission delay times are determined based on multiple global timestamps in the multiple first-frame images and multiple local times when the encoded data corresponding to the multiple first-frame images is transmitted to the playback buffer.

5. The method according to claim 4, characterized in that, The step of parsing the data storage unit sequence corresponding to each frame of image to obtain the global timestamp includes: Traverse the sequence of data storage units and parse the boundaries of each data storage unit to obtain the type of each data storage unit; Based on the type, target storage units with supplementary enhancement information are identified from the data storage unit sequence; Extract the global timestamp from the target storage unit.

6. The method according to claim 4, characterized in that, The determination of the multiple transmission delay times based on multiple global timestamps in the multiple first-frame images and multiple local times when the encoded data corresponding to the multiple first-frame images is transmitted to the playback buffer includes: Each first frame image is sequentially obtained from the plurality of first frame images as the current first frame image; Find the current local time corresponding to the current first frame image from the plurality of local times; Determine the first time difference between the current local time and the current global timestamp of the current first frame image; The second time difference between the first time difference and the delay time constant is determined as the current transmission delay time.

7. The method according to claim 1, characterized in that, The step of controlling the synchronous playback of multiple live streams corresponding to the multiple video streams based on the multiple transmission delay times includes: The maximum transmission delay time is determined from the plurality of transmission delay times; The target playback time is determined by the sum of the current time and the maximum transmission delay time. At the target playback time, control the multiple live stream feeds to play synchronously.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: If any of the multiple transmission delay times exceeds a time threshold, the audio playback speed of the portion of the video stream corresponding to that portion of the transmission delay time is adjusted. Based on the adjustment parameters of the audio playback speed, the playback speed of the video frame of the portion of the video stream is adjusted synchronously to obtain the adjusted delay time, wherein the adjusted delay time is less than or equal to the maximum transmission delay time.

9. A processing apparatus for multiple video streams, characterized in that, include: The first acquisition unit is used to acquire multiple video streams for real-time playback of live broadcast images; The encoding unit is used to encode the multiple video streams and add a global timestamp to the data storage unit sequence corresponding to each frame of the encoded image. The first processing unit is used to determine multiple transmission delay times of multiple frames with the same global timestamp in multiple video streams based on the global timestamp and the local time when each encoded frame image is transmitted to the playback buffer. The second processing unit is used to control the synchronous playback of multiple live broadcast images corresponding to the multiple video streams based on the multiple transmission delay times.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program can be executed by a terminal device or computer at runtime as described in any one of claims 1 to 8.

11. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to perform the method as described in any one of claims 1 to 8 through the computer program.

Images