Method and apparatus for synchronizing data streams, electronic device, and storage medium
By acquiring the first end-to-end delay of multiple data streams and determining the target end-to-end delay for synchronous playback control, the problem of low synchronization accuracy of multiple audio streams is solved, achieving higher synchronization accuracy and playback effect.
Patent Information
- Application Number
- CN202410027728.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-08
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-01-08
AI Technical Summary
In real-time choral applications, the synchronization accuracy of multiple audio streams is low, and existing technologies cannot effectively solve the problem of non-convergence caused by the mutual pull of delays between multiple streams.
By acquiring the first end-to-end delay of multiple data streams, the target end-to-end delay to be synchronized is determined, and synchronous playback control is performed based on the target end-to-end delay, avoiding the selection of a single data stream as a benchmark, and comprehensively considering the delay of multiple data streams.
It improves the synchronization accuracy of multiple data streams, avoids the inability to converge due to mutual delays, and enhances the synchronization and playback effects of multiple data streams.
Smart Images

Figure CN117856959B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and more particularly to a method, apparatus, electronic device, and storage medium for synchronizing data streams. Background Technology
[0002] Currently, in real-time choral applications, the receiving end needs to receive audio streams from different singers, which places very high demands on the synchronization of these audio streams. Accurate alignment of audio streams from different devices is a challenge. Current technologies are primarily based on alignment using the singers' Network Time Protocol (NTP) to agree on simultaneous playback progress and ensure users sing at the same time. The receiving end then aligns the received audio streams based on the uplink-aligned NTP time.
[0003] For the alignment of received audio streams, existing technologies generally rely on the synchronization of two streams. The synchronization of multiple streams also follows the same two-to-two synchronization scheme, with one stream as the reference stream and the other streams synchronized with the reference stream. This approach does not take a global perspective, and the delays can lead to mutual pulling and non-convergence issues, resulting in low synchronization accuracy of multiple streams. Summary of the Invention
[0004] This disclosure provides a data stream synchronization method, apparatus, electronic device, and storage medium to improve the synchronization accuracy of multiple streams.
[0005] In a first aspect, embodiments of this disclosure provide a data stream synchronization method, comprising:
[0006] Obtain the first end-to-end delay of at least two data streams, wherein the at least two data streams are data streams to be synchronized;
[0007] The target end-to-end delay to which the at least two data streams need to be synchronized is determined based on the first end-to-end delay.
[0008] Synchronous playback control is performed on the at least two data streams based on the target end-to-end delay.
[0009] Secondly, embodiments of this disclosure also provide a data stream synchronization device, comprising:
[0010] The delay acquisition module is used to acquire the first end-to-end delay of at least two data streams, wherein the at least two data streams are data streams to be synchronized;
[0011] The delay determination module is used to determine the target end-to-end delay to which the at least two data streams are to be synchronized based on the first end-to-end delay;
[0012] The playback control module is used to perform synchronous playback control on the at least two data streams based on the target end-to-end delay.
[0013] Thirdly, embodiments of this disclosure also provide an electronic device, including:
[0014] One or more processors;
[0015] Memory, used to store one or more programs.
[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the data stream synchronization method as described in the embodiments of this disclosure.
[0017] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the data stream synchronization method as described in embodiments of this disclosure.
[0018] The data stream synchronization method, apparatus, electronic device, and storage medium provided in this disclosure acquire first end-to-end delays of at least two data streams, which are data streams to be synchronized; determine a target end-to-end delay to which the at least two data streams are to be synchronized based on the first end-to-end delays; and perform synchronized playback control on the at least two data streams based on the target end-to-end delays. This embodiment utilizes the above technical solution to determine a common target end-to-end delay for synchronized playback control of multiple data streams based on the first end-to-end delays of the multiple data streams to be synchronized. It does not require using a single data stream as a reference, and can comprehensively consider the delays of the multiple data streams to be synchronized, avoiding convergence failures due to mutual delay interference, and improving the synchronization accuracy of multiple data streams. Attached Figure Description
[0019] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.
[0020] Figure 1 A flowchart illustrating a data stream synchronization method provided in an embodiment of this disclosure;
[0021] Figure 2 A flowchart illustrating another data stream synchronization method provided in this embodiment of the disclosure;
[0022] Figure 3 A flowchart illustrating an optional data stream synchronization method provided in an embodiment of this disclosure;
[0023] Figure 4 This is a schematic diagram illustrating the disassembly of an end-to-end delay provided in an embodiment of the present disclosure;
[0024] Figure 5 This is another schematic diagram illustrating the disassembly of end-to-end delay provided in an embodiment of the present disclosure;
[0025] Figure 6 A schematic diagram illustrating the adjustment of a target end-to-end delay provided in an embodiment of this disclosure;
[0026] Figure 7 A schematic diagram of a playback control feedback system provided in an embodiment of this disclosure;
[0027] Figure 8 A structural block diagram of a data stream synchronization device provided in an embodiment of this disclosure;
[0028] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0029] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0030] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.
[0031] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.
[0032] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0033] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0034] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
[0035] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.
[0036] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.
[0037] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0038] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.
[0039] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.
[0040] Figure 1 This is a flowchart illustrating a data stream synchronization method provided in an embodiment of this disclosure. The method can be executed by a data stream synchronization device, which can be implemented in software and / or hardware and can be configured in an electronic device, typically a mobile phone or tablet. The data stream synchronization method provided in this disclosure is applicable to scenarios involving the synchronization of multiple data streams. Figure 1 As shown, the data stream synchronization method provided in this embodiment may include:
[0041] S101. Obtain the first end-to-end delay of at least two data streams, wherein the at least two data streams are data streams to be synchronized.
[0042] The first end-to-end delay can be understood as a certain end-to-end delay of at least two data streams to be synchronized, such as the maximum end-to-end delay, minimum end-to-end delay, average end-to-end delay (i.e., the average of end-to-end delays), or median end-to-end delay (i.e., the median of end-to-end delays) of at least two data streams within the current acquisition period. Optionally, the first end-to-end delay can be the maximum value among the minimum end-to-end delays of each data stream within the current acquisition period. In other words, the first end-to-end delay can be the maximum value among multiple minimum end-to-end delays, which can be the minimum end-to-end delays of each data stream to be synchronized within the current acquisition period. The number of first end-to-end delays obtained can be one, meaning that multiple data streams to be synchronized can correspond to the same first end-to-end delay within the current acquisition period.
[0043] The at least two data streams to be synchronized may include audio and / or video streams, which can be live or non-live data streams. The number of data streams to be synchronized is unlimited; for example, there may be at least two data streams, such as three or more. The end-to-end delay can be the duration of the end-to-end delay, such as the time interval between data stream acquisition and playback. It can be determined by calculating the difference between the start time of a data packet and its acquisition time, which can be carried in the data packet. The current acquisition period can be the current acquisition period, which can be understood as the period for acquiring the first end-to-end delay. That is, in this embodiment, the first end-to-end delay can be acquired and updated periodically according to a pre-set acquisition period.
[0044] In this embodiment, the first end-to-end delay of at least two data streams to be synchronized can be obtained so that the target end-to-end delay shared by the at least two data streams during synchronization can be determined based on the first end-to-end delay.
[0045] S102. Determine the target end-to-end delay to which the at least two data streams are to be synchronized based on the first end-to-end delay.
[0046] The target end-to-end delay can be understood as the end-to-end delay shared by the various data streams to be synchronized during synchronization. For example, the reference end-to-end delay when synchronizing the various data streams to be synchronized can be the end-to-end delay that the various data streams need to be aligned during playback.
[0047] Specifically, after obtaining the first end-to-end delay, the target end-to-end delay shared by each data stream to be synchronized can be determined based on the first end-to-end delay. For example, the target end-to-end delay can be updated based on the first end-to-end delay so that the at least two data streams to be synchronized can be controlled for synchronous playback based on the updated target end-to-end delay.
[0048] In some implementations, determining the target end-to-end delay to which the at least two data streams are to be synchronized based on the first end-to-end delay includes: obtaining the original value of the target end-to-end delay; determining the target value of the target end-to-end delay based on the original value and the delay value of the first end-to-end delay; and adjusting the target end-to-end delay from the original value to the target value.
[0049] The original value of the target end-to-end delay can be the delay value last adjusted to before the current adjustment of the target end-to-end delay. For example, the target end-to-end delay can be adjusted periodically according to a preset adjustment period. In this case, the first end-to-end delay obtained can be the first end-to-end delay of at least two data streams to be adjusted in the current adjustment period. The original value of the target end-to-end delay can be the delay value of the target end-to-end delay in the previous adjustment period. When there is no previous adjustment period, such as when the current adjustment period is the first adjustment period of the target end-to-end delay, the target end-to-end delay may not have an original value. Alternatively, the original value of the target end-to-end delay can be a preset initial value (such as a preset invalid value). The following explanation uses the example of the initial value of the target end-to-end delay being a preset invalid value when there is no previous adjustment period.
[0050] The target value of the end-to-end delay can be understood as the delay value that the end-to-end delay needs to be adjusted to in this adjustment, such as the delay value of the end-to-end delay within the current adjustment period. The length of the adjustment period is not limited; for example, the length of the adjustment period can be set to 0.5s, 1s, or 2s, etc.
[0051] In the above implementation, the target end-to-end delay can be adjusted based on the first end-to-end delay, such as adjusting the target end-to-end delay from the original value to the target value.
[0052] For example, the delay value of the target end-to-end delay in the previous adjustment cycle can be obtained as the original value of the target end-to-end delay; the target value of the target end-to-end delay can be determined based on the original value of the target end-to-end delay and the delay value of the first end-to-end delay obtained, and the target end-to-end delay can be adjusted from the original value to the target value, so that the at least two data streams to be synchronized can be controlled for synchronous playback based on the adjusted target end-to-end delay.
[0053] In the above embodiments, the method for determining the target value of the target end-to-end delay based on the original value of the target end-to-end delay and the delay value of the acquired first end-to-end delay is not limited. Optionally, the method for determining the target value of the target end-to-end delay can be based on whether the original value of the target end-to-end delay is a preset invalid value, and the target value of the target end-to-end delay can be determined using this method based on the original value of the target end-to-end delay and the first end-to-end delay.
[0054] In some implementations, determining the target value of the target end-to-end delay based on the original value and the delay value of the first end-to-end delay includes: if the original value is a preset invalid value, then using the delay value of the first end-to-end delay as the target value of the target end-to-end delay; and / or, if the original value is not a preset invalid value, then calculating the difference between the delay value of the first end-to-end delay and the original value, and determining the target value of the target end-to-end delay based on the difference.
[0055] For example, if the original value of the target end-to-end delay is a preset invalid value, it indicates that the current adjustment period is the first adjustment period for the target end-to-end delay. In this case, since there is no valid original value for the target end-to-end delay, the target end-to-end delay can be adjusted only based on the first end-to-end delay of the acquired at least two data streams within the current adjustment period. For example, the delay values of the first end-to-end delay of these two data streams within the current adjustment period can be directly used as the target value of the target end-to-end delay within the current adjustment period. And / or, if the original value of the target end-to-end delay is not a preset invalid value, it indicates that the current adjustment period is not the first adjustment period for the target end-to-end delay. In this case, there is a valid original value for the target end-to-end delay. Therefore, the difference between the delay value of the first end-to-end delay and the original value of the target end-to-end delay can be calculated, and the target value of the target end-to-end delay can be determined based on this difference.
[0056] In the above embodiments, the method for determining the target value of the target end-to-end delay based on the difference between the delay value of the first end-to-end delay and the original value of the target end-to-end delay is not limited. For example, the original value of the target end-to-end delay can be adjusted based on the relative magnitude between the difference between the delay value of the first end-to-end delay and the original value of the target end-to-end delay and a preset threshold (such as a first preset threshold and / or a second preset threshold) to obtain the target value of the target end-to-end delay.
[0057] Optionally, determining the target value of the target end-to-end delay based on the difference includes at least one of the following: if the difference is greater than a first preset threshold, the original value is increased by a first adjustment step to obtain the target value of the target end-to-end delay, wherein the first preset threshold is a positive value; if the difference is less than or equal to the first preset threshold and greater than or equal to a second preset threshold, the original value is used as the target value of the target end-to-end delay; if the difference is less than the second preset threshold, the original value is decreased by a second adjustment step to obtain the target value of the target end-to-end delay, wherein the second preset threshold is a negative value.
[0058] The first preset threshold and the second preset threshold can be different preset thresholds. The first preset threshold can be a positive value, and the second preset threshold can be a negative value. The relative magnitude between the absolute values of the first preset threshold and the second preset threshold is not limited; for example, the absolute values of the first preset threshold and the second preset threshold can be equal or unequal.
[0059] Taking a first preset threshold as positive and a second preset threshold as negative as an example, the first adjustment step size and the second adjustment step size can both be positive values. The relative size between the first adjustment step size and the second adjustment step size is not limited; for example, the first adjustment step size and the second adjustment step size can be equal or unequal. The first adjustment step size and / or the second adjustment step size can be fixed values or dynamically determined values; this embodiment does not impose any restrictions on this. Optionally, the first adjustment step size can be the minimum absolute value of the first preset step size and the above-mentioned difference (i.e., the difference between the delay value of the first end-to-end delay and the original value of the target end-to-end delay); and / or, the second adjustment step size can be the minimum absolute value of the second preset step size and the above-mentioned difference, to further improve the smoothness effect when adjusting the target end-to-end delay. Here, the first preset step size and the second preset step size can be two pre-set step sizes. The first preset step size and the second preset step size can both be positive values, and the relative size between the first preset step size and the second preset step size is not limited; for example, the first preset step size and the second preset step size can be equal or unequal.
[0060] In this embodiment, the target end-to-end delay can be smoothly adjusted based on the adjustment step size to avoid introducing new asynchrony due to excessive adjustment. By controlling the global adjustment of the target end-to-end delay, the impact of data stream acceleration and deceleration can be reduced, further improving the synchronization and playback effects of the data stream.
[0061] For example, if the difference between the delay value of the first end-to-end delay and the original value of the target end-to-end delay is greater than the first preset threshold, it means that the original value of the target end-to-end delay is less than the delay value of the first end-to-end delay and the difference between the two is large. In this case, the original value of the target end-to-end delay can be increased by the first adjustment step, such as increasing the original value of the target end-to-end delay by the minimum value between the first preset step and the difference, and the increased original value is used as the target value of the target end-to-end delay.
[0062] If the difference between the first end-to-end delay value and the original value of the target end-to-end delay is less than or equal to the first preset threshold and greater than or equal to the second preset threshold, it indicates that the original value of the target end-to-end delay is close to the delay value of the first end-to-end delay. In this case, the original value of the target end-to-end delay can be used directly as the target value of the target end-to-end delay without adjustment.
[0063] If the difference between the delay value of the first end-to-end delay and the original value of the target end-to-end delay is less than the second preset threshold, it means that the original value of the target end-to-end delay is greater than the delay value of the first end-to-end delay and the difference between the two is large. In this case, the original value of the target end-to-end delay can be reduced by the second adjustment step size, such as reducing the original value of the target end-to-end delay by the second preset step size and the minimum value of the difference, and the reduced original value is taken as the target value of the target end-to-end delay.
[0064] S103. Perform synchronous playback control on the at least two data streams based on the target end-to-end delay.
[0065] In this embodiment, after determining the target end-to-end delay shared by at least two data streams to be synchronized, playback control can be performed on the at least two data streams based on the target end-to-end delay to achieve synchronized playback of the at least two data streams.
[0066] In some implementations, the step of controlling the synchronous playback of the at least two data streams based on the target end-to-end delay includes: for each of the at least two data streams, obtaining the actual end-to-end delay of the current data stream; calculating the synchronization difference duration of the current data stream based on the actual end-to-end delay and the target end-to-end delay; and controlling the playback of the current data stream based on the synchronization difference duration.
[0067] Here, the current data stream can be understood as the data stream currently being controlled for playback. It can be any one of the at least two data streams to be synchronized. In other words, each of the at least two data streams to be synchronized can be used as the current data stream for playback control. The actual end-to-end delay of the current data stream can be understood as the actual end-to-end delay of the current data stream at present, such as the true end-to-end delay of the current data stream within the current playback control cycle.
[0068] Synchronization difference duration can be understood as the duration of the difference between the actual end-to-end delay and the target end-to-end delay. The current playback control cycle can be the current playback control cycle. In this embodiment, playback control of each data stream can be performed periodically according to a preset playback control cycle. Here, the cycle length of this playback control cycle is not limited; for example, it can be set to 7ms, 10ms, or 20ms, etc.
[0069] In the above implementation, playback control can be performed on each data stream based on the synchronization difference between the actual end-to-end delay and the determined target end-to-end delay, so as to adjust the actual end-to-end delay of the data stream to be within a preset range of the difference between it and the target end-to-end delay, thereby achieving synchronous playback between different data streams.
[0070] For example, for each of the at least two data streams to be synchronized, the actual end-to-end delay of the current data stream can be obtained first, such as calculating the time interval between the playback time point and the acquisition time point of the data stream, as the actual end-to-end delay of the current data stream.
[0071] Then, the synchronization difference duration of the current data stream is calculated based on the actual end-to-end delay and the target end-to-end delay. For example, the difference between the actual end-to-end delay and the target end-to-end delay is calculated as the synchronization difference duration of the current data stream.
[0072] Subsequently, playback control is applied to the current data stream based on its synchronization difference duration. For example, if the synchronization difference duration of the current data stream exceeds the preset maximum difference duration threshold, it indicates that the current data stream has excessive latency. In this case, the playback speed of the current data stream can be accelerated, such as increasing the playback speed to a times its original playback speed before adjustment (a > 1). If the synchronization difference duration of the current data stream is less than the preset minimum difference duration threshold, it indicates that the current data stream is playing too fast. In this case, the playback speed of the current data stream can be decelerated and aligned, such as reducing the playback speed to b times its original playback speed before adjustment (0 < b < 1). If the synchronization difference duration of the current data stream is greater than the preset minimum difference duration threshold but less than the preset maximum difference duration threshold, it indicates that the synchronization of the current data stream is good. In this case, the playback speed of the current data stream can be kept unchanged, i.e., no adjustment is made to the playback speed of the current data stream.
[0073] Here, the maximum difference duration threshold is greater than the minimum difference duration threshold. Both the maximum and minimum difference duration thresholds can be flexibly set as needed. For example, the maximum difference duration threshold can be positive, and the minimum difference duration threshold can be negative. The relative magnitude between the absolute values of the maximum and minimum difference duration thresholds is not limited; for example, the absolute values of the maximum and minimum difference duration thresholds can be equal or unequal. The specific values of a and b are not limited; for example, a can be set to a value such as 1.5, and / or b can be set to a value such as 0.5.
[0074] The data stream synchronization method provided in this embodiment obtains the first end-to-end delay of at least two data streams, which are data streams to be synchronized; determines the target end-to-end delay to which the at least two data streams need to be synchronized based on the first end-to-end delay; and performs synchronous playback control on the at least two data streams based on the target end-to-end delay. This embodiment utilizes the above technical solution to determine the target end-to-end delay shared by the multiple data streams for synchronous playback control based on the first end-to-end delay of the multiple data streams to be synchronized. It does not require using a single data stream as a reference, and can comprehensively consider the delay of the multiple data streams to be synchronized, avoiding convergence failures due to mutual delay interference, and improving the synchronization accuracy of multiple data streams.
[0075] Figure 2This is a flowchart illustrating another data stream synchronization method provided in an embodiment of this disclosure. The scheme in this embodiment can be combined with one or more optional schemes in the above embodiments. Optionally, obtaining the first end-to-end delay of at least two data streams includes: obtaining the second end-to-end delay of each of the at least two data streams; and determining the first end-to-end delay of the at least two data streams based on the second end-to-end delay of the at least two data streams.
[0076] Correspondingly, such as Figure 2 As shown, the data stream synchronization method provided in this embodiment may include:
[0077] S201. Obtain the second end-to-end delay of each of at least two data streams.
[0078] The second end-to-end delay of a certain data stream can be understood as a specific end-to-end delay of that data stream, such as the maximum end-to-end delay, minimum end-to-end delay, average end-to-end delay (i.e., the average of end-to-end delays), or median end-to-end delay (i.e., the median of end-to-end delays) of that data stream within the current acquisition period. The current acquisition period can be the current acquisition period. In this embodiment, the second end-to-end delay of each data stream can be acquired periodically according to a preset acquisition period. The period length of this acquisition period is not limited; for example, the period length can be set to 300ms, 500ms, or 1000ms, etc. The period length of this acquisition period can be less than or equal to the period length of the adjustment period for the target end-to-end delay.
[0079] In this embodiment, the second end-to-end delay of each of the at least two data streams to be synchronized can be obtained. For example, based on the specific meaning of the second end-to-end delay, the second end-to-end delay of each data stream in the current acquisition period can be obtained respectively.
[0080] In some implementations, acquiring the second end-to-end delay of each of the at least two data streams includes: acquiring multiple propagation delays of the current data stream within a current acquisition period for each of the at least two data streams; determining a target propagation delay of the current data stream within the current acquisition period based on the multiple propagation delays; and calculating the sum of the target propagation delay and the jitter immunity delay of the current data stream as the second end-to-end delay of the current data stream.
[0081] The current data stream can be any one of the at least two data streams to be synchronized, where the second end-to-end delay is determined. In other words, each of the at least two data streams to be synchronized can be used as the current data stream to determine its second end-to-end delay.
[0082] Propagation delay can be defined as the time taken for a data stream to propagate, such as the time interval between data acquisition and reception. It can be determined by calculating the difference between the reception time and acquisition time of a data packet. The reception time can be understood as the time when the data packet is received. Target propagation delay can be understood as the propagation delay of a data stream within the current acquisition period, such as the maximum propagation delay, minimum propagation delay, average propagation delay (i.e., the average value of the propagation delay), or median propagation delay (i.e., the median value of the propagation delay) within the current acquisition period.
[0083] Jitter-resistant latency can be understood as the latency caused by network jitter. It can be the latency corresponding to the jitter buffer. The jitter-resistant latency can be determined using the mature logic of the existing jitter buffer in the prior art. This embodiment does not limit this.
[0084] In the above implementation, the second end-to-end delay of a certain data stream can be calculated based on the propagation delay and jitter resistance delay of that data stream. That is, the end-to-end delay can be split into two parts: propagation delay and jitter resistance delay. This allows the original jitter buffer delay estimation to be reused to take into account the impact of network jitter. At the same time, it is decoupled from the jitter buffer delay estimation, which can improve the synchronization effect of multiple data streams, enhance the data stream synchronization's resistance to weak networks, and reduce end-to-end delay.
[0085] Specifically, for each of the at least two data streams to be synchronized, multiple propagation delays calculated for the current data stream within the current acquisition period can be obtained; the target propagation delay for the current data stream within the current acquisition period can be determined based on these multiple propagation delays; the jitter-resistant delay of the current data stream can be obtained, such as the jitter-resistant delay of the current data stream within the current acquisition period; the target propagation delay and the jitter-resistant delay can be added together to obtain the sum of the target propagation delay and the jitter-resistant delay, and the sum of the target propagation delay and the jitter-resistant delay can be used as the second end-to-end delay of the current data stream.
[0086] In the above embodiments, the method for determining the multiple propagation delays of the current data stream within the current acquisition period is not limited. For example, the propagation delays of at least two data packets received by the current data stream within the current acquisition period can be calculated as the propagation delay of the current data stream within the current acquisition period. For instance, the propagation delay of the current data stream can be calculated periodically according to a preset calculation period, and the multiple propagation delays of the current data stream calculated within the current acquisition period can be obtained as the propagation delay of the current data stream within the current acquisition period. Optionally, before obtaining the multiple propagation delays of the current data stream within the current acquisition period, the method further includes: periodically calculating the propagation delay of the current data stream, wherein the length of the calculation period for the propagation delay is less than the length of the current acquisition period, and the length of the calculation period is the length of the calculation period. This calculation period can be understood as the period for calculating the propagation delay, and the length of the calculation period (i.e., the calculation period length) can be less than the length of the current acquisition period to ensure that multiple propagation delays can be calculated within the current acquisition period. The calculation period length is unlimited; for example, it can be set to 50ms, 80ms, or 100ms.
[0087] In this embodiment, the target propagation delay can correspond to the second end-to-end delay. For example, when the second end-to-end delay is the maximum end-to-end delay, the minimum end-to-end delay, the average end-to-end delay, or the median end-to-end delay, the target propagation delay can correspond to the maximum propagation delay, the minimum propagation delay, the average propagation delay (i.e., the average value of the propagation delay), or the median propagation delay (i.e., the median of the propagation delay), respectively.
[0088] Taking the second end-to-end delay as the minimum end-to-end delay as an example, the target propagation delay can be the minimum propagation delay of the current data stream within the current acquisition period. Therefore, after acquiring multiple propagation delays of the current data stream within the current acquisition period, the minimum propagation delay among these multiple propagation delays (i.e., the minimum value among these multiple propagation delays) can be used as the target propagation delay of the current data stream within the current acquisition period. Optionally, determining the target propagation delay of the current data stream within the current acquisition period based on the multiple propagation delays includes: using the minimum propagation delay among the multiple propagation delays as the target propagation delay of the current data stream within the current acquisition period.
[0089] S202. Determine the first end-to-end delay of the at least two data streams based on the second end-to-end delay of the at least two data streams.
[0090] Specifically, after obtaining the second end-to-end delay of each of the at least two data streams to be synchronized, the first end-to-end delay of these at least two data streams can be determined based on the second end-to-end delay of each data stream.
[0091] In this embodiment, the determination of the first end-to-end delay can be based on the specific meaning of the first end-to-end delay. For example, when the first end-to-end delay is the maximum end-to-end delay, minimum end-to-end delay, average end-to-end delay, or median end-to-end delay of at least two data streams in the current acquisition period, the maximum second end-to-end delay, the minimum second end-to-end delay, the average of all second end-to-end delays, or the median of all second end-to-end delays among the second end-to-end delays of each data stream can be obtained as the common first end-to-end delay of the at least two data streams.
[0092] In some implementations, the first end-to-end delay can be the maximum end-to-end delay of the at least two data streams to be synchronized within the current acquisition period. Therefore, after acquiring the second end-to-end delay of each data stream, the largest second end-to-end delay among the second end-to-end delays of each data stream (i.e., the maximum value among all second end-to-end delays) can be used as the first end-to-end delay of these at least two data streams. Optionally, determining the first end-to-end delay of the at least two data streams based on their second end-to-end delays includes: using the largest second end-to-end delay among the second end-to-end delays of the at least two data streams as the first end-to-end delay of the at least two data streams.
[0093] S203. Determine the target end-to-end delay to which the at least two data streams are to be synchronized based on the first end-to-end delay.
[0094] S204. Perform synchronous playback control on the at least two data streams based on the target end-to-end delay.
[0095] The data stream synchronization method provided in this embodiment can improve the synchronization accuracy of multiple data streams and enhance the synchronization effect between multiple data streams.
[0096] Figure 3 A flowchart illustrating an optional data stream synchronization method provided in this embodiment of the disclosure is shown below. Figure 3 As shown in the figure (taking three data streams to be synchronized as an example), in some optional embodiments, a common target end-to-end delay can be determined for all data streams to be synchronized. The playback of all data streams is aligned with this target end-to-end delay, making the actual end-to-end delays of all data streams equal or similar, thereby achieving synchronization. The data stream synchronization method provided in this embodiment can be described as follows:
[0097] S0) Initialize the common target end-to-end delay final_target_e2e_delay of all data streams to 0 (i.e., a default invalid value), and cyclically execute the following operations S1-S5 according to the time intervals (i.e., period lengths) corresponding to the steps below:
[0098] S1) For each data stream, calculate the propagation delay (e.g., the delay from acquisition to reception) every update_duration (i.e., the period length of the calculation cycle, such as 80ms), and estimate the minimum propagation delay (i.e., the target propagation delay) of the data stream individually. For example, the propagation delay can be filtered by a minimum value in a 500ms window (e.g., take the minimum value of the data stream's propagation delay prop_delay within that 500ms interval), thus obtaining the minimum propagation delay min_prop_delay. This minimum propagation delay is effective within its 500ms window.
[0099] S2) For each data stream, estimate the jitter buffer delay required for jitter mitigation (i.e., jitter mitigation delay) separately. Network jitter estimation can reuse the existing mature jitter buffer logic.
[0100] S3) For each data stream, estimate its minimum required end-to-end delay target_e2e_delay (i.e., the second end-to-end delay), that is, target_e2e_delay = min_prop_delay + jitter_buffer_delay.
[0101] In this embodiment, the minimum end-to-end delay of a single data stream can be decomposed into minimum propagation delay + jitter resistance delay. Propagation delay can be considered as the delay required for propagation (such as the inherent delay of optical fiber / network cable, the inherent delay of switch processing, etc.), and jitter resistance delay can be considered as the jitter delay introduced by the processing stage and network transmission. The minimum propagation delay can be the minimum delay from acquisition to reception at the receiving end, and the jitter resistance delay can be the buffer delay from reception to playback at the receiving end. These two segments are mutually exclusive. Figure 4 and 5 As shown in the figure. Since there are already mature methods for calculating jitter-resistant delay, it is only necessary to estimate the minimum propagation delay.
[0102] S4) The decision module performs the following operation every 1 second (i.e., the period length of the adjustment period): obtain the minimum required end-to-end delay target_e2e_delay for each data stream, and take the maximum value max_target_e2e_delay (i.e., the first end-to-end delay) among the minimum end-to-end delays of all data streams.
[0103] If the target end-to-end delay final_target_e2e_delay is currently a preset invalid value of 0, then the target end-to-end delay final_target_e2e_delay will be directly adjusted to the currently determined maximum value max_target_e2e_delay.
[0104] If the target end-to-end delay `final_target_e2e_delay` is not currently a preset invalid value of 0, then `final_target_e2e_delay` is smoothly adjusted based on the difference between `final_target_e2e_delay` and `max_target_e2e_delay` to avoid desynchronization caused by excessively rapid adjustment of `final_target_e2e_delay`.
[0105] If max_target_e2e_delay - final_target_e2e_delay > inc_threshold (i.e., the first preset threshold), a first adjustment step can be added to final_target_e2e_delay. The first adjustment step can be min(inc_step, max_target_e2e_delay - final_target_e2e_delay).
[0106] If max_target_e2e_delay - final_target_e2e_delay < -dec_threshold (i.e., the second preset threshold), final_target_e2e_delay can be reduced by a second adjustment step, which can be min(dec_step, final_target_e2e_delay - max_target_e2e_delay).
[0107] Therefore, as Figure 6 As shown, the target end-to-end delay can exhibit a step-like increasing or step-like decreasing trend when adjusted. Figure 6 (Taking the step-by-step descent as an example).
[0108] Here, the threshold and step size can be configured as needed. The threshold can be used to control the sensitivity of playback control, and the step size can be used to control the acceleration and deceleration speed.
[0109] In this embodiment, it is assumed that there are N data streams that need to be synchronized, and the minimum required delays for these N data streams are: D1, D2, ..., D... NThis means that the i-th stream will only stop lagging once it reaches this delay. The target delay for all data streams is T. When T < D i This can introduce stuttering. Therefore, synchronization can be transformed into an optimization problem.
[0110] This allows all data streams to be sorted in ascending order of their required latency. Assuming m data streams experience buffering and Nm data streams introduce additional latency, the cost function is as follows:
[0111]
[0112] k 1,i K is the stuttering coefficient of the i-th data stream. 2,i It is the delay coefficient of the i-th data stream. For synchronous scenarios, higher demands are placed on synchronization quality, while delay requirements are not as stringent; viewers may not even be concerned about delay. Therefore, the delay coefficient k can be... 2,i Setting it to 0 means the cost primarily comes from stuttering. When optimizing for stuttering, m can be set to 0, meaning all streams are free of stuttering. Therefore, we have:
[0113] T≥max(D1, D2, ..., D) N ).
[0114] Therefore, the maximum value max_target_e2e_delay can be taken from the target_e2e_delay of all data streams to ensure that all data streams are not interrupted by a certain delay.
[0115] S5) Estimate the actual end-to-end delay of each stream every 10ms (i.e., the length of the playback control cycle) using the playback thread: real_e2e_delay = playout_time (playout time) - capture_time (capture time). Compare this delay with the target end-to-end delay to obtain the synchronization difference duration of the data stream: sync_diff = real_e2e_delay - final_target_e2e_delay. Adjust playback speed based on the difference between the real_e2e_delay and the target end-to-end delay. Generally, a threshold can be set (e.g., 10ms). Within the threshold range, no adjustment is needed to speed up or slow down the data stream.
[0116] If sync_diff is greater than the maximum threshold (i.e., the maximum difference duration threshold, such as 10ms), it means that the actual end-to-end latency of the data stream is higher than the target end-to-end latency, and the playback of the data stream can be accelerated.
[0117] If sync_diff is less than the minimum threshold (i.e., the minimum difference duration threshold, such as -10ms), it means that the data stream is playing too fast, and the data stream can be slowed down and aligned.
[0118] If sync_diff is within the threshold range (i.e., greater than or equal to the minimum threshold and less than or equal to the maximum threshold), it indicates that the data stream has good synchronization and the playback speed of the data stream does not need to be adjusted.
[0119] The structure of a feedback system is as follows Figure 7 As shown, each data stream is continuously accelerated or decelerated until the playback delay is close to or equal to the target end-to-end delay.
[0120] Furthermore, in this embodiment, the capture time (i.e., the capture time point) can be parsed from the received data packet (e.g., the sender sends the data packet to the receiver using an extended transmission method). The data packet carries the capture time (e.g., the sender's NTP time), and all end times have been synchronized, such as N sender times already synchronized. The receive time (i.e., the receive time point) can be the time the receiver receives the data packet (e.g., the NTP time the receiver receives the data packet). The playout time (i.e., the playout time point) can be the time the receiver plays the data packet (e.g., the NTP time the receiver plays the corresponding data packet).
[0121] real_e2e_delay = playback time - capture time;
[0122] prop_delay = Receive time - Acquisition time;
[0123] target_e2e_delay=min_prop_delay+jitter_buffer_delay;
[0124] sync_diff=real_e2e_delay-final_target_e2e_delay.
[0125] When determining real_e2e_delay, prop_delay (such as min_prop_delay), and target_e2e_delay, the NTP time difference between the sender and receiver can be ignored. Therefore, the NTP time difference between the sender and receiver can be canceled out when calculating sync_diff. This further reduces the computational load required during synchronization while ensuring synchronization effectiveness.
[0126] Therefore, this embodiment provides a synchronization scheme without a reference stream. It introduces minimum propagation delay, decomposes the delay of each data stream into minimum propagation delay and anti-jitter delay, thereby obtaining the minimum required delay of each data stream. Then, it combines the results of all data streams to determine the global shared delay (i.e., the target end-to-end delay), smooths the global shared delay, and accurately controls the playback progress, thereby achieving global synchronization of all data streams.
[0127] For example, by introducing minimum propagation delay, the end-to-end delay is reasonably decomposed, allowing the reuse of the original jitter buffer delay estimation while taking into account the impact of network jitter. At the same time, it is decoupled from the jitter buffer delay estimation, achieving good resistance to weak networks, minimum end-to-end delay, high synchronization, and low performance overhead. By periodically deciding on the globally optimal target end-to-end delay, all data streams are aligned with the target end-to-end delay, thus ensuring accurate synchronization by guaranteeing that the end-to-end delays of all data streams are similar. By controlling the adjustment of the global target end-to-end delay, the impact of acceleration and deceleration is reduced, improving the user experience.
[0128] The data stream synchronization method provided in this embodiment can solve the problem of how to stably and continuously synchronize multiple data streams, adapt to the synchronization of multiple data streams, ensure that all data streams reach or approach the target end-to-end latency, while taking into account network jitter, and achieves simple logic and low performance overhead, and can maintain synchronization in various weak network scenarios.
[0129] Figure 8 This is a structural block diagram of a data stream synchronization device provided in an embodiment of this disclosure. The device can be implemented in software and / or hardware, and can be configured in an electronic device, typically a mobile phone or tablet computer. It can synchronize multiple data streams by executing a data stream synchronization method. Figure 8 As shown, the data stream synchronization device provided in this embodiment may include: a delay acquisition module 801, a delay determination module 802, and a playback control module 803, wherein,
[0130] The delay acquisition module 801 is used to acquire the first end-to-end delay of at least two data streams, wherein the at least two data streams are data streams to be synchronized;
[0131] Delay determination module 802 is used to determine the target end-to-end delay to be synchronized to the at least two data streams based on the first end-to-end delay;
[0132] The playback control module 803 is used to perform synchronous playback control on the at least two data streams based on the target end-to-end delay.
[0133] The data stream synchronization device provided in this embodiment acquires the first end-to-end delay of at least two data streams through a delay acquisition module, wherein the at least two data streams are data streams to be synchronized; a delay determination module determines the target end-to-end delay to which the at least two data streams are to be synchronized based on the first end-to-end delay; and a playback control module performs synchronous playback control on the at least two data streams based on the target end-to-end delay. This embodiment utilizes the above technical solution to determine the target end-to-end delay shared by the multiple data streams for synchronous playback control based on the first end-to-end delay of the multiple data streams to be synchronized. It does not require using a single data stream as a reference, and can comprehensively consider the delay of the multiple data streams to be synchronized, avoiding convergence failures due to mutual delay interference, and improving the synchronization accuracy of the multiple data streams.
[0134] Optionally, the delay acquisition module 801 includes: a first delay acquisition unit, configured to acquire the second end-to-end delay of each of the at least two data streams; and a delay determination unit, configured to determine the first end-to-end delay of the at least two data streams based on the second end-to-end delay of the at least two data streams.
[0135] Optionally, the first delay acquisition unit includes: a delay acquisition subunit, configured to acquire multiple propagation delays of the current data stream within a current acquisition period for each of the at least two data streams; a delay determination subunit, configured to determine a target propagation delay of the current data stream within the current acquisition period based on the multiple propagation delays; and a delay calculation subunit, configured to calculate the sum of the target propagation delay and the jitter resistance delay of the current data stream as the second end-to-end delay of the current data stream.
[0136] Furthermore, the data stream synchronization device provided in this embodiment may further include: a delay calculation module, used to periodically calculate the propagation delay of the current data stream before acquiring multiple propagation delays of the current data stream within the current acquisition period, wherein the calculation period length of the propagation delay is less than the period length of the current acquisition period, and the calculation period length is the period length of the calculation period.
[0137] Optionally, the delay determination subunit is specifically used to: take the smallest propagation delay among the plurality of propagation delays as the target propagation delay of the current data stream within the current acquisition period.
[0138] Optionally, the delay determination unit is specifically used to: take the largest second end-to-end delay among the second end-to-end delays of the at least two data streams as the first end-to-end delay of the at least two data streams.
[0139] Optionally, the delay determination module 802 includes: an original value acquisition unit, used to acquire the original value of the target end-to-end delay; a target value determination unit, used to determine the target value of the target end-to-end delay based on the original value and the delay value of the first end-to-end delay; and a delay adjustment unit, used to adjust the target end-to-end delay from the original value to the target value.
[0140] Optionally, the target value determination unit includes: a first determination subunit, configured to take the delay value of the first end-to-end delay as the target value of the target end-to-end delay when the original value is a preset invalid value; and / or, a second determination subunit, configured to calculate the difference between the delay value of the first end-to-end delay and the original value when the original value is not a preset invalid value, and determine the target value of the target end-to-end delay based on the difference.
[0141] Optionally, the second determining subunit is configured to perform at least one of the following: if the difference is greater than a first preset threshold, then the original value is increased by a first adjustment step to obtain a target value for the target end-to-end delay, wherein the first preset threshold is a positive value; if the difference is less than or equal to the first preset threshold, and greater than or equal to a second preset threshold, then the original value is used as the target value for the target end-to-end delay; if the difference is less than the second preset threshold, then the original value is decreased by a second adjustment step to obtain a target value for the target end-to-end delay, wherein the second preset threshold is a negative value.
[0142] Optionally, the playback control module 803 includes: a second delay acquisition unit, used to acquire the actual end-to-end delay of the current data stream for each of the at least two data streams; a duration calculation unit, used to calculate the synchronization difference duration of the current data stream based on the actual end-to-end delay and the target end-to-end delay; and a playback control unit, used to perform playback control on the current data stream based on the synchronization difference duration.
[0143] The data stream synchronization apparatus provided in this disclosure can execute the data stream synchronization method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the data stream synchronization method. Technical details not described in detail in this embodiment can be found in the data stream synchronization method provided in any embodiment of this disclosure.
[0144] The following is for reference. Figure 9The diagram illustrates a structural schematic of an electronic device (e.g., a terminal device) 900 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0145] like Figure 9 As shown, electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 902 or a program loaded from storage device 908 into random access memory (RAM) 903. RAM 903 also stores various programs and data required for the operation of electronic device 900. Processing device 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.
[0146] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.
[0147] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.
[0148] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0149] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.
[0150] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.
[0151] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire a first end-to-end delay of at least two data streams, wherein the at least two data streams are data streams to be synchronized; determine a target end-to-end delay to which the at least two data streams are to be synchronized based on the first end-to-end delay; and perform synchronous playback control on the at least two data streams based on the target end-to-end delay.
[0152] Computer program code for performing the operations of this disclosure can be written in one or more programming languages or a combination thereof, including but not limited to object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0154] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of modules do not necessarily limit the specific unit itself.
[0155] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.
[0156] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0157] According to one or more embodiments of this disclosure, Example 1 provides a method for synchronizing a data stream, including:
[0158] Obtain the first end-to-end delay of at least two data streams, wherein the at least two data streams are data streams to be synchronized;
[0159] The target end-to-end delay to which the at least two data streams need to be synchronized is determined based on the first end-to-end delay.
[0160] Synchronous playback control is performed on the at least two data streams based on the target end-to-end delay.
[0161] According to one or more embodiments of this disclosure, Example 2, based on the method described in Example 1, includes obtaining the first end-to-end delay of at least two data streams, comprising:
[0162] Obtain the second end-to-end delay for each of at least two data streams;
[0163] The first end-to-end delay of the at least two data streams is determined based on the second end-to-end delay of the at least two data streams.
[0164] According to one or more embodiments of this disclosure, Example 3, based on the method described in Example 2, includes obtaining the second end-to-end delay of each of at least two data streams, comprising:
[0165] For each of the at least two data streams, obtain multiple propagation delays of the current data stream within the current acquisition period;
[0166] The target propagation delay of the current data stream within the current acquisition period is determined based on the multiple propagation delays;
[0167] The sum of the target propagation delay and the jitter-resistant delay of the current data stream is calculated as the second end-to-end delay of the current data stream.
[0168] According to one or more embodiments of this disclosure, Example 4, based on the method of Example 3, further includes, before acquiring multiple propagation delays of the current data stream within the current acquisition period:
[0169] The propagation delay of the current data stream is calculated periodically, wherein the calculation period length of the propagation delay is less than the period length of the current acquisition period, and the calculation period length is the period length of the calculation period.
[0170] According to one or more embodiments of this disclosure, Example 5 describes the method described in Example 3, wherein determining the target propagation delay of the current data stream within the current acquisition period based on the plurality of propagation delays includes:
[0171] The smallest of the multiple propagation delays is taken as the target propagation delay of the current data stream within the current acquisition period.
[0172] According to one or more embodiments of this disclosure, Example 6 describes the method of Example 2, wherein determining the first end-to-end delay of the at least two data streams based on the second end-to-end delay of the at least two data streams includes:
[0173] The largest second end-to-end delay among the second end-to-end delays of the at least two data streams is taken as the first end-to-end delay of the at least two data streams.
[0174] According to one or more embodiments of this disclosure, Example 7, based on the method of Example 1, the step of determining the target end-to-end delay to which the at least two data streams are to be synchronized based on the first end-to-end delay includes:
[0175] Obtain the original value of the target end-to-end delay;
[0176] The target value of the target end-to-end delay is determined based on the original value and the delay value of the first end-to-end delay.
[0177] The target end-to-end delay is adjusted from the original value to the target value.
[0178] According to one or more embodiments of this disclosure, Example 8 describes the method of Example 7, wherein determining the target value of the target end-to-end delay based on the original value and the delay value of the first end-to-end delay includes:
[0179] If the original value is a preset invalid value, then the delay value of the first end-to-end delay is taken as the target value of the target end-to-end delay; and / or
[0180] If the original value is not a preset invalid value, the difference between the delay value of the first end-to-end delay and the original value is calculated, and the target value of the target end-to-end delay is determined based on the difference.
[0181] According to one or more embodiments of this disclosure, Example 9 describes the method of Example 8, wherein determining the target value of the target end-to-end delay based on the difference includes at least one of the following:
[0182] If the difference is greater than the first preset threshold, the original value is increased by the first adjustment step to obtain the target value of the target end-to-end delay, wherein the first preset threshold is a positive value;
[0183] If the difference is less than or equal to a first preset threshold and greater than or equal to a second preset threshold, then the original value is taken as the target value for the target end-to-end delay.
[0184] If the difference is less than the second preset threshold, the original value is reduced by the second adjustment step to obtain the target value of the target end-to-end delay, wherein the second preset threshold is a negative value.
[0185] According to one or more embodiments of this disclosure, Example 10, based on any one of Examples 1-9, includes performing synchronous playback control on the at least two data streams based on the target end-to-end delay, comprising:
[0186] For each of the at least two data streams, obtain the actual end-to-end delay of the current data stream;
[0187] The synchronization difference duration of the current data stream is calculated based on the actual end-to-end delay and the target end-to-end delay;
[0188] Playback control of the current data stream is performed based on the synchronization difference duration.
[0189] According to one or more embodiments of this disclosure, Example 11 provides a data stream synchronization apparatus, comprising:
[0190] The delay acquisition module is used to acquire the first end-to-end delay of at least two data streams, wherein the at least two data streams are data streams to be synchronized;
[0191] The delay determination module is used to determine the target end-to-end delay to which the at least two data streams are to be synchronized based on the first end-to-end delay;
[0192] The playback control module is used to perform synchronous playback control on the at least two data streams based on the target end-to-end delay.
[0193] According to one or more embodiments of this disclosure, Example 12 provides an electronic device, including:
[0194] One or more processors;
[0195] Memory, used to store one or more programs.
[0196] When the one or more programs are executed by the one or more processors, the one or more processors implement a data stream synchronization method as described in any of Examples 1-10.
[0197] According to one or more embodiments of the present disclosure, Example 13 provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements a method for synchronizing data streams as described in any of Examples 1-10.
[0198] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.
[0199] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.
[0200] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.
Claims
1. A method for synchronizing a data stream, characterized in that, include: Obtain the first end-to-end delay of at least two data streams, wherein the at least two data streams are data streams to be synchronized; The target end-to-end delay to which the at least two data streams need to be synchronized is determined based on the first end-to-end delay. Synchronous playback control of at least two data streams is performed based on the target end-to-end delay. The step of determining the target end-to-end delay to which the at least two data streams are to be synchronized based on the first end-to-end delay includes: Obtain the original value of the target end-to-end delay, wherein the original value of the target end-to-end delay is the delay value of the target end-to-end delay in the previous adjustment period; The target value of the target end-to-end delay is determined based on the original value and the delay value of the first end-to-end delay. The target end-to-end delay is adjusted from the original value to the target value.
2. The method according to claim 1, characterized in that, The acquisition of the first end-to-end delay of at least two data streams includes: Obtain the second end-to-end delay for each of at least two data streams; The first end-to-end delay of the at least two data streams is determined based on the second end-to-end delay of the at least two data streams.
3. The method according to claim 2, characterized in that, The acquisition of the second end-to-end delay of each of at least two data streams includes: For each of the at least two data streams, obtain multiple propagation delays of the current data stream within the current acquisition period; The target propagation delay of the current data stream within the current acquisition period is determined based on the multiple propagation delays; The sum of the target propagation delay and the jitter-resistant delay of the current data stream is calculated as the second end-to-end delay of the current data stream.
4. The method according to claim 3, characterized in that, Before acquiring the multiple propagation delays of the current data stream within the current acquisition period, the method further includes: The propagation delay of the current data stream is calculated periodically, wherein the calculation period length of the propagation delay is less than the period length of the current acquisition period, and the calculation period length is the period length of the calculation period.
5. The method according to claim 3, characterized in that, Determining the target propagation delay of the current data stream within the current acquisition period based on the plurality of propagation delays includes: The smallest of the multiple propagation delays is taken as the target propagation delay of the current data stream within the current acquisition period.
6. The method according to claim 2, characterized in that, Determining the first end-to-end delay of the at least two data streams based on the second end-to-end delay of the at least two data streams includes: The largest second end-to-end delay among the second end-to-end delays of the at least two data streams is taken as the first end-to-end delay of the at least two data streams.
7. The method according to claim 1, characterized in that, Determining the target value of the target end-to-end delay based on the original value and the delay value of the first end-to-end delay includes: If the original value is a preset invalid value, then the delay value of the first end-to-end delay is taken as the target value of the target end-to-end delay; and / or If the original value is not a preset invalid value, the difference between the delay value of the first end-to-end delay and the original value is calculated, and the target value of the target end-to-end delay is determined based on the difference.
8. The method according to claim 7, characterized in that, Determining the target value of the target end-to-end delay based on the difference includes at least one of the following: If the difference is greater than the first preset threshold, the original value is increased by the first adjustment step to obtain the target value of the target end-to-end delay, wherein the first preset threshold is a positive value; If the difference is less than or equal to a first preset threshold and greater than or equal to a second preset threshold, then the original value is taken as the target value for the target end-to-end delay. If the difference is less than the second preset threshold, the original value is reduced by the second adjustment step to obtain the target value of the target end-to-end delay, wherein the second preset threshold is a negative value.
9. The method according to any one of claims 1-8, characterized in that, The synchronous playback control of the at least two data streams based on the target end-to-end delay includes: For each of the at least two data streams, obtain the actual end-to-end delay of the current data stream; The synchronization difference duration of the current data stream is calculated based on the actual end-to-end delay and the target end-to-end delay; Playback control of the current data stream is performed based on the synchronization difference duration.
10. A data stream synchronization device, characterized in that, include: The delay acquisition module is used to acquire the first end-to-end delay of at least two data streams, wherein the at least two data streams are data streams to be synchronized; The delay determination module is used to determine the target end-to-end delay to which the at least two data streams are to be synchronized based on the first end-to-end delay; A playback control module is used to perform synchronous playback control on the at least two data streams based on the target end-to-end delay. The delay determination module includes: The original value acquisition unit is used to acquire the original value of the target end-to-end delay, wherein the original value of the target end-to-end delay is the delay value of the target end-to-end delay in the previous adjustment period; A target value determination unit is used to determine the target value of the target end-to-end delay based on the original value and the delay value of the first end-to-end delay. The delay adjustment unit is used to adjust the target end-to-end delay from the original value to the target value.
11. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the data stream synchronization method according to any one of claims 1-9.
12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method for synchronizing the data stream as described in any one of claims 1-9.
Citation Information
Patent Citations
Data stream synchronization method and device
CN112135177A
