Method and apparatus for synchronizing multiple media streams, medium, and device
By calculating the relative delay of the target media stream based on the timestamp modification amount of the first media stream and updating the timestamps of each media stream, the problem of multiple media streams is solved, real-time and accurate media stream synchronization is achieved, and it is suitable for live broadcast and video conferencing and other scenarios.
Patent Information
- Application Number
- PCT/CN2024/094181
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-23
- Filing Date
- 2024-05-20
- Publication Date
- 2024-11-28
AI Technical Summary
The prior art is difficult to realize real-time synchronization of multiple media streams in scenarios such as live broadcast and video conferencing, resulting in the possibility of out-of-synchronization of media streams when playing.
When receiving the media packet of the target media stream, the initial timestamp modification amount of the target media stream is determined based on the current timestamp modification amount of the first media stream, the relative delay of the target media stream relative to the first media stream is calculated, and the already updated The time stamp modification amount of each received media stream is modified, and the packet time stamp of each media stream is modified to realize media stream synchronization.
Real-time synchronization of multiple media streams is achieved, ensuring that the new media streams received by media streams can be accurately synchronized at any time, avoiding the problem of timestamp fallback, and is suitable for live broadcasts and video conferencing in unreliable transmission scenarios.
Smart Images

Figure CN2024094181_28112024_PF_FP_ABST
Abstract
Description
Method, device, medium and equipment for synchronizing multiple media streams
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is based on and claims priority to a Chinese application with application number 202310583556.7, filing date May 23, 2023, entitled “Methods, devices, media and equipment for synchronizing multiple media streams”. The disclosed content of the Chinese application is hereby introduced as a whole into this application. Technical Field
[0003] The embodiments of this specification relate to the field of information technology, and in particular to a method, apparatus, medium, and device for synchronizing multiple media streams. Background Art
[0004] With the rapid development of computer technology and Internet technology, applications involving multimedia data are becoming more and more popular and deeply loved by users.
[0005] Summary of the Invention
[0006] The embodiments of this specification describe a method, apparatus, medium, and device for synchronizing multiple media streams. When a target media stream is received, the timestamps of the data packets of each media stream can be modified based on the relative delay of the target media stream with respect to a first media stream, thereby achieving real-time synchronization of the multiple media streams.
[0007] According to a first aspect, a method for synchronizing multiple media streams is provided, the method comprising: in response to receiving a media data packet of a target media stream, determining an initial timestamp modification amount for the media data packet of the target media stream based on a current timestamp modification amount of a received first media stream; determining a relative delay of the target media stream relative to the first media stream; updating a timestamp modification amount for each received media stream based on the relative delay; and modifying the timestamps of the data packets of each media stream based on the timestamp modification amount of each media stream.
[0008] In one embodiment, updating the timestamp modification amount of each received media stream based on the relative delay includes: determining the relative magnitude relationship between the delay of the target media stream and the delay of the first media stream based on the relative delay; and, in response to determining that the delay of the target media stream is smaller than the delay of the first media stream, adding the modification amount corresponding to the relative delay to the current timestamp modification amount of each received media stream other than the target media stream. Thus, media stream synchronization is achieved when the delay of the target media stream is smaller than the delay of the first media stream.
[0009] In one embodiment, updating the timestamp modification amount of each received media stream based on the relative delay further includes: in response to determining that the delay of the target media stream is greater than the delay of the first media stream, adding the modification amount corresponding to the relative delay to the current timestamp modification amount of the target media stream. In this way, media stream synchronization is achieved when the delay of the target media stream is greater than the delay of the first media stream.
[0010] In one embodiment, the method further includes: receiving synchronization information of the target media stream; and determining the relative delay of the target media stream relative to the first media stream includes calculating the relative delay of the target media stream relative to the first media stream based on the synchronization information of the target media stream and previously received synchronization information of the first media stream, wherein the synchronization information includes time information in a sender report. Thus, the relative delay between media streams is determined based on the time information in the sender report.
[0011] In one embodiment, before determining the relative delay of the target media stream relative to the first media stream, the method further includes: receiving and storing synchronization information of the first media stream, thereby achieving the reception and storage of the synchronization information of the first media stream.
[0012] In one embodiment, before receiving the media data packet of the target media stream, the method further includes: receiving a data packet of a first media stream, and setting the initial timestamp modification value of the first media stream to 0. Thus, the initial timestamp modification value of the first media stream is set.
[0013] In one embodiment, before receiving the media data packet of the target media stream, media data packets of one or more media streams have been received, thereby achieving synchronization of two or more media streams including the target media stream.
[0014] According to a second aspect, a device for synchronizing multiple media streams is provided, comprising: a first determining unit configured to, in response to receiving a media data packet of a target media stream, determine an initial timestamp modification amount of the media data packet of the target media stream based on the current timestamp modification amount of the received first media stream; a second determining unit configured to determine a relative delay of the target media stream relative to the first media stream; an updating unit configured to update the timestamp modification amount of each received media stream based on the relative delay; and a modifying unit configured to modify the timestamp of the data packet of each media stream based on the timestamp modification amount of each media stream.
[0015] According to a third aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above methods in the first aspect.
[0016] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute any one of the methods in the first aspect.
[0017] According to a fifth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, any one of the above methods in the first aspect is implemented.
[0018] According to the method and apparatus for synchronizing multiple media streams provided by the embodiments of this specification, when a media data packet of a target media stream is received, the initial timestamp modification amount of the media data packet of the target media stream can be determined based on the current timestamp modification amount of the first media stream. Then, the relative delay of the target media stream relative to the first media stream is determined, and the timestamp modification amount of each received media stream is updated based on the relative delay. Finally, the timestamp of the data packet of each media stream is modified based on the timestamp modification amount of each media stream. Thus, when the target media stream is received, the timestamp of the data packet of each media stream can be modified based on the relative delay of the target media stream relative to the first media stream, thereby achieving real-time synchronization of multiple media streams (including new media streams received at any time). BRIEF DESCRIPTION OF THE DRAWINGS
[0019] FIG1 is a schematic diagram showing an application scenario in which the embodiments of this specification can be applied;
[0020] FIG2 is a schematic flow chart showing a method for synchronizing multiple media streams according to an embodiment;
[0021] FIG3( a ) shows a schematic diagram of the time axis distribution of five media streams in a misaligned state;
[0022] FIG3( b ) shows a schematic diagram of the effect after the first media stream, subsequent media stream ( 2 ), subsequent media stream ( 3 ) and subsequent media stream ( 4 ) are moved downward by t1 ;
[0023] FIG3( c ) shows a schematic diagram of the effect of shifting the subsequent media stream ( 2 ) downward by t0 ;
[0024] FIG3( d ) shows a schematic diagram of the effect of moving the first media stream, the subsequent media stream ( 1 ), the subsequent media stream ( 2 ) and the subsequent media stream ( 4 ) downward by t1 ;
[0025] FIG3(e) shows a schematic diagram of the effect after the first media stream, the subsequent media stream (1), the subsequent media stream (2) and the subsequent media stream (3) are moved downward by t2;
[0026] FIG4 shows a schematic block diagram of an apparatus for synchronizing multiple media streams according to an embodiment;
[0027] FIG5 shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION
[0028] The technical solutions provided in this specification are further described in detail below in conjunction with the accompanying drawings and embodiments. It will be understood that the specific embodiments described herein are intended only to explain the relevant inventions and are not intended to limit the inventions. It should also be noted that, for ease of description, only portions relevant to the relevant inventions are shown in the accompanying drawings. It should be noted that, unless there is a conflict, the embodiments of this specification and the features therein may be combined with one another.
[0029] When there are multiple media streams, it is necessary to synchronize the multiple media streams. Taking live broadcast, video conferencing and other scenarios as examples, it is very necessary to perform real-time time synchronization of multimedia data, which is related to whether the media streams are synchronized when playing. To this end, an embodiment of this specification provides a method for synchronizing multiple media streams, thereby achieving synchronization of multiple media streams. As shown in Figure 1, Figure 1 shows a schematic diagram of an application scenario in which the embodiment of this specification can be applied. In the application scenario shown in Figure 1, device 101 can receive multiple media streams. Each time a new media stream is received, after receiving the synchronization-related data of the media stream, synchronization can be performed based on the synchronization-related data of the media stream. During synchronization, the timestamps of the data packets of each received media stream may be modified. Here, the timestamps of the data packets to be modified may include DTS (Decoding Time Stamp) and PTS (Presentation Time Stamp). In this example, the first media stream received is firstMedia, the current timestamp modification amount of the first media stream is ct(0), and the media streams received after the first media stream are the subsequent media stream (1) laterMedia(1), the subsequent media stream (2) laterMedia(2), ... as an example, wherein the current timestamp modification amount of the first media stream can refer to the change in the timestamp of the first media stream at the current moment compared with the original timestamp. When the media data packet of the target media stream laterMedia(i) (where i ≥ 1 and is a positive integer) is received, the initial timestamp modification amount of the media data packet of the target media stream can be determined based on the current timestamp modification amount ct(0) of the first media stream. That is, the timestamp modification amount ct(i) of the target media stream is initialized. For example, during initialization, ct(i) = ct(0) can be set. At this time, the timestamp of the media data packet of the target media stream can be modified based on ct(i), so that the target media stream inherits the current timestamp modification amount of the first media stream, without changing the original relative delay between the target media stream and the first media stream. Afterwards, synchronization data related to the target media stream can be received and, based on the synchronization data, the relative delay (relative_relay(i)) of the target media stream relative to the first media stream can be determined. Then, the timestamp modification amount of each received media stream is updated based on relative_relay(i). Finally, the timestamps of the data packets of each media stream are modified based on the timestamp modification amount of each media stream. In this way, synchronization of multiple media streams can be achieved.
[0030] Continuing with FIG2 , FIG2 shows a flow chart of a method for synchronizing multiple media streams according to one embodiment. The method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. For example, it can be executed by a terminal device. For another example, it can also be executed by other devices, equipment, etc., and then the synchronized multiple media streams are sent to the terminal device. As shown in FIG2 , the method for synchronizing multiple media streams can include the following steps:
[0031] Step 201: In response to receiving a media data packet of a target media stream, determining an initial timestamp modification amount of the media data packet of the target media stream according to a current timestamp modification amount of a received first media stream.
[0032] In this embodiment, the media stream may include a video stream, an audio stream, a subtitle stream, and the like. A media data packet may refer to a data packet containing media data. Taking a video stream as an example, a media data packet may refer to a data packet containing video frames. When a media data packet of a target media stream is received, the initial timestamp modification of the media data packet of the target media stream may be determined based on the current timestamp modification of the received first media stream. For example, the current timestamp modification of the first media stream may be used as the initial timestamp modification of the target media stream. Thereafter, the timestamp of the media data packet of the target media stream may be modified based on the initial timestamp modification. In this way, the target media stream may inherit the current timestamp modification of the first media stream, thereby ensuring that the relative delay between the target media stream and the first media stream remains unchanged. In this example, the target media stream is the media stream received after the first media stream.
[0033] It can be understood that before receiving the media data packets of the target media stream, media data packets of one or more media streams have already been received. In other words, the target media stream is the Nth media stream received, where N ≥ 2 and N is a positive integer. This allows synchronization of two or more media streams, including the target media stream.
[0034] In some implementations, before receiving the media data packet of the target media stream, the method for synchronizing multiple media streams may further include: receiving a data packet of a first media stream, and setting the initial timestamp modification amount of the first media stream to 0. Thus, the initial timestamp modification amount of the first media stream can be set.
[0035] Step 202: Determine the relative delay of the target media stream relative to the first media stream.
[0036] In this embodiment, the relative delay of the target media stream relative to the first media stream may be determined in various ways.
[0037] As an example, the delay information of the media stream can be carried in the RTCP (Real-time Transport Control Protocol) message of the media stream. For example, the delay information can include a delay value delay. In this way, the relative delay of the target media stream relative to the first media stream can be calculated based on the delay information of the target media stream and the delay information of the first media stream. Taking the target media stream as laterMedia(i) and the first media stream as firstMedia as an example, where i≥1 and i is a positive integer, the relative delay relative_relay(i) of laterMedia(i) relative to firstMedia can be determined by the following method:
[0038] relative_delay(i)=FirstMedia.delay-laterMedia(i).delay,
[0039] Among them, FirstMedia.delay can represent the delay value of firstMedia, and laterMedia(i).delay can represent the delay value of laterMedia(i).
[0040] As another example, the relative delay of the media stream relative to the first media stream can also be directly carried in the RTCP message of the media stream. In this way, the relative delay of the target media stream relative to the first media stream can be directly determined based on the RTCP message.
[0041] It is understandable that the RTCP messages in the above two examples may be received after receiving some media packets of the target media stream. Therefore, synchronization may not be performed when the first media packet of the target media stream is received, and synchronization may be performed after receiving some media packets.
[0042] In some implementations, the above-mentioned method for synchronizing multiple media streams may further include the following steps not shown in FIG. 2 : receiving synchronization information of a target media stream.
[0043] In this implementation, the synchronization information may include ntp (Network Time Protocol) and rtp (Real-time Transport Protocol) time information carried in an RTCP sender report (SR).
[0044] Based on this, the above step 202 may be specifically performed as follows: calculating the relative delay of the target media stream with respect to the first media stream according to the synchronization information of the target media stream and the synchronization information of the first media stream received in advance.
[0045] It is understandable that the synchronization information of the first media stream may be received and stored before this. That is, the target media stream is the second or nth (n>2, and n is a positive integer) media stream received.
[0046] Taking the first media stream as firstMedia and the target media stream as laterMedia(i) as an example, the relative delay relative_relay(i) of laterMedia(i) relative to firstMedia can be determined as follows:
[0047] relative_delay(i)=(FirstMedia.RtpPts-laterMedia(i).RtpPts)-(FirstMedia.NtpTime-laterMedia(i).NtpTime),
[0048] Wherein, RtpPts=rtpTimestamp / clockRate, rtpTimestamp may represent an rtp timestamp, clockRate may represent a clock frequency, and NtpTime may represent an ntp timestamp. FirstMedia.RtpPts may represent the RtpPts of firstMedia. laterMedia(i).RtpPts may represent the RtpPts of laterMedia(i). FirstMedia.NtpTime may represent the NtpTime of FirstMedia. laterMedia(i).NtpTime may represent the NtpTime of laterMedia(i).
[0049] In practice, the synchronization information contained in the SR of RTCP, namely,<rtp,ntp> The timestamp pair can be used to locate firstMedia and laterMedia(i) on an absolute timeline. The difference is then calculated as the relative deviation of the timeline, that is, the relative delay. This makes it possible to determine the relative delay between media streams based on the time information in the sender report.
[0050] Step 203: Update the timestamp modification value of each received media stream according to the relative delay.
[0051] In this embodiment, the current timestamp modification value of each received media stream can be updated based on the relative delay determined in step 202. For example, the timestamp modification value of the received target media stream can be updated based on the relative delay, thereby eliminating the relative delay between the target media stream and the first media stream and achieving synchronization. For another example, the timestamp modification value of each media stream received before the target media stream can be updated based on the relative delay, thereby eliminating the relative delay between other media streams, including the first media stream, and the target media stream and achieving synchronization. Although the above two examples achieve synchronization, they do not consider the issue of timestamp rollback.
[0052] In practice, the timestamps of data packets in a media stream should be monotonically increasing as the packet sequence numbers increase, and timestamp rollback should not occur. For example, suppose a media stream includes 1000 data packets with sequence numbers 1-1000. In theory, as the sequence numbers increase monotonically, the timestamps of the data packets should be monotonically increasing. Otherwise, timestamp rollback will occur. For example, if the timestamp of the packet with sequence number 102 is smaller than the timestamp of the packet with sequence number 101, it means that timestamp rollback has occurred. If timestamp rollback occurs, it may affect the subsequent decoding, rendering, and playback of the media data. Therefore, when synchronizing media streams, timestamp rollback should be avoided.
[0053] In order to avoid the problem of timestamp rollback, the above step 203 can be implemented as follows:
[0054] S1. Determine the relative size relationship between the delay of the target media stream and the delay of the first media stream according to the relative delay.
[0055] In this implementation, it is possible to determine, based on the relative delay, whether the target media stream or the first media stream has a larger delay or a smaller delay.
[0056] S2: In response to determining that the delay of the target media stream is smaller than the delay of the first media stream, adding a modification amount corresponding to the relative delay to the current timestamp modification amount of each received media stream except the target media stream.
[0057] In this implementation, if the delay of the target media stream is smaller than the delay of the first media stream, it means that when the data packets are sent at the same time, the data packets of the target media stream arrive earlier. As a result, the timestamp of the data packets of the target media stream received at a certain moment may be too large. Therefore, it is necessary to increase the delay of the target media stream, that is, it is necessary to reduce the timestamp of the data packets of the target media stream, but this may cause the problem of timestamp rollback. In order to avoid the problem of timestamp rollback, the delay of each received media stream other than the target media stream can be reduced, that is, the modification amount corresponding to the relative delay can be added to the current timestamp modification amount of each received media stream other than the target media stream. For example, assuming the relative delay is 0.01ms (milliseconds), 0.01ms can be used as the modification amount corresponding to the relative delay.
[0058] S3: In response to determining that the delay of the target media stream is greater than the delay of the first media stream, adding a modification amount corresponding to the relative delay to the current timestamp modification amount of the target media stream.
[0059] In this implementation, if the delay of the target media stream is greater than the delay of the first media stream, this means that, when sending data packets at the same time, the data packets of the target media stream arrive later. Consequently, the timestamps of the data packets of the target media stream received at a given moment may be too small. Therefore, it is necessary to reduce the delay of the target media stream, that is, to increase the timestamps of the data packets of the target media stream. However, increasing the timestamps of data packets does not cause timestamp rollback. Therefore, when the delay of the target media stream is greater than the delay of the first media stream, the current timestamp of the target media stream can be modified by adding a modifier corresponding to the relative delay to achieve time synchronization.
[0060] Through this implementation method, time synchronization is achieved based on the relative size relationship between the delay of the target media stream and the delay of the first media stream, and the timestamp modification amounts of those media streams are selected to be updated according to the relative size relationship of the delays. This is more in line with the actual situation and will not cause the DTS of the media data packet to fall back, thereby making the synchronization process more reasonable and the method has a wider range of applications.
[0061] Continuing with Figures 3(a)-3(e), Figures 3(a)-3(e) illustrate an example diagram for synchronizing five media streams. In the example shown in Figures 3(a)-3(e), in the order in which the media streams are received, the five media streams are: first media stream (firstMedia), subsequent media stream (1) (laterMedia1), subsequent media stream (2) (laterMedia2), subsequent media stream (3) (laterMedia3), and subsequent media stream (4) (laterMedia4).
[0062] Among them, Figure 3(a) shows a schematic diagram of the time axis distribution of 5 media streams in an unaligned state. In Figure 3(a), the current timestamp modification amount ct of each media stream is 0. That is, ct(0) corresponding to the first media stream = 0, ct(1) corresponding to the subsequent media stream (1) = 0, ct(2) corresponding to the subsequent media stream (2) = 0, ct(3) corresponding to the subsequent media stream (3) = 0, and ct(4) corresponding to the subsequent media stream (4) = 0. In Figure 3(a), the vertical axis represents a time axis. If two media streams are on the same horizontal line, it is considered that the two media streams are synchronized in time. The 5 media streams in Figure 3(a) are not on the same horizontal line, so they need to be synchronized. If each laterMedia is moved in the direction of the arrow in the figure, synchronization can be achieved. However, the media stream cannot be moved according to the upward arrow, because moving the media stream upward may cause the timestamp to be non-incremental, which may cause the timestamp to be rolled back. Therefore, other media streams need to be shifted downward.
[0063] Based on the above description, when aligning the subsequent media stream (1) with the first media stream, the subsequent media stream (1) cannot be moved upward, so the first media stream needs to be moved downward by t1. If only the first media stream is moved, the relative delay between other laterMedia and the first media stream will be affected. Therefore, the subsequent media stream (2), subsequent media stream (3) and subsequent media stream (4) also need to be moved accordingly with the first media stream and also moved downward by t1. The effect after the movement is shown in Figure 3(b). Figure 3(b) shows a schematic diagram of the effect after the first media stream, subsequent media stream (2), subsequent media stream (3) and subsequent media stream (4) are moved downward by t1. At this time, ct(0) = t1, ct(1) = 0, ct(2) = t1, ct(3) = t1, ct(4) = t1.
[0064] For the subsequent media stream (2), during synchronization, since the alignment of the subsequent media stream (2) will not cause the timestamp to roll back, the subsequent media stream (2) can be directly moved downward by t0 for alignment without affecting other media streams. Figure 3(c) shows a schematic diagram of the effect after the subsequent media stream (2) is moved downward by t0. At this time, ct(0) = t1, ct(1) = 0, ct(2) = t1 + t0, ct(3) = t1, and ct(4) = t1.
[0065] For the subsequent media stream (3), during synchronization, since the subsequent media stream (3) cannot be moved upward, the first media stream, the subsequent media stream (1), the subsequent media stream (2), and the subsequent media stream (4) need to be moved downward by t1. Figure 3(d) shows the effect of moving the first media stream, the subsequent media stream (1), the subsequent media stream (2), and the subsequent media stream (4) downward by t1. At this time, ct(0) = 2t1, ct(1) = t1, ct(2) = 2t1 + t0, ct(3) = t1, and ct(4) = 2t1.
[0066] For the subsequent media stream (4), during synchronization, since the subsequent media stream (4) cannot be moved upward, the first media stream, the subsequent media stream (1), the subsequent media stream (2), and the subsequent media stream (3) need to be moved downward by t2. Figure 3(e) shows a schematic diagram of the effect after the first media stream, the subsequent media stream (1), the subsequent media stream (2), and the subsequent media stream (3) are moved downward by t2. At this time, ct(0) = 2t1 + t2, ct(1) = t1 + t2, ct(2) = 2t1 + t0 + t2, ct(3) = t1 + t2, and ct(4) = 2t1.
[0067] The translation process shown in Figures 3(a) to 3(e) above can complete the time synchronization of the five media streams. Afterwards, if a new media stream is received, the new media stream can first be allowed to inherit the current value of ct(0) of the first media stream. Then, synchronization is performed again. The specific synchronization method can refer to the synchronization of subsequent media streams (1), subsequent media streams (2), subsequent media streams (3) and subsequent media streams (4), which will not be repeated here. Therefore, the method of the present application can achieve synchronization of new media streams received at any time.
[0068] It is understandable that for some applications such as live broadcast and video conferencing in unreliable transmission scenarios, such as UDP (User Datagram Protocol) / RTP (Real-time Transport Protocol) transmission based on WebRTC (Web Real-Time Communications), due to the unreliability of network transmission, the data packets used to calculate the relative delay may be received relatively late or even lost, thereby affecting the synchronization time of the media stream. However, the method of the present application does not change the original relative delay with other media streams before synchronization.
[0069] Step 204: Modify the timestamps of the media data packets of each media stream according to the timestamp modification amount of each media stream.
[0070] In this embodiment, the timestamps of packets in each media stream can be modified based on the timestamp modification amount of each media stream, thereby achieving time synchronization. As an example, the timestamps of all packets in a media stream can be modified. In this example, all received media packets can be buffered until synchronization data is received. After synchronization is performed, the buffered media packets are read out and the timestamps modified based on the synchronization results to achieve synchronization. However, this approach increases latency, which should be minimized for scenarios such as live broadcasts and video conferencing. Furthermore, in unreliable transmission scenarios, there is no guarantee of when synchronization data will be received, so the duration of the buffering is uncontrollable. As another example, only the timestamps of packets after synchronization can be modified, while the timestamps of packets before synchronization remain unchanged. In this example, packets before synchronization are not buffered, and only the timestamp of the target media stream is modified based on the initial timestamp modification amount of the target media stream, thereby maintaining the original relative delay between the target media stream and the first media stream. After synchronization, the timestamps of the synchronized packets are modified. This approach avoids buffering and ensures time synchronization of the synchronized media streams.
[0071] Reviewing the above process, in the above embodiment of the present specification, when a media data packet of a target media stream is received, the initial timestamp modification amount of the media data packet of the target media stream can be determined based on the current timestamp modification amount of the first media stream. Then, the relative delay of the target media stream relative to the first media stream is determined, and the timestamp modification amount of each received media stream is updated based on the relative delay. Finally, the timestamp of the data packet of each media stream is modified based on the timestamp modification amount of each media stream. Thus, when the target media stream is received, the timestamp of the data packet of each media stream can be modified based on the relative delay of the target media stream relative to the first media stream, thereby achieving real-time synchronization of multiple media streams (including new media streams received at any time).
[0072] According to another embodiment, a device for synchronizing multiple media streams is provided. The device for synchronizing multiple media streams can be deployed on any device, equipment, platform, or device cluster with computing and processing capabilities.
[0073] FIG4 shows a schematic block diagram of an apparatus for synchronizing multiple media streams according to an embodiment. The apparatus shown in FIG4 is used to execute the method shown in FIG2 . As shown in FIG4 , the apparatus 400 for synchronizing multiple media streams includes: a first determining unit 401 configured to, in response to receiving a media data packet of a target media stream, determine an initial timestamp modification amount of the media data packet of the target media stream according to the current timestamp modification amount of the received first media stream, wherein the target media stream is a media stream received after the first media stream; a second determining unit 402 configured to determine a relative delay of the target media stream relative to the first media stream; an updating unit 403 configured to update the timestamp modification amount of each received media stream according to the relative delay; and a modifying unit 404 configured to modify the timestamp of the data packet of each media stream according to the timestamp modification amount of each media stream.
[0074] In some optional implementations of this embodiment, the updating unit 403 is further configured to: determine, based on the relative delay, a relative size relationship between the delay of the target media stream and the delay of the first media stream; and in response to determining that the delay of the target media stream is smaller than the delay of the first media stream, add a modification amount corresponding to the relative delay to the current timestamp modification amount of each received media stream other than the target media stream.
[0075] In some optional implementations of this embodiment, the updating unit 403 is further configured to: in response to determining that the delay of the above-mentioned target media stream is greater than the delay of the above-mentioned first media stream, add a modification amount corresponding to the above-mentioned relative delay to the current timestamp modification amount of the above-mentioned target media stream.
[0076] In some optional implementations of this embodiment, the device 400 also includes: a receiving unit (not shown in the figure), configured to receive the synchronization information of the above-mentioned target media stream; and the second determination unit 402 is further configured to: calculate the relative delay of the above-mentioned target media stream relative to the above-mentioned first media stream based on the synchronization information of the above-mentioned target media stream and the synchronization information of the pre-received first media stream, wherein the synchronization information includes the time information in the sender report.
[0077] In some optional implementations of this embodiment, the apparatus 400 further includes: a receiving and storing unit (not shown in the figure), configured to receive and store the synchronization information of the first media stream.
[0078] In some optional implementations of this embodiment, the apparatus 400 further includes: a setting unit (not shown in the figure), configured to receive a data packet of a first media stream, and set an initial timestamp modification amount of the first media stream to 0.
[0079] In some optional implementations of this embodiment, before receiving the media data packet of the target media stream, media data packets of one or more media streams have been received.
[0080] The above-mentioned device embodiments correspond to the method embodiments. For detailed descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For detailed descriptions, please refer to the corresponding method embodiments.
[0081] According to another embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in FIG. 2 .
[0082] According to yet another embodiment, an electronic device is provided, including a memory and a processor, wherein executable code is stored in the memory, and when the processor executes the executable code, the method described in FIG. 2 is implemented.
[0083] The foregoing description describes specific embodiments of the present disclosure, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than that described in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily need to be performed in the specific order shown or in a sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0084] 5, which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiments of the present application. The electronic device shown in FIG5 is only an example and should not limit the functions and scope of use of the embodiments of the present application.
[0085] As shown in Figure 5, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0086] Typically, the following devices can be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 can allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although FIG5 shows an electronic device 500 with various devices, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively. Each box shown in FIG5 may represent one device, or may represent multiple devices as needed.
[0087] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present application are performed.
[0088] The embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute the method provided in this specification.
[0089] It should be noted that the computer-readable medium described in the embodiments of this specification may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the embodiments of this specification, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the embodiments of this specification, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or convey a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on the computer-readable medium may be conveyed using any suitable medium, including but not limited to wires, optical cables, RF (Radio Frequency), or any suitable combination thereof.
[0090] The computer-readable medium may be included in the electronic device, or may exist independently and not be incorporated into the electronic device. The computer-readable medium carries one or more programs. When executed by the electronic device, the electronic device: in response to receiving a media data packet of a target media stream, determines an initial timestamp modification amount for the media data packet of the target media stream based on the current timestamp modification amount of the received first media stream; determines a relative delay of the target media stream relative to the first media stream; updates the timestamp modification amount of each received media stream based on the relative delay; and modifies the timestamp of the data packet of each media stream based on the timestamp modification amount of each media stream.
[0091] Computer program code for performing the operations of the embodiments of the present specification can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or electronic device. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0092] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences between the other embodiments. In particular, the storage medium and computing device embodiments are described briefly because they are generally similar to the method embodiments. For relevant portions, refer to the description of the method embodiments.
[0093] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described in the embodiments of the present disclosure may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.
[0094] The specific implementation methods described above further illustrate the purpose, technical solutions, and beneficial effects of the embodiments of the present disclosure. It should be understood that the above description is only a specific implementation method of the embodiments of the present disclosure and is not intended to limit the scope of protection of the present disclosure. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present disclosure shall be included in the scope of protection of the present disclosure.
Claims
1. A method for synchronizing multiple media streams, the method comprising: In response to receiving a media data packet of a target media stream, determining an initial timestamp modification amount for the media data packet of the target media stream according to a current timestamp modification amount of the received first media stream, wherein the target media stream is a media stream received after the first media stream; determining a relative delay of the target media stream relative to the first media stream; updating the timestamp modification amount of each received media stream according to the relative delay; Modify the timestamp of the data packet of each media stream according to the timestamp modification amount of each media stream.
2. The method according to claim 1, wherein The updating of the timestamp modification amount of each received media stream according to the relative delay includes: Determining, based on the relative delay, a relative magnitude relationship between the delay of the target media stream and the delay of the first media stream; In response to determining that the delay of the target media stream is smaller than the delay of the first media stream, a modification amount corresponding to the relative delay is added to the current timestamp modification amount of each received media stream except the target media stream.
3. The method according to claim 2, wherein: The updating of the timestamp modification amount of each received media stream according to the relative delay further includes: In response to determining that the delay of the target media stream is greater than the delay of the first media stream, a modification amount corresponding to the relative delay is added to the current timestamp modification amount of the target media stream.
4. The method according to any one of claims 1 to 3, wherein: The method further comprises: receiving synchronization information of the target media stream; and Determining the relative delay of the target media stream relative to the first media stream includes: The relative delay of the target media stream relative to the first media stream is calculated according to the synchronization information of the target media stream and the synchronization information of the first media stream received in advance, wherein the synchronization information includes the time information in the sender report.
5. The method according to claim 4, wherein Before determining the relative delay of the target media stream relative to the first media stream, the method further includes: Receive and store synchronization information of the first media stream.
6. The method according to any one of claims 1 to 5, wherein: Before receiving the media data packet of the target media stream, the method further includes: A data packet of a first media stream is received, and an initial timestamp modification amount of the first media stream is set to 0.
7. The method according to any one of claims 1 to 5, wherein: Before receiving the media data packet of the target media stream, media data packets of one or more media streams have been received.
8. A device for synchronizing multiple media streams, the device comprising: a first determining unit configured to, in response to receiving a media data packet of a target media stream, determine an initial timestamp modification amount of the media data packet of the target media stream according to a current timestamp modification amount of the received first media stream, wherein the target media stream is a media stream received after the first media stream; a second determining unit configured to determine a relative delay of the target media stream relative to the first media stream; an updating unit configured to update a timestamp modification amount of each received media stream according to the relative delay; The modification unit is configured to modify the timestamp of the data packet of each media stream according to the timestamp modification amount of each media stream.
9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.
10. An electronic device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 7 is implemented.
11. A computer program product comprising a computer program, which, when executed in a computer, causes the computer to perform the method according to any one of claims 1 to 7.
12. A computer program, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Media stream switching synchronization method and streaming media server
CN102065060A
Media stream recording method and device, computer storage medium and electronic equipment
CN113630643A
Streaming media data timestamp updating method and device
CN114979681A
Method and device for synchronizing multiple paths of media streams, medium and equipment
CN116599944A
Checksum friendly timestamp update
US20170111483A1
