Method, apparatus, medium and device for multi-path media stream synchronization

Abstract

Embodiments of the present specification provide a method, device, medium and equipment for synchronizing multiple media streams. A specific embodiment of the method comprises: in response to receiving a media data packet of a target media stream, determining an initial timestamp modification amount of the media data packet of the target media stream according to a current timestamp modification amount of a first media stream, wherein the target media stream is a media stream received after the first media stream; determining a relative delay of the target media stream relative to the first media stream; updating the timestamp modification amount of each received media stream according to the relative delay; and modifying the timestamp of the data packet of each media stream according to the timestamp modification amount of each media stream, thereby synchronizing the multiple media streams in real time.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the field of information technology, and in particular, to a method, apparatus, medium and device for multi-path media stream synchronization. BACKGROUND

[0002] With the rapid development of computer technology and Internet technology, application programs involving multimedia data are becoming more and more popular and are deeply loved by users. When there are multiple media streams, it is necessary to synchronize the multiple media streams. For example, in live broadcast, video conference and other scenarios, real-time time synchronization of multimedia data is very necessary, which relates to whether the media streams are synchronized when played. SUMMARY

[0003] Embodiments of the present specification describe a method, apparatus, medium and device for multi-path media stream synchronization, which can modify the timestamps of the data packets of each media stream based on the relative delay of the target media stream relative to the first media stream when the target media stream is received, thereby realizing real-time synchronization of multiple media streams.

[0004] According to a first aspect, a method for multi-path media stream synchronization is provided, the method comprising: in response to receiving a media data packet of a target media stream, determining an initial timestamp modification amount of the media data packet of the target media stream according to a current timestamp modification amount of a first media stream; determining a relative delay of the target media stream relative to the first media stream; updating the timestamp modification amounts of the received media streams according to the relative delay; and modifying the timestamps of the data packets of the media streams according to the timestamp modification amounts of the media streams.

[0005] In one embodiment, the updating of the timestamp modification amounts of the received media streams according to the relative delay comprises: determining the relative size relationship between the delay of the target media stream and the delay of the first media stream according to the relative delay; and in response to determining that the delay of the target media stream is smaller than the delay of the first media stream, increasing the current timestamp modification amount of each received media stream except the target media stream by a modification amount corresponding to the relative delay. Thus, media stream synchronization is realized when the delay of the target media stream is smaller than the delay of the first media stream.

[0006] In one embodiment, the updating of the timestamp modification amounts of the received media streams according to the relative delay further comprises: in response to determining that the delay of the target media stream is greater than the delay of the first media stream, increasing the current timestamp modification amount of the target media stream by a modification amount corresponding to the relative delay. Thus, media stream synchronization is realized when the delay of the target media stream is greater than the delay of the first media stream.

[0007] In an embodiment, the method further comprises: receiving synchronization information of the target media stream; and the determining the relative delay of the target media stream relative to the first media stream comprises: calculating the relative delay of the target media stream relative to the first media stream according to the synchronization information of the target media stream and the previously received synchronization information of the first media stream, wherein the synchronization information comprises time information in a sender report. Thus, the relative delay between media streams is determined based on the time information in the sender report.

[0008] In an embodiment, before the determining the relative delay of the target media stream relative to the first media stream, the method further comprises: receiving and storing the synchronization information of the first media stream. Thus, the receiving and storing of the synchronization information of the first media stream is achieved.

[0009] In an embodiment, before the receiving the media data packet of the target media stream, the method further comprises: receiving a data packet of the first media stream, and setting an initial timestamp modification amount of the first media stream as 0. Thus, the setting of the initial timestamp modification amount of the first media stream is achieved.

[0010] In an embodiment, before the receiving the media data packet of the target media stream, media data packets of one or more media streams have been received. Thus, the synchronization of two or more media streams including the target media stream is achieved.

[0011] According to a second aspect, a device for synchronizing multiple media streams is provided, the device comprising: a first determining unit configured to, in response to receiving a media data packet of a target media stream, determine an initial timestamp modification amount of the media data packet of the target media stream according to a current timestamp modification amount of a first media stream received; a second determining unit configured to determine a relative delay of the target media stream relative to the first media stream; an updating unit configured to update timestamp modification amounts of the received media streams according to the relative delay; and a modifying unit configured to modify timestamps of data packets of the media streams according to the timestamp modification amounts of the media streams.

[0012] According to a third aspect, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method according to any one of the first aspect.

[0013] According to a fourth aspect, a computer-readable storage medium is provided, having stored thereon a computer program which, when executed by a computer, causes the computer to perform the method according to any one of the first aspect.

[0014] According to a fifth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method according to any one of the first aspect.

[0015] According to the method and device for synchronizing multiple media streams provided in the embodiments of the present application, when the media data packet of the target media stream is received, the initial timestamp modification amount of the media data packet of the target media stream can be determined according to the current timestamp modification amount of the first media stream. Then, the relative delay of the target media stream relative to the first media stream is determined, and the timestamp modification amount of each received media stream is updated according to the relative delay. Finally, the timestamps of the data packets of each media stream are modified according to the timestamp modification amount of each media stream. In this way, when the target media stream is received, the timestamps of the data packets of each media stream can be modified based on the relative delay of the target media stream relative to the first media stream, so that the multiple media streams (including a new media stream received at any time) can be synchronized in real time. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A schematic diagram showing an application scenario to which the embodiments of the present application can be applied is shown;

[0017] Figure 2 A flowchart showing a method for synchronizing multiple media streams according to an embodiment is shown;

[0018] FIG. 3(a) shows a schematic diagram of the time axis distribution of five media streams in a misaligned state;

[0019] FIG. 3(b) shows a schematic diagram of the effect of moving the first media stream, the subsequent media stream (2), the subsequent media stream (3) and the subsequent media stream (4) downward by t1;

[0020] FIG. 3(c) shows a schematic diagram of the effect of moving the subsequent media stream (2) downward by t0;

[0021] FIG. 3(d) shows a schematic diagram of the effect of moving the first media stream, the subsequent media stream (1), the subsequent media stream (2) and the subsequent media stream (4) downward by t1;

[0022] FIG. 3(e) shows a schematic diagram of the effect of moving the first media stream, the subsequent media stream (1), the subsequent media stream (2) and the subsequent media stream (3) downward by t2;

[0023] Figure 4 A schematic block diagram of a device for synchronizing multiple media streams according to an embodiment is shown;

[0024] Figure 5 A schematic diagram of an electronic device suitable for implementing the embodiments of the present application is shown. DETAILED DESCRIPTION

[0025] The technical solutions provided in the specification are described in further detail below in combination with the accompanying drawings and examples. It can be understood that the specific examples described herein are merely intended to explain the related application, and not to limit the application. In addition, it should be noted that only the parts related to the application are shown in the accompanying drawings for ease of description. It should be noted that the embodiments of the specification and the features in the examples can be combined with each other without conflict.

[0026] As mentioned above, it is necessary to time synchronize the multimedia data, which relates to whether the media stream is synchronized when playing. To this end, the embodiments of the specification provide a method for synchronizing multiple media streams, thereby realizing synchronization of multiple media streams. As shown in Figure 1 Figure 1 A schematic diagram of an application scenario to which the embodiments of the specification can be applied is shown. In Figure 1 ​In the shown application scenario, the device 101 can receive multiple media streams, and each time a new media stream is received, after receiving the synchronization-related data of the media stream, the device 101 can perform a synchronization based on the synchronization-related data of the media stream. During the synchronization, the timestamps of the data packets of the received media streams can be modified. Here, the timestamps of the data packets to be modified can include DTS (Decoding Time Stamp) and PTS (Presentation Time Stamp). In this example, the first received media stream is denoted as firstMedia, and the current timestamp modification amount of the first media stream is denoted as ct(0). The media streams received after the first media stream are denoted as laterMedia(1), laterMedia(2), and so on. Here, the current timestamp modification amount of the first media stream can refer to the change amount of the timestamp of the first media stream at the current time compared with the original timestamp. When the media data packet of a target media stream laterMedia(i) (where i≥1, and i is a positive integer) is received, the initial timestamp modification amount of the media data packet of the target media stream can be determined according to the current timestamp modification amount ct(0) of the first media stream. That is, the timestamp modification amount ct(i) of the target media stream is initialized, for example, when initialized, ct(i) can be set as ct(i) = ct(0). At this time, the timestamp of the media data packet of the target media stream can be modified according to ct(i), so that the target media stream inherits the current timestamp modification amount of the first media stream, without changing the original relative delay between the target media stream and the first media stream. Then, the synchronization-related data of the target media stream can be received, and the relative delay relative_relay(i) of the target media stream relative to the first media stream can be determined according to the synchronization-related data. Then, the timestamp modification amounts of the received media streams are updated according to relative_relay(i). Finally, the timestamps of the data packets of the media streams are modified according to the timestamp modification amounts of the media streams. In this way, the synchronization of the multiple media streams can be achieved.

[0027] Continuing to refer to Figure 2 , Figure 2 A flowchart of a method for synchronizing multiple media streams according to an embodiment is shown. The method can be performed by any device, apparatus, platform, cluster of devices having computing and processing capabilities. For example, the method can be performed by a terminal device. For another example, the method can be performed by other devices, apparatuses, etc., and then the synchronized multiple media streams can be sent to a terminal device. As shown in Figure 2 the method for synchronizing multiple media streams can include the following steps:

[0028] At step 201, in response to receiving the media data packet of the target media stream, an initial timestamp modification amount of the media data packet of the target media stream is determined according to the current timestamp modification amount of the received first media stream.

[0029] In the embodiment, the media stream can include a video stream, an audio stream, a subtitle stream, etc. The media data packet can refer to a data packet containing media data. Taking the video stream as an example, the media data packet can refer to a data packet containing a video frame. When the media data packet of the target media stream is received, the initial timestamp modification amount of the media data packet of the target media stream can be determined according to the current timestamp modification amount of the received first media stream. For example, the current timestamp modification amount of the first media stream can be taken as the initial timestamp modification amount of the target media stream. Then, the timestamp of the media data packet of the target media stream can be modified according to the initial timestamp modification amount. In this way, the target media stream can inherit the current timestamp modification amount of the first media stream, so that the relative delay between the target media stream and the first media stream remains unchanged. In the example, the target media stream is a media stream received after the first media stream.

[0030] It can be understood that before the media data packet of the target media stream is received, the media data packets of one or more media streams have been received. That is, the target media stream is the Nth media stream received, where N≥2 and N is a positive integer. In this way, the synchronization of two or more media streams including the target media stream can be achieved.

[0031] In some implementations, before the media data packet of the target media stream is received, the method for synchronizing the multiple media streams can further include receiving the data packet of the first media stream and setting the initial timestamp modification amount of the first media stream to 0. In this way, the setting of the initial timestamp modification amount of the first media stream can be achieved.

[0032] At step 202, the relative delay of the target media stream relative to the first media stream is determined.

[0033] In the embodiment, the relative delay of the target media stream relative to the first media stream can be determined in various ways.

[0034] As an example, the delay information of the media stream can be carried in the RTCP (Real-time Transport Control Protocol) message of the media stream, for example, the delay information can include a delay value delay. In this way, the relative delay of the target media stream relative to the first media stream can be calculated according to the delay information of the target media stream and the delay information of the first media stream. Taking the target media stream as laterMedia(i) and the first media stream as firstMedia as an example, where i≥1 and i is a positive integer, the relative delay relative_relay(i) of laterMedia(i) relative to firstMedia can be determined in the following way:

[0035] relative_delay(i) = FirstMedia.delay - laterMedia(i).delay,

[0036] where FirstMedia.delay can represent the delay value of firstMedia, and laterMedia(i).delay can represent the delay value of laterMedia(i).

[0037] As another example, the relative delay of the media stream relative to the first media stream can also be directly carried in the RTCP message of the media stream. In this way, the relative delay of the target media stream relative to the first media stream can be directly determined according to the RTCP message.

[0038] It can be understood that the RTCP message in the above two examples can be received after receiving part of the media data packets of the target media stream. Therefore, it can not be possible to synchronize when the first media data packet of the target media stream is received, and it can be possible to synchronize after receiving part of the media data packets.

[0039] In some implementations, the method for synchronizing multiple media streams described above can further include Figure 2 the following step not shown in the above method: receiving synchronization information of the target media stream.

[0040] In this implementation, the synchronization information can include ntp (Network Time Protocol) and rtp (Real-time Transport Protocol) time information carried in the RTCP SR (Sender Report).

[0041] According to the synchronization information of the target media stream and the pre-received synchronization information of the first media stream, the relative delay of the target media stream relative to the first media stream is calculated.

[0042] It can be understood that the synchronization information of the first media stream can be received and stored before. That is, the target media stream is the second or nth (n>2, and n is a positive integer) media stream received.

[0043] Taking the first media stream as firstMedia and the target media stream as laterMedia(i) as an example, the relative delay relative_relay(i) of laterMedia(i) relative to firstMedia can be determined as follows:

[0044] relative_delay(i) = (FirstMedia.RtpPts - laterMedia(i).RtpPts) - (FirstMedia.NtpTime - laterMedia(i).NtpTime),

[0045] wherein RtpPts = rtpTimestamp / clockRate, rtpTimestamp can represent rtp timestamp, clockRate can represent clock frequency, and NtpTime can represent ntp timestamp. FirstMedia.RtpPts can represent RtpPts of firstMedia. laterMedia(i).RtpPts can represent RtpPts of laterMedia(i). FirstMedia.NtpTime can represent NtpTime of firstMedia. laterMedia(i).NtpTime can represent NtpTime of laterMedia(i).

[0046] In practice, the synchronization information contained in the SR of RTCP, i.e., the <rtp, ntp> timestamp pair, can locate firstMedia and laterMedia(i) to an absolute time axis, and the difference value is the relative deviation of the time axis, i.e., the relative delay. Thus, the relative delay between media streams based on the time information in the sender report can be determined.

[0047] In step 203, the time stamp modification amount of each received media stream is updated according to the relative delay.

[0048] In the embodiment, the current timestamp modification amount of each received media stream can be updated according to the relative delay determined in step 202. For example, the timestamp modification amount of the received target media stream can be updated according to the relative delay, so as to eliminate the relative delay between the target media stream and the first media stream, and realize synchronization. For another example, the timestamp modification amount of each received media stream before the target media stream can be updated according to the relative delay, so as to eliminate the relative delay between the target media stream and other media streams including the first media stream, and realize synchronization. Although the above two examples realize synchronization, the above two examples do not consider the timestamp rollback problem.

[0049] In practice, the timestamp of a data packet in a media stream should be monotonically non-decreasing according to the increasing of the sequence number of the data packet, and should not appear timestamp rollback. For example, it is assumed that a media stream includes 1000 data packets with sequence numbers 1-1000. In theory, the timestamp of the data packet should be monotonically non-decreasing according to the monotonically increasing of the sequence number. Otherwise, timestamp rollback appears. For example, if the timestamp of the data packet with sequence number 102 is smaller than the timestamp of the data packet with sequence number 101, it means that timestamp rollback appears. If timestamp rollback appears, it may affect the subsequent decoding, rendering, playing and the like of the media data. Therefore, when synchronizing the media streams, the problem of timestamp rollback should be avoided.

[0050] In order to avoid the problem of timestamp rollback, the step 203 can be implemented as follows:

[0051] S1, determining the relative size relationship between the delay of the target media stream and the delay of the first media stream according to the relative delay.

[0052] In the implementation, it can be determined according to the relative delay that who has larger delay and who has smaller delay between the target media stream and the first media stream.

[0053] S2, in response to determining that the delay of the target media stream is smaller than the delay of the first media stream, increasing the modification amount corresponding to the relative delay on the current timestamp modification amount of each received media stream except the target media stream.

[0054] In this implementation, if the delay of the target media stream is smaller than that of the first media stream, it means that the data packets of the target media stream arrived earlier when data packets were sent at the same time. Therefore, the timestamp of the data packets of the target media stream received at a certain moment may be too large. Thus, it is necessary to increase the delay of the target media stream, i.e., to decrease the timestamp of the data packets of the target media stream. However, this may cause timestamp rollback. To avoid timestamp rollback, the delays of all received media streams other than the target media stream can be reduced. That is, the modification amount corresponding to the relative delay can be added to the current timestamp modification amount of all received media streams other than the target media stream. For example, assuming the relative delay is 0.01ms, then 0.01ms can be used as the modification amount corresponding to the relative delay.

[0055] S3, in response to determining that the delay of the target media stream is greater than the delay of the first media stream, adds the modification amount corresponding to the relative delay to the current timestamp modification amount of the target media stream.

[0056] In this implementation, if the delay of the target media stream is greater than that of the first media stream, it means that, given the same data packets being sent at the same time, the data packets of the target media stream arrive later. Consequently, the timestamp of a data packet received at a certain moment from the target media stream may be too low. Therefore, it is necessary to reduce the delay of the target media stream, i.e., to increase the timestamp of the data packets in the target media stream. Increasing the timestamp of the data packets does not cause timestamp rollback. Therefore, when the delay of the target media stream is greater than that of the first media stream, the modification amount corresponding to the delay can be directly added to the current timestamp modification amount of the target media stream to achieve time synchronization.

[0057] This implementation achieves time synchronization based on the relative magnitude of the delay of the target media stream and the delay of the first media stream. Furthermore, it selects which media streams' timestamp modifications to update based on the relative magnitude of the delays, which is more in line with reality and will not cause the media data packet's DTS rollback. This makes the synchronization process more reasonable and the method has a wider range of applications.

[0058] See also Figures 3(a)-3(e) , Figures 3(a)-3(e) An example diagram illustrating the synchronization of five media streams is shown. Figures 3(a)-3(e) In the example shown, the media streams are received in the following order: firstMedia, laterMedia1, laterMedia2, laterMedia3, and laterMedia4, for a total of 5 media streams.

[0059] In FIG. 3(a), the vertical coordinate represents a time axis. If two media streams are on the same horizontal line, it is considered that the two media streams are synchronized in time. In FIG. 3(a), the five media streams are not on the same horizontal line, and thus need to be synchronized. If each laterMedia is moved in the direction of the arrow according to the arrow in the figure, synchronization can be achieved. However, the media stream cannot be moved upward, because moving the media stream upward can cause the time stamp to be non-increasing, and can cause the time stamp to be rolled back. Therefore, the other media streams need to be moved downward.

[0060] Based on the above description, when aligning the subsequent media stream (1) with the first media stream, the subsequent media stream (1) cannot be moved upward, and thus the first media stream needs to be moved downward by t1. If only the first media stream is moved, the relative delay of the other laterMedia with respect to the first media stream will be affected, and thus the subsequent media stream (2), the subsequent media stream (3), and the subsequent media stream (4) also need to be moved downward by t1 with respect to the first media stream. The effect after the movement is shown in FIG. 3(b), which shows a schematic diagram of the effect after the first media stream, the subsequent media stream (2), the subsequent media stream (3), and the subsequent media stream (4) are moved downward by t1. At this time, ct(0)=t1, ct(1)=0, ct(2)=t1, ct(3)=t1, and ct(4)=t1.

[0061] For the subsequent media stream (2), when synchronized, because the alignment of the subsequent media stream (2) does not cause the time stamp to be rolled back, the subsequent media stream (2) can be directly moved downward by t0 for alignment, without affecting other media streams. FIG. 3(c) shows a schematic diagram of the effect after the subsequent media stream (2) is moved downward by t0. At this time, ct(0)=t1, ct(1)=0, ct(2)=t1+t0, ct(3)=t1, and ct(4)=t1.

[0062] For the subsequent media stream (3), when synchronizing, since the subsequent media stream (3) cannot be moved upwards, the first media stream, the subsequent media stream (1), the subsequent media stream (2) and the subsequent media stream (4) need to be moved downwards by t1. Fig. 3(d) shows a schematic diagram of the effect of moving the first media stream, the subsequent media stream (1), the subsequent media stream (2) and the subsequent media stream (4) downwards by t1. At this time, ct(0) = 2t1, ct(1) = t1, ct(2) = 2t1+ t0, ct(3) = t1, ct(4) = 2t1.

[0063] For the subsequent media stream (4), when synchronizing, since the subsequent media stream (4) cannot be moved upwards, the first media stream, the subsequent media stream (1), the subsequent media stream (2) and the subsequent media stream (3) need to be moved downwards by t2. Fig. 3(e) shows a schematic diagram of the effect of moving the first media stream, the subsequent media stream (1), the subsequent media stream (2) and the subsequent media stream (3) downwards by t2. At this time, ct(0) = 2t1+ t2, ct(1) = t1+ t2, ct(2) = 2t1+ t0+ t2, ct(3) = t1+ t2, ct(4) = 2t1.

[0064] After the translation process shown in Figs. 3(a)-(e), the time synchronization of the five media streams can be completed. After that, if a new media stream is received, the new media stream can first inherit the current value of ct(0) of the first media stream. Then, synchronization is performed again. The specific synchronization method can refer to the synchronization of the subsequent media stream (1), the subsequent media stream (2), the subsequent media stream (3) and the subsequent media stream (4), which will not be described here. Thus, the method of the present application can realize the synchronization of any new media stream received at any time.

[0065] It can be understood that for some live broadcast, video conference and other applications in a non-reliable transmission scenario, for example, webrtc (Web Real-Time Communications)-based udp (User Datagram Protocol) / rtp (Real-time Transport Protocol) transmission, due to the unreliability of network transmission, the data packet to which the data used to calculate the relative delay belongs to may be received relatively late or even lost, and thus the time of media stream synchronization may be affected. However, the method of the present application does not change the original relative delay with other media streams before synchronization.

[0066] Step 204, modifying the time stamp of each media data packet of each media stream according to the time stamp modification amount of each media stream.

[0067] In the embodiment, the timestamps of the data packets of the media streams can be modified according to the timestamp modification amounts of the media streams, so as to realize time synchronization. As an example, the timestamps of all the data packets of the media streams can be modified. In the example, all the received media data packets can be buffered first, and after the synchronization operation is performed after the relevant data for synchronization is received, the buffered media data packets are read out, the timestamps are modified according to the synchronization result, and the synchronization is performed. However, in this way, the buffering increases the delay, and for live broadcast, video conference and other scenarios, the delay should be reduced as much as possible. In addition, in the unreliable transmission scenario, it cannot be ensured when the relevant data for synchronization is obtained, so the duration of the buffering is uncontrollable. As another example, the timestamps of the data packets after the synchronization can also be modified, and the timestamps of the data packets before the synchronization are not modified. In the example, the data packets before the synchronization are not buffered, and only the initial timestamp modification amount of the target media stream is modified according to the initial timestamp modification amount of the target media stream, so as not to change the original relative delay between the target media stream and the first media stream. After the synchronization, the timestamps of the data packets after the synchronization are modified. In this way, the buffering can be avoided, and the time synchronization of the media streams after the synchronization can be ensured.

[0068] In the above process, in the above embodiment of the present specification, when the media data packet of the target media stream is received, the initial timestamp modification amount of the media data packet of the target media stream can be determined according to the current timestamp modification amount of the first media stream. Then, the relative delay of the target media stream relative to the first media stream is determined, and the timestamp modification amounts of the received media streams are updated according to the relative delay. Finally, the timestamps of the data packets of the media streams are modified according to the timestamp modification amounts of the media streams. Thus, when the target media stream is received, the timestamps of the data packets of the media streams can be modified based on the relative delay of the target media stream relative to the first media stream, so as to realize real-time synchronization of the multiple media streams (including a new media stream received at any time).

[0069] According to another aspect, an apparatus for multiple media stream synchronization is provided. The apparatus for multiple media stream synchronization can be deployed in any device, equipment, platform, cluster of equipment having computing and processing capabilities.

[0070] Figure 4 A schematic block diagram of an apparatus for multiple media stream synchronization according to an embodiment is shown. Figure 4 The apparatus is configured to perform the method. Figure 2 The method is shown. As Figure 4As shown, the device 400 for synchronizing multiple media streams comprises: a first determining unit 401 configured to, in response to receiving a media data packet of a target media stream, determine an initial timestamp modification amount of the media data packet of the target media stream according to a received current timestamp modification amount of a first media stream, wherein the target media stream is a media stream received after the first media stream; a second determining unit 402 configured to determine a relative delay of the target media stream relative to the first media stream; an updating unit 403 configured to update timestamp modification amounts of received media streams according to the relative delay; and a modifying unit 404 configured to modify timestamps of data packets of the media streams according to the timestamp modification amounts of the media streams.

[0071] In some optional implementations of the present embodiment, the updating unit 403 is further configured to: determine a relative size relationship between the delay of the target media stream and the delay of the first media stream according to the relative delay; and in response to determining that the delay of the target media stream is smaller than the delay of the first media stream, increase the current timestamp modification amount of each of the received media streams other than the target media stream by a modification amount corresponding to the relative delay.

[0072] In some optional implementations of the present embodiment, the updating unit 403 is further configured to: in response to determining that the delay of the target media stream is larger than the delay of the first media stream, increase the current timestamp modification amount of the target media stream by a modification amount corresponding to the relative delay.

[0073] In some optional implementations of the present embodiment, the device 400 further comprises: a receiving unit (not shown in the figure) configured to receive synchronization information of the target media stream; and the second determining unit 402 is further configured to calculate the relative delay of the target media stream relative to the first media stream according to the synchronization information of the target media stream and pre-received synchronization information of the first media stream, wherein the synchronization information comprises time information in a sender report.

[0074] In some optional implementations of the present embodiment, the device 400 further comprises: a receiving and storing unit (not shown in the figure) configured to receive and store the synchronization information of the first media stream.

[0075] In some optional implementations of the present embodiment, the device 400 further comprises: a setting unit (not shown in the figure) configured to receive a data packet of the first media stream and set the initial timestamp modification amount of the first media stream to 0.

[0076] In some optional implementations of the present embodiment, the media data packet of one or more media streams is received before the media data packet of the target media stream is received.

[0077] The device embodiments correspond to the method embodiments, and specific descriptions can be referred to the descriptions of the method embodiments, which will not be repeated here. The device embodiments are based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. Specific descriptions can be referred to the corresponding method embodiments.

[0078] According to another aspect of the embodiments, a computer readable storage medium is also provided, which stores a computer program. When the computer program is executed in a computer, the computer program causes the computer to execute the method described above. Figure 2 The method described above.

[0079] According to another aspect of the embodiments, an electronic device is also provided, which includes a memory and a processor. The memory stores executable code, and the processor executes the executable code to implement the method described above. Figure 2 The method described above.

[0080] The above describes specific embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from the order in the embodiments, and the desired results can still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be implemented in the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible, or can be advantageous.

[0081] Reference will now be made to Figure 5 , which shows a structural schematic diagram of an electronic device 500 suitable for implementing the embodiments of the present application. Figure 5 The electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0082] As shown in Figure 5 , the electronic device 500 can include a processing device (such as a central processor, a graphics processor, etc.) 501, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 502 or programs loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0083] Generally, the following devices can be connected to the I / O interface 505: input device 506, including, for example, a touch screen, touch pad, keyboard, mouse, etc.; output device 507, including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 508, including, for example, a magnetic tape, hard disk, etc.; and communication device 509. The communication device 509 can allow the electronic device 500 to communicate wirelessly or wired with other devices to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present. Figure 5 Each block shown in the flowcharts can represent a device or multiple devices as needed.

[0084] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the methods of embodiments of the present application are performed.

[0085] The embodiments of the present specification also provide a computer readable storage medium, having stored thereon a computer program, which, when executed in a computer, causes the computer to perform the method provided by the present specification.

[0086] It should be noted that the computer readable medium in the embodiments of the present specification can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present specification, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the embodiments of the present specification, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in connection with an instruction execution system, device or apparatus. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, an optical fiber, an RF (Radio Frequency) or the like, or any suitable combination of the above.

[0087] The computer readable medium described above can be included in the electronic device described above; or can exist separately and not be assembled into the electronic device. The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device is caused to: in response to receiving a media data packet of a target media stream, determine an initial timestamp modification amount of the media data packet of the target media stream according to a current timestamp modification amount of a first media stream; determine a relative delay of the target media stream relative to the first media stream; update the timestamp modification amount of each received media stream according to the relative delay; and modify the timestamp of the data packet of each media stream according to the timestamp modification amount of each media stream.

[0088] Computer program code for carrying out operations of embodiments of the present specification can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or electronic device. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0089] The various embodiments described in this specification are described in progressively greater detail. Each embodiment described in this specification builds upon previously described embodiments and focuses on differences between the previously described embodiments and the embodiment being discussed. In particular, the storage medium and computing device embodiments are described in less detail than the method embodiments because they are substantially similar to the method embodiments. The relevant portions of the method embodiments are referenced for the storage medium and computing device embodiments.

[0090] Those skilled in the art should understand that, in one or more examples described above, the functions described in the embodiments of the present application can be implemented in hardware, software, firmware or any combination thereof. When implemented in software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or code on a computer readable medium.

[0091] The above detailed description has shown, described, and pointed out the aspects of the present application as applied to the illustrative embodiments. Based upon these teachings, one skilled in the art will recognize that various changes can be made to the present application without departing from the scope of the present application, and the scope of the present application should not be limited to what has been described above. Accordingly, all changes coming within the meaning and equivalency range of the claims are intended to be embraced in the claim.

Claims

1. A method for synchronizing multiple media streams, the method comprising: In response to receiving a media data packet of a target media stream, the initial timestamp modification amount of the media data packet of the target media stream is determined based on the current timestamp modification amount of the received first media stream, wherein the target media stream is a media stream received after the first media stream; Determine the relative delay of the target media stream relative to the first media stream; Based on the relative delay, update the timestamp modification amount of each received media stream; Modify the timestamp of the data packets of each media stream based on the amount of timestamp modification for each media stream; The step of updating the timestamp modification amount of each received media stream according to the relative delay includes: Based on the relative delay, determine the relative magnitude relationship between the delay of the target media stream and the delay of the first media stream; In response to determining that the delay of the target media stream is less than the delay of the first media stream, the modification amount corresponding to the relative delay is added to the current timestamp modification amount of each received media stream other than the target media stream.

2. The method according to claim 1, wherein, The step of updating the timestamp modification amount of each received media stream according to the relative delay further includes: In response to determining that the delay of the target media stream is greater than the delay of the first media stream, the modification amount corresponding to the relative delay is added to the current timestamp modification amount of the target media stream.

3. The method according to claim 1, wherein, The method further includes: Receive synchronization information of the target media stream; and Determining the relative delay of the target media stream relative to the first media stream includes: The relative delay of the target media stream relative to the first media stream is calculated based on the synchronization information of the target media stream and the synchronization information of the first media stream received in advance, wherein the synchronization information includes time information reported by the sender.

4. The method according to claim 3, wherein, Before determining the relative delay of the target media stream relative to the first media stream, the method further includes: Receive and store the synchronization information of the first media stream.

5. The method according to claim 1, wherein, Before receiving the media data packets of the target media stream, the method further includes: Receive data packets from the first media stream and set the initial timestamp modification amount of the first media stream to 0.

6. The method according to claim 1, wherein, Media data packets from one or more media streams have been received before the media data packets from the target media stream are received.

7. A device for synchronizing multiple media streams, the device comprising: The first determining unit is configured to, in response to receiving a media data packet of a target media stream, determine the initial timestamp modification amount of the media data packet of the target media stream based on the current timestamp modification amount of the received first media stream, wherein the target media stream is a media stream received after the first media stream; The second determining unit is configured to determine the relative delay of the target media stream relative to the first media stream; The update unit is configured to: determine the relative magnitude relationship between the delay of the target media stream and the delay of the first media stream based on the relative delay; and in response to determining that the delay of the target media stream is smaller than the delay of the first media stream, add the modification amount corresponding to the relative delay to the current timestamp modification amount of each received media stream other than the target media stream. The modification unit is configured to modify the timestamp of the data packets of each media stream based on the timestamp modification amount of each media stream.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-6.

9. An electronic device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-6.

Images