Audio and video synchronous playing method, playing device and playing equipment

Abstract

The audio and video synchronous playing method, the playing device and the playing equipment, wherein the audio and video synchronous playing method comprises: extracting the distribution position of the first frame audio in the code stream and extracting the distribution position of the first frame video in the code stream; according to the distribution position characteristics of the first frame audio and the first frame video in the code stream, adopting the corresponding synchronous strategy to process the audio and the video, so that the audio and the video are synchronous. By adopting the technical scheme, the audio and video synchronization under different starting scenes can be compatible, so as to meet different starting requirements.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of audio and video processing technology, and in particular to an audio and video synchronous playback method, playback device and playback equipment. Background Technology

[0002] When audio and video start playing, audio and video may become out of sync. In this case, it is necessary to process the audio and video to make them synchronized.

[0003] Currently, audio and video synchronization can be achieved by using video frames as a reference and matching the presentation time stamp (PTS) of the video frames with the audio frames, or by using audio frames as a reference and matching the PTS of the audio frames with the video frames.

[0004] However, the above-mentioned audio and video synchronization methods can only solve the audio and video synchronization problem in specific playback scenarios and cannot meet the different playback needs of users. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an audio and video synchronous playback method, playback device, and playback equipment that are compatible with audio and video synchronization in different playback scenarios, thereby meeting different playback requirements.

[0006] First, embodiments of the present invention provide a method for synchronized audio and video playback, including:

[0007] Extract the distribution position of the first frame of audio in the bitstream, and extract the distribution position of the first frame of video in the bitstream;

[0008] Based on the distribution characteristics of the first frame of audio and the first frame of video in the bitstream, the audio and the video are processed using corresponding synchronization strategies to synchronize them.

[0009] Accordingly, embodiments of the present invention also provide a playback device suitable for synchronously playing audio and video, including:

[0010] The first extraction unit is adapted to extract the distribution position of the first frame audio in the bitstream;

[0011] The second extraction unit is adapted to extract the distribution position of the first frame of video in the bitstream;

[0012] The processing unit is adapted to process the audio and video using a corresponding synchronization strategy based on the distribution location characteristics of the first frame audio and the first frame video in the bitstream, so that the audio and video are synchronized.

[0013] Accordingly, embodiments of the present invention also provide a playback device, including: a memory, a processor, a display, and an audio player, wherein the memory stores computer instructions that can be executed on the processor, and when the processor executes the computer instructions, it performs the steps of the audio and video synchronous playback method described in the foregoing embodiments.

[0014] The audio and video synchronized playback method in this embodiment of the invention extracts the distribution positions of the first frame of audio and the first frame of video in the bitstream, and processes the audio and video using corresponding synchronization strategies based on the distribution position characteristics of the first frame of audio and the first frame of video in the bitstream, thereby synchronizing the audio and video. This audio and video synchronized playback process can adapt to playback synchronization scenarios where the first frame of audio and the first frame of video have different distribution positions in the bitstream, and adopts corresponding synchronization strategies to process the audio and video for different playback scenarios, thus being compatible with audio and video synchronization under different playback scenarios and meeting different playback requirements. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart of an audio and video synchronous playback method according to an embodiment of the present invention is shown.

[0017] Figure 2 This diagram illustrates a uniform distribution of audio and video in a bitstream according to an embodiment of the present invention.

[0018] Figure 3 This diagram illustrates an embodiment of the present invention where audio and video are unevenly distributed in the bitstream.

[0019] Figure 4 This diagram illustrates a bitstream containing only audio data in an embodiment of the present invention.

[0020] Figure 5 This diagram illustrates a bitstream containing only video content, as shown in an embodiment of the present invention.

[0021] Figures 6a to 6h This diagram illustrates an audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame.

[0022] Figure 7a and Figure 7b This diagram illustrates an audio and video synchronization method in an embodiment of the present invention, where the timestamp of the first audio frame is greater than the timestamp of the first video frame.

[0023] Figure 8 This diagram illustrates an audio and video synchronization method in an embodiment of the present invention, where the timestamp of the first audio frame is equal to the timestamp of the first video frame.

[0024] Figure 9 A schematic diagram of the structure of a playback device according to an embodiment of the present invention is shown.

[0025] Figure 10 A schematic diagram of the structure of a playback device according to an embodiment of the present invention is shown. Detailed Implementation

[0026] As described in the background section, current audio and video synchronization methods can only solve the audio and video synchronization problem in specific playback scenarios and cannot meet different playback requirements.

[0027] To address the aforementioned issues, the distribution positions of the first audio and video frames in the bitstream are extracted. Based on the distribution position characteristics of the first audio and video frames in the bitstream, corresponding synchronization strategies are applied to the audio and video to ensure synchronization. This synchronization is compatible with audio and video synchronization in different playback scenarios, thereby meeting various playback requirements.

[0028] To enable those skilled in the art to better understand, implement and realize the embodiments of the present invention, the following detailed description is provided with reference to the accompanying drawings and specific embodiments.

[0029] Reference Figure 1 The flowchart shown in this embodiment of the invention illustrates the method for synchronizing audio and video playback. Specifically, in this embodiment, the audio and video can be synchronized according to the following steps.

[0030] S11, extract the distribution position of the first frame audio in the bitstream, and extract the distribution position of the first frame video in the bitstream.

[0031] It should be noted that, in this embodiment of the invention, the timing of extracting the first frame of audio and the first frame of video is not required. The first frame of audio can be extracted first, the first frame of video can be extracted first, or the first frame of audio and the first frame of video can be extracted simultaneously.

[0032] Wherein, the first audio frame refers to the first audio frame in the bitstream to be played, and the first video frame refers to the first video frame in the bitstream to be played.

[0033] S12, based on the distribution characteristics of the first frame audio and the first frame video in the bitstream, the audio and the video are processed using a corresponding synchronization strategy to synchronize the audio and the video.

[0034] Specifically, for different types of playback sources, the audio and video are distributed in different positions in the bitstream. By extracting the distribution positions of the first frame of audio and the first frame of video in the bitstream, the distribution position characteristics of the first frame of audio and the first frame of video in the bitstream can be determined. Based on the distribution position characteristics of the first frame of audio and the first frame of video in the bitstream, a corresponding synchronization strategy is adopted to process the audio and video, so that the audio and the video are synchronized.

[0035] In specific implementation, the playback scenarios of the source file can be divided according to the distribution relationship between the first frame of audio and the first frame of video in the bitstream, and corresponding synchronization strategies can be adopted for different playback scenarios to synchronize the audio and the video.

[0036] Among them, "starting playback" refers to the start of playback of the bitstream in the acquired video source file at a certain moment. Specifically, starting playback can include various playback scenarios such as starting playback, on-demand playback, channel switching, live playback, resume playback, and replay playback.

[0037] As shown above, by extracting the distribution positions of the first audio and video frames in the bitstream, and based on their distribution position characteristics, a corresponding synchronization strategy is applied to process the audio and video, thus synchronizing them. This audio and video synchronization playback process can adapt to different playback synchronization scenarios where the first audio and video frames are distributed at different positions in the bitstream. It employs corresponding synchronization strategies for different playback scenarios, ensuring compatibility with audio and video synchronization in various playback scenarios and meeting diverse playback requirements.

[0038] To enable those skilled in the art to better understand and implement the audio and video synchronous playback method in the embodiments of the present invention, the following detailed description is provided in conjunction with the accompanying drawings and specific application scenarios.

[0039] For ease of description and understanding, in the embodiments of the present invention, v0, v1, ..., v m ..., v n This represents the video frames in the bitstream. The display timestamp of the first video frame v0 is denoted by `frist_vpts`, the display timestamp of the second video frame v1 is denoted by `v1_apts`, and correspondingly, the display timestamp of the (n+1)th video frame v... n Use v to display the timestamp.n _apts represents: a0, a1, ..., a m ... a n This represents the audio frames in the bitstream, where the display timestamp of the first audio frame a0 is denoted by frist_apts, the display timestamp of the second audio frame a1 is denoted by a1_apts, and correspondingly, the display timestamp of the (n+1)th audio frame a... n Use 'a' to display the timestamp. n _apts represents the initial value of the system clock PCR, which is represented by first_pcr. The length of the rectangle represents the duration of the audio or video display, where m and n are both integers greater than or equal to 1, and n is greater than m.

[0040] It should be noted that, in the embodiments of the present invention, a0 and v0, a m With v m , and a n With v n There is no corresponding relationship.

[0041] In this embodiment of the invention, the audio and video can be processed using a corresponding synchronization strategy based on their distributional positional relationship within the bitstream, thereby synchronizing the audio and video. For example, the difference between the timestamp of the first audio frame and the timestamp of the first video frame can be used to characterize their distributional positional relationship within the bitstream.

[0042] The distribution positional relationship between the first frame audio and the first frame video in the bitstream may include whether their distribution in the bitstream is uniform.

[0043] Specifically, when the interval between the extracted first frame audio and the first frame video in the bitstream is less than a preset threshold, it can be considered that the audio and the video are evenly distributed in the bitstream.

[0044] For example, when the absolute value of the difference between the timestamps of the first audio frame and the first video frame is less than 3 seconds, it is determined that the audio and video are evenly distributed in the bitstream. (Refer to...) Figure 2 The diagram shown here illustrates the uniform distribution of audio and video in the bitstream in an embodiment of the present invention. Figure 2 It can be seen that when the absolute value of the difference between the extracted first frame audio display timestamp first_apts and the first frame video display timestamp first_vpts is less than 3s, the audio and video are evenly distributed in the bitstream.

[0045] In this embodiment of the invention, when the interval between the extracted first frame audio and the first frame video in the bitstream is greater than a preset threshold, it can be considered that the audio and the video are unevenly distributed in the bitstream.

[0046] In practice, uneven distribution of audio and video in the bitstream includes two situations:

[0047] 1) The first video frame is positioned before the first audio frame in the bitstream;

[0048] 2) The first audio frame is positioned before the first video frame in the bitstream.

[0049] For example, when the absolute value of the difference between the first frame audio display timestamp `first_apts` and the first frame video display timestamp `first_vpts` is greater than 3 seconds, i.e., |first_apts - first_vpts| > 3 seconds, it can be considered that the audio and video are unevenly distributed in the bitstream. (Refer to...) Figure 3 The schematic diagram of uneven distribution of audio and video in the bitstream in the embodiment of the present invention is shown below. Figure 3 The neutron graph (a) shows that the first frame of video v0 is located before the first frame of audio a0 in the bitstream, at which point first_vpts-first_apts>3s; Figure 3 The neutron graph (b) shows that the first frame audio a0 is located before the first frame video v0 in the bitstream. At this time, first_apts-first_vpts>3s.

[0050] In practice, the bitstream in the source file being played may consist entirely of audio frames, or entirely of video frames, without any audio frames.

[0051] Reference Figure 4 , Figure 5 The diagram shown in the embodiment of the present invention illustrates the bitstream distribution of audio and video, wherein... Figure 4 This indicates that only audio frames are present. Figure 5 This indicates that only video frames are present.

[0052] It should be noted that the above description of the distribution of audio and video in the bitstream is for illustrative purposes only and does not represent the actual distribution of audio and video frames in the bitstream.

[0053] To enable those skilled in the art to better understand and implement audio and video synchronization playback methods in different scenarios, the following describes the synchronization strategies corresponding to different distribution states of the audio and video in the bitstream through specific examples.

[0054] In this embodiment of the invention, when the distribution position interval between the first frame of audio and the first frame of video in the bitstream is less than a preset threshold, the display timestamp of the first frame of audio and the display timestamp of the first frame of video are extracted, and the audio and the video are processed according to the display timestamp of the first frame of audio and the display timestamp of the first frame of video using a corresponding synchronization strategy, so that the audio and the video are synchronized. The display timestamp of the first frame of audio and the display timestamp of the first frame of video are generated based on the same system clock.

[0055] In specific implementation, the timestamp of the first audio frame and the timestamp of the first video frame are related as follows: the timestamp of the first audio frame is equal to the timestamp of the first video frame; the timestamp of the first audio frame is less than the timestamp of the first video frame; and the timestamp of the first audio frame is greater than the timestamp of the first video frame.

[0056] The audio and video synchronization strategies in the above situations are described below.

[0057] Scenario 1: When the interval between the distribution positions of the first audio frame and the first video frame in the bitstream is less than a preset threshold, and the display timestamp of the first audio frame is less than the display timestamp of the first video frame.

[0058] In this embodiment of the invention, for source files where the interval between the distribution positions of the first frame of audio and the first frame of video in the bitstream is less than a preset threshold, and the display timestamp of the first frame of audio is less than the display timestamp of the first frame of video, this embodiment of the invention provides eight synchronization strategies to synchronize the audio and the video.

[0059] In specific implementation, when the timestamp of the first frame of the audio is less than the timestamp of the first frame of the video, the audio is played directly, and the initial value of the system clock is determined by the timestamp of the first frame of the audio; when the system clock increases to match the timestamp of the first frame of the video, the video is played.

[0060] Reference Figure 6a The diagram shown is a schematic representation of an audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6a As shown, at the first_apts time, the audio can be played directly, and the initial value of the system clock PCR, first_pcr, is determined by the timestamp first_apts of the first frame of the audio, where first_apts = first_apts - buf_threshold.

[0061] In this embodiment of the invention, buf_threshold represents the buffer threshold.

[0062] As a specific example, buf_threshold can be equal to 300ms * 90, where 300ms is the buffer time and 90 is the conversion unit for timestamps. During the period from first_apts to first_vpts, the first frame a0 and the second frame a1 of the audio are played. Since the timestamp first_vpts of the first frame v0 of the video is greater than the current timestamp of the system clock PCR, the video is not played.

[0063] At time first_vpts, since the current timestamp of the system clock PCR increases to be the same as the display timestamp first_vpts of the first frame v0 of the video, the video is played from the first frame v0 to the (n+1)th frame v. n Meanwhile, the audio continues to play from the third frame a2 to the (n+1)th frame a. n .

[0064] Since the initial value of the system clock is determined by the timestamp of the first audio frame, and the output of the first video frame is determined by the current timestamp of the system clock and the timestamp of the first audio frame, the audio display timestamp and the video display timestamp are both associated with the same system clock, and the video and the audio can be synchronized at the first_vpts time.

[0065] It should be noted that, in order to avoid data fluctuations in the link, some audio data needs to be buffered before playback starts. Therefore, the initial value of the system clock PCR, frist_pcr, can be less than the first frame display timestamp of the audio, first_apts.

[0066] In practical implementation, for example, during channel switching, audio can be played directly, and the video can be played only when the system clock matches the timestamp displayed in the first frame of the video, thus achieving synchronization between audio and video.

[0067] However, in such Figure 6a During the first_apts to first_vpts period shown, there will be a period of black screen due to no video being played, which affects the visual experience.

[0068] To address this, the first frame of the video can be played directly and its display status maintained to eliminate the black screen issue. Specifically, when the timestamp of the first audio frame is less than the timestamp of the first video frame, both the audio and the first video frame are played directly, and the initial value of the system clock is determined by the timestamp of the first audio frame. The display status of the first video frame is maintained until the system clock increases to match the timestamp of the second video frame, at which point the remaining frames of the video are played.

[0069] Reference Figure 6b The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6b As shown, at the first_apts time, the audio and the first frame v0 of the video are played directly (i.e., the first frame v0 of the video is output directly without referencing the system clock PCR), and the initial value of the system clock PCR, first_pcr, is determined by the timestamp first_apts of the first audio frame, where first_pcr = first_apts - buf_threshold.

[0070] In this embodiment of the invention, buf_threshold represents the buffer threshold.

[0071] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0072] During the period from first_apts to v1_apts, the first frame a0, the second frame a1, and the third frame a2 of the audio are played. By controlling the refresh state of the second frame v1 of the video, the display state of the first frame v0 of the video is maintained. For example, the second frame v1 of the video can be not refreshed in order to maintain the display state of the first frame v0 of the video.

[0073] At time v1_apts, since the timestamp of the second frame v1 of the video is the same as the current timestamp of the system clock PCR, the second frame v1 of the video is played to the (n+1)th frame v. n Meanwhile, the audio continues to play from frame a3 to frame a (n+1) of the fourth frame. n .

[0074] Since the initial value of the system clock is determined by the display timestamp of the first audio frame, and the output of the second video frame is determined by the current display timestamp of the system clock and the display timestamp of the first audio frame, the audio display timestamp and the video display timestamp are both associated with the same system clock, and the video and the audio can be synchronized at v1_apts time.

[0075] By employing the two synchronization strategies described above, audio and video can be synchronized through appropriate synchronization processing. Furthermore, since no frame dropping is performed during playback, complete audio and video information can be preserved, improving visual integrity.

[0076] In this embodiment of the invention, during the playback of the source file, the audio can be muted and the video can be processed accordingly to achieve audio and video synchronization.

[0077] In a specific implementation, when the timestamp of the first frame of the audio is less than the timestamp of the first frame of the video, the audio is played silently, and the initial value of the system clock is determined by the timestamp of the first frame of the audio; when the system clock increases to match the timestamp of the first frame of the video, the video and the audio are played simultaneously.

[0078] Reference Figure 6c The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6c As shown, due to the muting of the audio, the first_pcr of the system clock PCR is very close to the first_apts of the first frame a0 of the audio.

[0079] Specifically, first_pcr = first_apts - buf_threshold, where buf_threshold represents the buffer threshold.

[0080] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0081] During the time period from first_apts to first_vpts, the audio from the first frame a0 to the mth frame a... m-1 Playback is muted, as shown in the "mute" diagram. Since the display timestamp `first_vpts` of the first frame (v0) of the video is greater than the timestamp of the system clock `PCR`, the video is not played.

[0082] At time first_vpts, since the current timestamp of the system clock PCR increases to match the display timestamp first_vpts of the first frame v0 of the video, the (m+1)th frame a of the audio is played. m Up to frame n+1 a n Simultaneously playing the first frame v0 to the (n+1)th frame v of the video. n .

[0083] In this embodiment of the invention, to achieve synchronized output of the audio and video at the timestamp displayed in the first frame of the video, the number of frames in which the audio is muted can be controlled based on the timestamps of the first video frame (first_vpts), the timestamps of the first audio frame (first_apts), the audio sampling rate (samplate), and the number of channels (byte_width). Specifically, the number of frames in which the audio is played muted is:

[0084] mute-size=((first_vpts-first_apts) / 90)*(samplate / 1000)*byte_width;

[0085] Where (first_vpts-first_apts) / 90 represents the time during which the audio is muted.

[0086] In this embodiment of the invention, to eliminate the black screen phenomenon at the start of playback, the first frame of the video can be output directly without referencing the system clock PCR. Specifically, when the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is muted and the first video frame is played, with the initial value of the system clock determined by the timestamp of the first audio frame; the display state of the first video frame is maintained until the system clock increases to be greater than or equal to the timestamp of the second video frame, at which point the audio and video are played.

[0087] Reference Figure 6d The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6d As shown, due to the muting of the audio, the first_pcr of the system clock PCR is very close to the first_apts of the first frame a0 of the audio.

[0088] Specifically, first_pcr = first_apts - buf_threshold, where buf_threshold represents the buffer threshold.

[0089] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0090] During the time period from first_apts to v1_vpts, the audio from the first frame a0 to the mth frame a... m-1Mute playback is performed as shown in the figure, and the first frame v0 of the video is played (that is, the first frame v0 of the video is output directly without referencing the system clock PCR). At the same time, the refresh state of the second frame v1 of the video is controlled to maintain the display state of the first frame v0 of the video. For example, the second frame v1 of the video can be not refreshed in order to maintain the display state of the first frame v0 of the video.

[0091] At time v1_vpts, since the current timestamp of the system clock PCR increases to match the display timestamp of the second frame v1 of the video, the (m+1)th frame a of the audio is played. m Up to frame n+1 a n And play the second frame v1 to the (n+1)th frame v of the video. n .

[0092] Since the initial value of the system clock is determined by the display timestamp of the first audio frame, and the output of the second video frame is determined by the current display timestamp of the system clock and the display timestamp of the first audio frame, the audio display timestamp and the video display timestamp are both associated with the same system clock, and the video and the audio can be synchronized at v1_vpts.

[0093] In practice, directly performing frame drop (DF) processing on the audio will complete the audio data processing faster than outputting a silence frame. Therefore, embodiments of the present invention can also shorten the time required for audio and video synchronization by performing frame drop processing on the audio.

[0094] Specifically, when the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is dropped, and the initial value of the system clock is determined by the timestamp of the first audio frame after the audio is dropped; when the system clock increases to match the timestamp of the first video frame, the audio and the video are played simultaneously.

[0095] Reference Figure 6e The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6e As shown, by performing frame dropping processing on the audio, the current display timestamp of the audio can be made the same as the display timestamp of the first frame of the video, first_vpts.

[0096] Specifically, by controlling the number of frames dropped in the audio, the first audio 'a' after the audio frame drop can be... m The corresponding display timestamp is the same as the display timestamp first_vpts of the first frame of the video, and is generated by the first audio 'a' after the audio frame is dropped.m The corresponding display timestamp a m _vpts determines the initial value of the system clock PCR, first_vpt, as follows: Figure 6e As shown, a m _vpts=first_vpt, first_pcr=a m _vpts-buf_threshold, where buf_threshold represents the buffer threshold;

[0097] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0098] During the time period from first_apts to first_vpts, the first frame a0 to the mth frame a of the audio... m-1 Frame dropping is performed, as shown in DF in the figure. Since the timestamp first_vpts of the first frame v0 of the video is greater than the initial value first_pcr of the system clock PCR, the video is not played.

[0099] At time first_vpts, the audio frame dropping process is completed. The display timestamp first_vpts of the first video frame v0 is equal to the current timestamp of the system clock PCR, which is equal to the display timestamp a corresponding to the first audio am after the audio frame dropping. m _vpts, simultaneously playing the video and audio, that is, playing the first frame v0 to the (n+1)th frame v of the video. n Simultaneously play the (m+1)th frame a of the audio. m Up to frame n+1 a n .

[0100] In this embodiment of the invention, to achieve synchronized output of audio and video after frame dropping at the timestamp displayed in the first frame of the video, the number of audio frames dropped can be controlled based on the timestamps of the first video frame (first_vpts), the timestamps of the first audio frame (first_apts), the audio sampling rate (samplate), and the number of channels (byte_width). Specifically, the number of audio frames dropped is as follows:

[0101] drop-size=((first_vpts-first_apts) / 90)*(samplate / 1000)*byte_width;

[0102] Where (first_vpts-first_apts) / 90 represents the time for dropping frames in the audio.

[0103] In this embodiment of the invention, to eliminate the black screen phenomenon at the start of playback, the first frame of the video can be output directly without referencing the system clock PCR. Specifically, when the display timestamp of the first audio frame is less than the display timestamp of the first video frame, the audio is dropped, and the first video frame is played. The initial value of the system clock is determined by the display timestamp corresponding to the first audio frame after the audio frame drop. The display state of the first video frame is maintained until the system clock increases to be greater than or equal to the display timestamp of the second video frame, at which point the audio and the video are played simultaneously.

[0104] Reference Figure 6f The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6f As shown, by performing frame dropping processing on the audio, the display timestamp of the audio can be made very close to the initial value first_pcr of the system clock PCR;

[0105] Specifically, first_pcr = a m _apts-buf_threshold, where buf_threshold represents the buffer threshold.

[0106] As a specific example, buf_threshold can be equal to 150ms*90, where 150ms is the buffer time and 90 represents the timestamp conversion unit.

[0107] During the time interval from first_apts to v1_apts, the first frame a0 to the mth frame a of the audio... m-1 Frame dropping is performed, and the first frame v0 of the video is played (i.e., the first frame v0 of the video is output directly without referencing the system clock PCR), while the refresh state of the second frame v1 of the video is controlled to maintain the display state of the first frame v0 of the video; for example, the second frame v1 of the video can be not refreshed in order to maintain the display state of the first frame v0 of the video.

[0108] At time v1_apts, since the current timestamp of the system clock PCR increases to be greater than or equal to the display timestamp of the second frame v1 of the video, the second frame v1 of the video is played up to the (n+1)th frame v. n Meanwhile, continue playing the (m+1)th frame a of the audio. m Up to frame n+1 a n .

[0109] In this embodiment of the invention, to achieve synchronized playback of the audio and video after frame dropping at the timestamp displayed on the first frame of the video, the number of frames dropped for the audio can be controlled based on the timestamps of the first video frame (first_vpts), the timestamps of the first audio frame (first_apts), the sampling rate of the audio (samplate), and the number of channels (byte_width). Specifically, the number of frames dropped for the audio is as follows:

[0110] drop-size=((first_vpts-first_apts) / 90)*(samplate / 1000)*byte_width;

[0111] Where (first_vpts-first_apts) / 90 represents the time for dropping frames in the audio.

[0112] In practice, when playing a video source file, the playback link may contain a lot of audio decoded pulse code data that has already been buffered. If played directly, the difference between the display timestamps of the audio and video will become larger and larger, resulting in audio and video being out of sync. Therefore, the audio and video can be synchronized by discarding the buffered audio data in the playback link.

[0113] Specifically, when the timestamp of the first audio frame is less than the timestamp of the first video frame, the number of frames in the audio buffer is obtained, and the number of dropped audio frames is calculated. The number of frames in the audio buffer is compared with the number of dropped audio frames. If the number of dropped audio frames is greater than the number of audio frames in the audio buffer, the number of dropped audio frames is processed by dropping frames in the audio buffer, and the first video frame is played. The initial value of the system clock is determined by the timestamp corresponding to the first audio after the dropped audio frames. The audio is played at the timestamp corresponding to the first audio after the dropped audio frames. Furthermore, when the system clock is inconsistent with the current timestamp of the video, the video is played in slow motion until the system clock is consistent with the current timestamp of the video, at which point the video is played at normal speed.

[0114] In this embodiment of the invention, the frame number bf of the audio buffer can be obtained, and the frame number df of the audio frame drops can be calculated based on the display timestamp first_vpts of the first video frame, the display timestamp first_apts of the first audio frame, the sampling rate of the audio, and the number of channels byte_width. The frame number bf of the audio buffer is compared with the frame number df of the audio frame drops. If bf is greater than df, it can be processed according to... Figure 6fThe diagram shown illustrates synchronized audio and video playback. The corresponding processing of the audio and video is not detailed here. If bf is less than df, the number of frames df in the audio buffer of the playback link can be discarded, and then proceeded according to... Figure 6g The diagram illustrates the audio and video synchronization method, which performs corresponding processing on the audio and video.

[0115] Reference Figure 6g The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6g As shown, in first_apts to a m-1 During the _apts period, the number of frames a0 to a10 of the audio buffer. m-2 Perform frame dropping processing while playing the first frame v0 of the video (i.e., the first frame v0 of the video is output directly without referencing the system clock PCR);

[0116] In a m-1 _apts to a m During the _apts period, continue processing the audio buffer for the remaining frames a. m-1 Frame dropping is performed, and the video is played in slow motion by controlling the video display duration. For example, the display duration of the video frame can be set from the original 50ms to 60ms to achieve slow playback of the video.

[0117] In a m At time _apts, the frame dropping processing of the audio is completed, and the (m+1)th frame a of the audio is played. m And the first audio a after the audio frame was dropped. m The corresponding display timestamp determines the initial value of the system clock PCR, first_pcr, where first_pcr = a m _apts-buf_threshold, continue playing the video in slow motion;

[0118] In this embodiment of the invention, buf_threshold represents the buffer threshold.

[0119] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0120] It should be noted that since the number of frames in the audio buffer (bf) is less than the number of dropped audio frames (df), therefore, in a m At the _apts time, the audio and video are not synchronized, and the current display timestamp of the audio is slightly smaller than the display timestamp of the video.

[0121] In a m _apts to v i During the _apts period, since the current display timestamp of the video is greater than the initial value first_pcr of the system clock PCR, the audio plays normally, and the video continues to play slowly, where i is an integer greater than 1 and less than n;

[0122] In v i At time _apts, the current timestamp of the system clock PCR is equal to the current display timestamp v of the video. i _apts, plays the video at normal speed, that is, plays the m-th frame of the video at normal speed. m Up to frame n v n .

[0123] Since the initial value of the system clock is determined by the display timestamp corresponding to the first audio frame after the audio frame is dropped, and the timestamp of the video playing at normal speed is determined by the display timestamp of the first audio frame and the current display timestamp of the system clock, both the audio display timestamp and the video display timestamp are associated with the same system clock. Therefore, the video and audio can play at normal speed. i _apts is synchronized at all times.

[0124] It should be noted that because it takes some time for a video to switch from normal speed to slow speed, the first frame of the video will be displayed for a longer duration than the slow playback, making the video playback smoother when switching from normal speed to slow speed.

[0125] In this embodiment of the invention, when executing a channel switching command and receiving the bitstream of the next channel, in order to avoid the black screen phenomenon caused by waiting for the screen to display, the first frame of the video in the bitstream can be played directly after receiving the bitstream, and then the video can be played in slow speed to achieve synchronization of audio and video in the bitstream.

[0126] In specific implementation, when the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is played directly, and the first video frame is played. The initial value of the system clock is determined by the timestamp of the first audio frame. The video is played slowly until the system clock increases to match the current timestamp of the video, at which point the video is played at normal speed.

[0127] Reference Figure 6h The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6hAs shown, at the time first_apts, the bitstream of the next channel is received, and the initial value of the system clock PCR first_pcr is determined by the value of the timestamp first_apts of the first frame a0 of the audio in the bitstream, where first_pcr = first_apts - buf_threshold;

[0128] In this embodiment of the invention, buf_threshold represents the buffer threshold.

[0129] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0130] During the period from first_apts to a1_apts, the audio is played directly, and the first frame v0 of the video is played (i.e., the first frame v0 of the video is output directly without referencing the system clock PCR).

[0131] From a1_apts to v i During the _apts period, continue playing the audio from frame a1 to frame a+2. t+1 Simultaneously, by controlling the display duration of the video, the second frame v1 to the (m+1)th frame v of the video are... m To slow down the playback speed, for example, the display duration of video frames can be changed from 50ms to 60ms, thus achieving slow playback of the video. i _apts represents the current display timestamp of the video;

[0132] In v i At time _apts, the current timestamp of the system clock PCR is the same as the current display timestamp of the video. i _apts, plays the remaining frames of the video at normal speed, and continues playing the remaining frames of the audio, i.e., plays the (m+2)th frame of the video at normal speed. m+1 Up to frame n+1 of the video v n Simultaneously play the audio at frame a (t+2). t+1 Up to frame n+1 a n .

[0133] It should be noted that during slow-motion playback of the video, the audio and video are not completely synchronized. The audio and video can only achieve synchronized playback when the current display timestamp of the video matches the current display timestamp of the audio.

[0134] It should be noted that the above eight synchronization strategies are only illustrative examples. In practical applications, other synchronization strategies can also be used to process audio and video in the bitstream where the distribution position interval of the first frame audio and the first frame video is less than a preset threshold and the display timestamp of the first frame audio is less than the display timestamp of the first frame video, so as to achieve synchronization of the audio and video.

[0135] Scenario 2: When the interval between the distribution positions of the first audio frame and the first video frame in the bitstream is less than a preset threshold, and the display timestamp of the first audio frame is greater than the display timestamp of the first video frame.

[0136] In this embodiment of the invention, for source files where the interval between the distribution positions of the first frame of audio and the first frame of video in the bitstream is less than a preset threshold, and the display timestamp of the first frame of audio is greater than the display timestamp of the first frame of video, the audio and video can be processed accordingly using the following two synchronization strategies.

[0137] Synchronization Strategy 1: When the timestamp of the first frame of the audio is greater than the timestamp of the first frame of the video, the video is played directly, and empty pulse-coded data is added to the audio. The initial value of the system clock is determined by the timestamp of the first frame of the video. When the system clock increases to match the timestamp of the first frame of the audio, the audio is played.

[0138] Reference Figure 7a The diagram shown is a schematic representation of an audio and video synchronization method in an embodiment of the present invention, where the timestamp of the first audio frame is greater than the timestamp of the first video frame. Figure 7a As shown, at the first_vpts time, the video can be played directly, and the initial value of the system clock PCR, first_pcr, is determined by the timestamp first_vpts of the first frame of the video, where first_pcr = first_vpts - buf_threshold.

[0139] In this embodiment of the invention, buf_threshold represents the buffer threshold.

[0140] As a specific example, buf_threshold can be equal to 300ms*90, where 300ms represents the buffer time and 90 represents the timestamp conversion unit.

[0141] During the period from first_vpts to first_apts, the first frame v0, the second frame v1, and the third frame v2 of the video are played. Since the timestamp first_apts of the first audio frame is greater than the current timestamp of the system clock PCR, empty pulse-coded data can be added to the audio, as shown in SF in the figure, that is, a silent frame (SF) is added before the first audio frame v0.

[0142] At time first_apts, after playing the empty burst encoded data added to the audio, the current timestamp of the system clock PCR is the same as the display timestamp first_apts of the first frame a0 of the audio. Then, the audio and video are played from the first frame a0 to the (n+1)th frame a. n .

[0143] In this embodiment of the invention, to achieve synchronized playback of audio and video with added empty pulse-coded data at the timestamp of the first frame of the audio, the number of frames in which empty pulse-coded data is added to the audio can be strictly controlled based on the timestamps of the first frame of the video (first_vpts), the timestamps of the first frame of the audio (first_apts), the sampling rate of the audio (samplate), and the number of channels (byte_width). Specifically, the number of frames in which empty pulse-coded data is added to the audio is as follows:

[0144] add-size=((first_apts-first_vpts) / 90)*(samplate / 1000)*byte_width;

[0145] Where (first_apts-first_vpts) / 90 represents the time for adding empty pulse-coded data to the audio.

[0146] It should be noted that, in order to avoid data fluctuations in the link, some audio data needs to be buffered before the start of playback. Therefore, the initial value of the system clock PCR, first_pcr, can be less than the first frame display timestamp of the video, first_vpts.

[0147] Synchronization Strategy 2: When the timestamp of the first audio frame is greater than the timestamp of the first video frame, the first video frame is played directly, and the display state of the first video frame is maintained. The initial value of the system clock is determined by the timestamp of the first audio frame. The video is processed by dropping frames until the current timestamp of the video matches the timestamp of the first audio frame, at which point the audio and the video are played.

[0148] Reference Figure 7bThe diagram shown is a schematic representation of an audio and video synchronization method in an embodiment of the present invention, where the timestamp of the first audio frame is greater than the timestamp of the first video frame. Figure 7b As shown, at the first_vpts time, the first frame v0 of the video is played directly, and the display state of the first frame v0 of the video is maintained by controlling the refresh state of the second frame v1. For example, the second frame v1 of the video is not refreshed in order to maintain the display state of the first frame v0 of the video.

[0149] The initial value of the system clock PCR, first_pcr, is determined by the timestamp first_apts displayed in the first frame a0 of the audio. Specifically, first_pcr = first_apts - buf_threshold, where buf_threshold represents the buffer threshold.

[0150] As a specific example, buf_threshold can be equal to 150ms*90, where 150ms represents the buffer time and 90 represents the timestamp conversion unit;

[0151] During the time interval from first_vpts to first_apts, the first frame v0 of the video is played and its display state is maintained, while the second frame v1 to the m-th frame v... m-1 Frame dropping is performed, as shown in DF in the figure. Since the timestamp first_apts of the first audio frame a0 is greater than the timestamp of the system clock PCR, the audio is not played.

[0152] At time first_apts, the display timestamp v1_vpts of the first video frame v1 is the same as the display timestamp first_apts of the first audio frame a0, and the audio from the first frame a0 to the (n+1)th frame a is played. n Simultaneously play the (m+1)th frame v of the video. m Up to frame n+1 v n .

[0153] In this embodiment of the invention, to achieve synchronized playback of the video and audio after frame drops at the timestamp displayed on the first frame of the audio, the number of dropped video frames can be controlled based on the timestamps of the first video frame (first_vpts), the timestamps of the first audio frame (first_apts), the sampling rate of the audio (samplate), and the number of channels (byte_width). Specifically, the number of dropped audio frames is as follows:

[0154] drop-size=((first_apts-first_tpts) / 90)*(samplate / 1000)*byte_width;

[0155] Where (first_apts-first_vpts) / 90 represents the time of the video frame loss.

[0156] Scenario 3: When the interval between the distribution positions of the first audio frame and the first video frame in the bitstream is less than a preset threshold, and the display timestamp of the first audio frame is equal to the display timestamp of the first video frame.

[0157] In this embodiment of the invention, for the situation described in scenario three, when the timestamp of the first frame of the audio is equal to the timestamp of the first frame of the video, the audio and the video are played directly.

[0158] Reference Figure 8 The diagram shown is a schematic representation of an audio and video synchronization method in an embodiment of the present invention, where the timestamp of the first audio frame is equal to the timestamp of the first video frame. Figure 8 As shown, the audio and video buffers some data, and at the first_apts time, the first frame a0 to the (n+1)th frame a of the audio are played directly. n+1 and the first frame v0 to the (n+1)th frame v of the video n .

[0159] In practice, the first frame of video and the first frame of audio in the source file are often spaced far apart, resulting in only one data format being available for a considerable period. Therefore, two scenarios exist: the first is where the video comes first, such as... Figure 3 The neutron diagram (a) is shown; the second type has audio first, as shown in the diagram. Figure 3 The neutron diagram (b) is shown.

[0160] In embodiments of the present invention, such as Figure 3 As shown in neutron diagram (a), when the interval between the distribution positions of the first frame audio and the first frame video in the bitstream is greater than a preset threshold, and the distribution position of the first frame video in the bitstream is before the first frame audio, in order to synchronize the audio and video, the video can be played first, and the current display timestamp of the video and the display timestamp of the first frame audio can be extracted. Based on the current display timestamp of the video and the display timestamp of the first frame audio, the audio can be processed using a corresponding synchronization strategy to synchronize the processed audio with the video.

[0161] Specifically, when the timestamp of the first frame of the audio is equal to the current timestamp of the video, the audio is played directly.

[0162] In this embodiment of the invention, when it is determined that the current display timestamp of the video and the display timestamp of the first frame of the audio are not aligned, and the display timestamp of the first frame of the audio is greater than the current display timestamp of the video, it indicates that the first frame of the audio is lagging behind the current video frame. At this time, the audio can be played silently until the display timestamp of the audio matches the display timestamp of the video, at which point the audio is played again to achieve synchronized playback of audio and video.

[0163] In specific implementation, the number of frames for silent playback of the audio is:

[0164] mute-size = ((first_apts-a i _vpts) / 90)*(samplate / 1000)*byte_width;

[0165] Where 'samplate' represents the audio sampling rate, 'byte_width' represents the number of channels, and 'a' represents the audio sampling rate. i _vpts represents the current display timestamp of the video, where i is an integer greater than or equal to 0.

[0166] When it is determined that the current display timestamp of the video and the first frame display timestamp of the current audio are not aligned, and the first frame display timestamp of the audio is less than the current display timestamp of the video, the audio is processed by dropping frames until the display timestamp of the audio matches the display timestamp of the video, at which point the audio is played to achieve synchronized playback of audio and video.

[0167] By muting or dropping frames in the audio, the integrity of the video information can be guaranteed while achieving synchronized playback of audio and video. This is especially important in situations where video-on-demand or live streaming starts, where the transmission cost of video frames in the bitstream is relatively high. If audio and video synchronization is achieved by dropping video frames, it often leads to a waste of video resources.

[0168] In this embodiment of the invention, when the interval between the distribution positions of the first frame audio and the first frame video in the bitstream is greater than a preset threshold, and the distribution position of the first frame video in the bitstream precedes that of the first frame audio, such as Figure 3 As shown in sub-figure (a), the display timestamp of the first frame of the video can be extracted, and then the display timestamp of the first frame of the audio can be extracted. Based on the display timestamps of the first frame of the video and the first frame of the audio, the video and the audio are processed using corresponding synchronization strategies to synchronize the video and the audio.

[0169] The timestamp of the first frame of the video and the timestamp of the first frame of the audio are generated based on the same system clock.

[0170] In specific implementation, the timestamp of the first audio frame and the timestamp of the first video frame are related as follows: the timestamp of the first audio frame is equal to the timestamp of the first video frame; the timestamp of the first audio frame is less than the timestamp of the first video frame; and the timestamp of the first audio frame is greater than the timestamp of the first video frame.

[0171] The audio and video synchronization strategies in the above situations are described below.

[0172] Scenario 4: When the interval between the first audio frame and the first video frame in the bitstream is greater than a preset threshold, and the first video frame is positioned before the first audio frame in the bitstream, the timestamp of the first audio frame is less than the timestamp of the first video frame.

[0173] In this embodiment of the invention, for source files where the interval between the distribution positions of the first audio frame and the first video frame in the bitstream is greater than a preset threshold, and the distribution position of the first video frame in the bitstream is before the first audio frame, and the display timestamp of the first audio frame is less than the display timestamp of the first video frame, this embodiment of the invention provides eight synchronization strategies to synchronize the audio and the video.

[0174] In specific implementation, when the timestamp of the first frame of the audio is less than the timestamp of the first frame of the video, the audio is played directly, and the initial value of the system clock is determined by the timestamp of the first frame of the audio; when the system clock increases to match the timestamp of the first frame of the video, the video is played.

[0175] Continue to refer to Figure 6a The diagram shown is a schematic representation of an audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6a As shown, at the first_apts time, the audio can be played directly, and the initial value of the system clock PCR, first_pcr, is determined by the timestamp first_apts of the first frame of the audio, where first_apts = first_apts - buf_threshold.

[0176] In this embodiment of the invention, buf_threshold represents the buffer threshold.

[0177] As a specific example, buf_threshold can be equal to 300ms * 90, where 300ms is the buffer time and 90 is the conversion unit for timestamps. During the period from first_apts to first_vpts, the first frame a0 and the second frame a1 of the audio are played. Since the timestamp first_vpts of the first frame v0 of the video is greater than the current timestamp of the system clock PCR, the video is not played.

[0178] At time first_vpts, since the current timestamp of the system clock PCR increases to be the same as the display timestamp first_vpts of the first frame v0 of the video, the video is played from the first frame v0 to the (n+1)th frame v. n Meanwhile, the audio continues to play from the third frame a2 to the (n+1)th frame a. n .

[0179] Since the initial value of the system clock is determined by the timestamp of the first audio frame, and the output of the first video frame is determined by the current timestamp of the system clock and the timestamp of the first audio frame, the audio display timestamp and the video display timestamp are both associated with the same system clock, and the video and the audio can be synchronized at the first_vpts time.

[0180] It should be noted that, in order to avoid data fluctuations in the link, some audio data needs to be buffered before playback starts. Therefore, the initial value of the system clock PCR, frist_pcr, can be less than the first frame display timestamp of the audio, first_apts.

[0181] In practical implementation, for example, during channel switching, audio can be played directly, and the video can be played only when the system clock matches the timestamp displayed in the first frame of the video, thus achieving synchronization between audio and video.

[0182] However, in such Figure 6a During the first_apts to first_vpts period shown, there will be a period of black screen due to no video being played, which affects the visual experience.

[0183] To address this, the first frame of the video can be played directly and its display status maintained to eliminate the black screen issue. Specifically, when the timestamp of the first audio frame is less than the timestamp of the first video frame, both the audio and the first video frame are played directly, and the initial value of the system clock is determined by the timestamp of the first audio frame. The display status of the first video frame is maintained until the system clock increases to match the timestamp of the second video frame, at which point the remaining frames of the video are played.

[0184] Continue to refer to Figure 6b The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6b As shown, at the first_apts time, the audio and the first frame v0 of the video are played directly (i.e., the first frame v0 of the video is output directly without referencing the system clock PCR), and the initial value of the system clock PCR, first_pcr, is determined by the timestamp first_apts of the first audio frame, where first_pcr = first_apts - buf_threshold.

[0185] In this embodiment of the invention, buf_threshold represents the buffer threshold.

[0186] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0187] During the period from first_apts to v1_apts, the first frame a0, the second frame a1, and the third frame a2 of the audio are played. By controlling the refresh state of the second frame v1 of the video, the display state of the first frame v0 of the video is maintained. For example, the second frame v1 of the video can be not refreshed in order to maintain the display state of the first frame v0 of the video.

[0188] At time v1_apts, since the timestamp of the second frame v1 of the video is the same as the current timestamp of the system clock PCR, the second frame v1 of the video is played to the (n+1)th frame v. n Meanwhile, the audio continues to play from frame a3 to frame a (n+1) of the fourth frame. n .

[0189] Since the initial value of the system clock is determined by the display timestamp of the first audio frame, and the output of the second video frame is determined by the current display timestamp of the system clock and the display timestamp of the first audio frame, the audio display timestamp and the video display timestamp are both associated with the same system clock, and the video and the audio can be synchronized at v1_apts time.

[0190] By employing the two synchronization strategies described above, audio and video can be synchronized through appropriate synchronization processing. Furthermore, since no frame dropping is performed during playback, complete audio and video information can be preserved, improving visual integrity.

[0191] In this embodiment of the invention, during the playback of the source file, the audio can be muted and the video can be processed accordingly to achieve audio and video synchronization.

[0192] In a specific implementation, when the timestamp of the first frame of the audio is less than the timestamp of the first frame of the video, the audio is played silently, and the initial value of the system clock is determined by the timestamp of the first frame of the audio; when the system clock increases to match the timestamp of the first frame of the video, the video and the audio are played simultaneously.

[0193] Continue to refer to Figure 6c The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6c As shown, due to the muting of the audio, the first_pcr of the system clock PCR is very close to the first_apts of the first frame a0 of the audio.

[0194] Specifically, first_pcr = first_apts - buf_threshold, where buf_threshold represents the buffer threshold.

[0195] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0196] During the time period from first_apts to first_vpts, the audio from the first frame a0 to the mth frame a... m-1 Playback is muted, as shown in the "mute" diagram. Since the display timestamp `first_vpts` of the first frame (v0) of the video is greater than the timestamp of the system clock `PCR`, the video is not played.

[0197] At time first_vpts, since the current timestamp of the system clock PCR increases to match the display timestamp first_vpts of the first frame v0 of the video, the (m+1)th frame a of the audio is played. m Up to frame n+1 a n Simultaneously playing the first frame v0 to the (n+1)th frame v of the video. n .

[0198] In this embodiment of the invention, to achieve synchronized output of the audio and video at the timestamp displayed in the first frame of the video, the number of frames in which the audio is muted can be controlled based on the timestamps of the first video frame (first_vpts), the timestamps of the first audio frame (first_apts), the audio sampling rate (samplate), and the number of channels (byte_width). Specifically, the number of frames in which the audio is played muted is:

[0199] mute-size=((first_vpts-first_apts) / 90)*(samplate / 1000)*byte_width;

[0200] Where (first_vpts-first_apts) / 90 represents the time during which the audio is muted.

[0201] In this embodiment of the invention, to eliminate the black screen phenomenon at the start of playback, the first frame of the video can be output directly without referencing the system clock PCR. Specifically, when the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is muted and the first video frame is played, with the initial value of the system clock determined by the timestamp of the first audio frame; the display state of the first video frame is maintained until the system clock increases to be greater than or equal to the timestamp of the second video frame, at which point the audio and video are played.

[0202] Continue to refer to Figure 6d The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6d As shown, due to the muting of the audio, the first_pcr of the system clock PCR is very close to the first_apts of the first frame a0 of the audio.

[0203] Specifically, first_pcr = first_apts - buf_threshold, where buf_threshold represents the buffer threshold.

[0204] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0205] During the time period from first_apts to v1_vpts, the audio from the first frame a0 to the mth frame a... m-1Mute playback is performed as shown in the figure, and the first frame v0 of the video is played (that is, the first frame v0 of the video is output directly without referencing the system clock PCR). At the same time, the refresh state of the second frame v1 of the video is controlled to maintain the display state of the first frame v0 of the video. For example, the second frame v1 of the video can be not refreshed in order to maintain the display state of the first frame v0 of the video.

[0206] At time v1_vpts, since the current timestamp of the system clock PCR increases to match the display timestamp of the second frame v1 of the video, the (m+1)th frame a of the audio is played. m Up to frame n+1 a n And play the second frame v1 to the (n+1)th frame v of the video. n .

[0207] Since the initial value of the system clock is determined by the display timestamp of the first audio frame, and the output of the second video frame is determined by the current display timestamp of the system clock and the display timestamp of the first audio frame, the audio display timestamp and the video display timestamp are both associated with the same system clock, and the video and the audio can be synchronized at v1_vpts.

[0208] In practice, directly performing frame drop (DF) processing on the audio will complete the audio data processing faster than outputting a silence frame. Therefore, embodiments of the present invention can also shorten the time required for audio and video synchronization by performing frame drop processing on the audio.

[0209] Specifically, when the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is dropped, and the initial value of the system clock is determined by the timestamp of the first audio frame after the audio is dropped; when the system clock increases to match the timestamp of the first video frame, the audio and the video are played simultaneously.

[0210] Continue to refer to Figure 6e The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6e As shown, by performing frame dropping processing on the audio, the current display timestamp of the audio can be made the same as the display timestamp of the first frame of the video, first_vpts.

[0211] Specifically, by controlling the number of frames dropped in the audio, the first audio 'a' after the audio frame drop can be... mThe corresponding display timestamp is the same as the display timestamp first_vpts of the first frame of the video, and is generated by the first audio 'a' after the audio frame is dropped. m The corresponding display timestamp a m _vpts determines the initial value of the system clock PCR, first_vpt, as follows: Figure 6e As shown, a m _vpts=first_vpt, first_pcr=a m _vpts-buf_threshold, where buf_threshold represents the buffer threshold;

[0212] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0213] During the time period from first_apts to first_vpts, the first frame a0 to the mth frame a of the audio... m-1 Frame dropping is performed, as shown in DF in the figure. Since the timestamp first_vpts of the first frame v0 of the video is greater than the initial value first_pcr of the system clock PCR, the video is not played.

[0214] At time first_vpts, the audio frame dropping process is completed. The display timestamp first_vpts of the first video frame v0 is equal to the current timestamp of the system clock PCR, which is equal to the display timestamp a corresponding to the first audio am after the audio frame dropping. m _vpts, simultaneously playing the video and audio, that is, playing the first frame v0 to the (n+1)th frame v of the video. n Simultaneously play the (m+1)th frame a of the audio. m Up to frame n+1 a n .

[0215] In this embodiment of the invention, to achieve synchronized output of audio and video after frame dropping at the timestamp displayed in the first frame of the video, the number of audio frames dropped can be controlled based on the timestamps of the first video frame (first_vpts), the timestamps of the first audio frame (first_apts), the audio sampling rate (samplate), and the number of channels (byte_width). Specifically, the number of audio frames dropped is as follows:

[0216] drop-size=((first_vpts-first_apts) / 90)*(samplate / 1000)*byte_width;

[0217] Where (first_vpts-first_apts) / 90 represents the time for dropping frames in the audio.

[0218] In this embodiment of the invention, to eliminate the black screen phenomenon at the start of playback, the first frame of the video can be output directly without referencing the system clock PCR. Specifically, when the display timestamp of the first audio frame is less than the display timestamp of the first video frame, the audio is dropped, and the first video frame is played. The initial value of the system clock is determined by the display timestamp corresponding to the first audio frame after the audio frame drop. The display state of the first video frame is maintained until the system clock increases to be greater than or equal to the display timestamp of the second video frame, at which point the audio and the video are played simultaneously.

[0219] Continue to refer to Figure 6f The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6f As shown, by performing frame dropping processing on the audio, the display timestamp of the audio can be made very close to the initial value first_pcr of the system clock PCR;

[0220] Specifically, first_pcr = a m _apts-buf_threshold, where buf_threshold represents the buffer threshold.

[0221] As a specific example, buf_threshold can be equal to 150ms*90, where 150ms is the buffer time and 90 represents the timestamp conversion unit.

[0222] During the time interval from first_apts to v1_apts, the first frame a0 to the mth frame a of the audio... m-1 Frame dropping is performed, and the first frame v0 of the video is played (i.e., the first frame v0 of the video is output directly without referencing the system clock PCR), while the refresh state of the second frame v1 of the video is controlled to maintain the display state of the first frame v0 of the video; for example, the second frame v1 of the video can be not refreshed in order to maintain the display state of the first frame v0 of the video.

[0223] At time v1_apts, since the current timestamp of the system clock PCR increases to be greater than or equal to the display timestamp of the second frame v1 of the video, the second frame v1 of the video is played up to the (n+1)th frame v. n Meanwhile, continue playing the (m+1)th frame a of the audio. m Up to frame n+1 a n .

[0224] In this embodiment of the invention, to achieve synchronized playback of the audio and video after frame dropping at the timestamp displayed on the first frame of the video, the number of frames dropped for the audio can be controlled based on the timestamps of the first video frame (first_vpts), the timestamps of the first audio frame (first_apts), the sampling rate of the audio (samplate), and the number of channels (byte_width). Specifically, the number of frames dropped for the audio is as follows:

[0225] drop-size=((first_vpts-first_apts) / 90)*(samplate / 1000)*byte_width;

[0226] Where (first_vpts-first_apts) / 90 represents the time for dropping frames in the audio.

[0227] In practice, when playing a video source file, the playback link may contain a lot of audio decoded pulse code data that has already been buffered. If played directly, the difference between the display timestamps of the audio and video will become larger and larger, resulting in audio and video being out of sync. Therefore, the audio and video can be synchronized by discarding the buffered audio data in the playback link.

[0228] Specifically, when the timestamp of the first audio frame is less than the timestamp of the first video frame, the number of frames in the audio buffer is obtained, and the number of dropped audio frames is calculated. The number of frames in the audio buffer is compared with the number of dropped audio frames. If the number of dropped audio frames is greater than the number of audio frames in the audio buffer, the number of dropped audio frames is processed by dropping frames in the audio buffer, and the first video frame is played. The initial value of the system clock is determined by the timestamp corresponding to the first audio after the dropped audio frames. The audio is played at the timestamp corresponding to the first audio after the dropped audio frames. Furthermore, when the system clock is inconsistent with the current timestamp of the video, the video is played in slow motion until the system clock is consistent with the current timestamp of the video, at which point the video is played at normal speed.

[0229] In this embodiment of the invention, the frame number bf of the audio buffer can be obtained, and the frame number df of the audio frame drops can be calculated based on the display timestamp first_vpts of the first video frame, the display timestamp first_apts of the first audio frame, the sampling rate of the audio, and the number of channels byte_width. The frame number bf of the audio buffer is compared with the frame number df of the audio frame drops. If bf is greater than df, it can be processed according to... Figure 6fThe diagram shown illustrates synchronized audio and video playback. The corresponding processing of the audio and video is not detailed here. If bf is less than df, the number of frames df in the audio buffer of the playback link can be discarded, and then proceeded according to... Figure 6g The diagram illustrates the audio and video synchronization method, which performs corresponding processing on the audio and video.

[0230] Continue to refer to Figure 6g The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6g As shown, in first_apts to a m-1 During the _apts period, the number of frames a0 to a10 of the audio buffer. m-2 Perform frame dropping processing while playing the first frame v0 of the video (i.e., the first frame v0 of the video is output directly without referencing the system clock PCR);

[0231] In a m-1 _apts to a m During the _apts period, continue processing the audio buffer for the remaining frames a. m-1 Frame dropping is performed, and the video is played in slow motion by controlling the video display duration. For example, the display duration of the video frame can be set from the original 50ms to 60ms to achieve slow playback of the video.

[0232] In a m At time _apts, the frame dropping processing of the audio is completed, and the (m+1)th frame a of the audio is played. m And the first audio a after the audio frame was dropped. m The corresponding display timestamp determines the initial value of the system clock PCR, first_pcr, where first_pcr = a m _apts-buf_threshold, continue playing the video in slow motion;

[0233] In this embodiment of the invention, buf_threshold represents the buffer threshold.

[0234] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0235] It should be noted that since the number of frames in the audio buffer (bf) is less than the number of dropped audio frames (df), therefore, in a m At the _apts time, the audio and video are not synchronized, and the current display timestamp of the audio is slightly smaller than the display timestamp of the video.

[0236] In a m _apts to v i During the _apts period, since the current display timestamp of the video is greater than the initial value first_pcr of the system clock PCR, the audio plays normally, and the video continues to play slowly, where i is an integer greater than 1 and less than n;

[0237] In v i At time _apts, the current timestamp of the system clock PCR is equal to the current display timestamp v of the video. i _apts, plays the video at normal speed, that is, plays the m-th frame of the video at normal speed. m Up to frame n v n .

[0238] Since the initial value of the system clock is determined by the display timestamp corresponding to the first audio frame after the audio frame is dropped, and the timestamp of the video playing at normal speed is determined by the display timestamp of the first audio frame and the current display timestamp of the system clock, both the audio display timestamp and the video display timestamp are associated with the same system clock. Therefore, the video and audio can play at normal speed. i _apts is synchronized at all times.

[0239] It should be noted that because it takes some time for a video to switch from normal speed to slow speed, the first frame of the video will be displayed for a longer duration than the slow playback, making the video playback smoother when switching from normal speed to slow speed.

[0240] In this embodiment of the invention, when executing a channel switching command and receiving the bitstream of the next channel, in order to avoid the black screen phenomenon caused by waiting for the screen to display, the first frame of the video in the bitstream can be played directly after receiving the bitstream, and then the video can be played in slow speed to achieve synchronization of audio and video in the bitstream.

[0241] In specific implementation, when the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is played directly, and the first video frame is played. The initial value of the system clock is determined by the timestamp of the first audio frame. The video is played slowly until the system clock increases to match the current timestamp of the video, at which point the video is played at normal speed.

[0242] Continue to refer to Figure 6h The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6hAs shown, at the time first_apts, the bitstream of the next channel is received, and the initial value of the system clock PCR first_pcr is determined by the value of the timestamp first_apts of the first frame a0 of the audio in the bitstream, where first_pcr = first_apts - buf_threshold;

[0243] In this embodiment of the invention, buf_threshold represents the buffer threshold.

[0244] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0245] During the period from first_apts to a1_apts, the audio is played directly, and the first frame v0 of the video is played (i.e., the first frame v0 of the video is output directly without referencing the system clock PCR).

[0246] From a1_apts to v i During the _apts period, continue playing the audio from frame a1 to frame a+2. t+1 Simultaneously, by controlling the display duration of the video, the second frame v1 to the (m+1)th frame v of the video are... m To slow down the playback speed, for example, the display duration of video frames can be changed from 50ms to 60ms, thus achieving slow playback of the video. i _apts represents the current display timestamp of the video;

[0247] In v i At time _apts, the current timestamp of the system clock PCR is the same as the current display timestamp of the video. i _apts, plays the remaining frames of the video at normal speed, and continues playing the remaining frames of the audio, i.e., plays the (m+2)th frame of the video at normal speed. m+1 Up to frame n+1 of the video v n Simultaneously play the audio at frame a (t+2). t+1 Up to frame n+1 a n .

[0248] It should be noted that during slow-motion playback of the video, the audio and video are not completely synchronized. The audio and video can only achieve synchronized playback when the current display timestamp of the video matches the current display timestamp of the audio.

[0249] It should be noted that the above eight synchronization strategies are only illustrative examples. In practical applications, other synchronization strategies can also be used to process audio and video in the bitstream where the distribution position interval of the first frame audio and the first frame video is less than a preset threshold and the display timestamp of the first frame audio is less than the display timestamp of the first frame video, so as to achieve synchronization of the audio and video.

[0250] Scenario 5: When the interval between the first audio frame and the first video frame in the bitstream is greater than a preset threshold, and the first video frame is positioned before the first audio frame in the bitstream, the timestamp of the first audio frame is greater than the timestamp of the first video frame.

[0251] In this embodiment of the invention, when the interval between the distribution positions of the first frame audio and the first frame video in the bitstream is greater than a preset threshold, and the distribution position of the first frame video in the bitstream is before the first frame audio, and the source file whose first frame display timestamp is greater than the first frame display timestamp of the video, the audio and video can be processed according to the following two synchronization strategies.

[0252] Synchronization Strategy 1: When the timestamp of the first frame of the audio is greater than the timestamp of the first frame of the video, the video is played directly, and empty pulse-coded data is added to the audio. The initial value of the system clock is determined by the timestamp of the first frame of the video. When the system clock increases to match the timestamp of the first frame of the audio, the audio is played.

[0253] Continue to refer to Figure 7a The diagram shown is a schematic representation of an audio and video synchronization method in an embodiment of the present invention, where the timestamp of the first audio frame is greater than the timestamp of the first video frame. Figure 7a As shown, at the first_vpts time, the video can be played directly, and the initial value of the system clock PCR, first_pcr, is determined by the timestamp first_vpts of the first frame of the video, where first_pcr = first_vpts - buf_threshold.

[0254] In this embodiment of the invention, buf_threshold represents the buffer threshold.

[0255] As a specific example, buf_threshold can be equal to 300ms*90, where 300ms represents the buffer time and 90 represents the timestamp conversion unit.

[0256] During the period from first_vpts to first_apts, the first frame v0, the second frame v1, and the third frame v2 of the video are played. Since the timestamp first_apts of the first audio frame is greater than the current timestamp of the system clock PCR, empty pulse-coded data can be added to the audio, as shown in SF in the figure, that is, a silent frame (SF) is added before the first audio frame v0.

[0257] At time first_apts, after playing the empty burst encoded data added to the audio, the current timestamp of the system clock PCR is the same as the display timestamp first_apts of the first frame a0 of the audio. Then, the audio and video are played from the first frame a0 to the (n+1)th frame a. n .

[0258] In this embodiment of the invention, to achieve synchronized playback of audio and video with added empty pulse-coded data at the timestamp of the first frame of the audio, the number of frames in which empty pulse-coded data is added to the audio can be strictly controlled based on the timestamps of the first frame of the video (first_vpts), the timestamps of the first frame of the audio (first_apts), the sampling rate of the audio (samplate), and the number of channels (byte_width). Specifically, the number of frames in which empty pulse-coded data is added to the audio is as follows:

[0259] add-size=((first_apts-first_vpts) / 90)*(samplate / 1000)*byte_width;

[0260] Where (first_apts-first_vpts) / 90 represents the time for adding empty pulse-coded data to the audio.

[0261] It should be noted that, in order to avoid data fluctuations in the link, some audio data needs to be buffered before the start of playback. Therefore, the initial value of the system clock PCR, first_pcr, can be less than the first frame display timestamp of the video, first_vpts.

[0262] Synchronization Strategy 2: When the timestamp of the first audio frame is greater than the timestamp of the first video frame, the first video frame is played directly, and the display state of the first video frame is maintained. The initial value of the system clock is determined by the timestamp of the first audio frame. The video is processed by dropping frames until the current timestamp of the video matches the timestamp of the first audio frame, at which point the audio and the video are played.

[0263] Reference Figure 7bThe diagram shown is a schematic representation of an audio and video synchronization method in an embodiment of the present invention, where the timestamp of the first audio frame is greater than the timestamp of the first video frame. Figure 7b As shown, at the first_vpts time, the first frame v0 of the video is played directly, and the display state of the first frame v0 of the video is maintained by controlling the refresh state of the second frame v1. For example, the second frame v1 of the video is not refreshed in order to maintain the display state of the first frame v0 of the video.

[0264] The initial value of the system clock PCR, first_pcr, is determined by the timestamp first_apts displayed in the first frame a0 of the audio. Specifically, first_pcr = first_apts - buf_threshold, where buf_threshold represents the buffer threshold.

[0265] As a specific example, buf_threshold can be equal to 150ms*90, where 150ms represents the buffer time and 90 represents the timestamp conversion unit;

[0266] During the time interval from first_vpts to first_apts, the first frame v0 of the video is played and its display state is maintained, while the second frame v1 to the m-th frame v... m-1 Frame dropping is performed, as shown in DF in the figure. Since the timestamp first_apts of the first audio frame a0 is greater than the timestamp of the system clock PCR, the audio is not played.

[0267] At time first_apts, the display timestamp v1_vpts of the first video frame v1 is the same as the display timestamp first_apts of the first audio frame a0, and the audio from the first frame a0 to the (n+1)th frame a is played. n Simultaneously play the (m+1)th frame v of the video. m Up to frame n+1 v n .

[0268] In this embodiment of the invention, to achieve synchronized playback of the video and audio after frame drops at the timestamp displayed on the first frame of the audio, the number of dropped video frames can be controlled based on the timestamps of the first video frame (first_vpts), the timestamps of the first audio frame (first_apts), the sampling rate of the audio (samplate), and the number of channels (byte_width). Specifically, the number of dropped audio frames is as follows:

[0269] drop-size=((first_apts-first_tpts) / 90)*(samplate / 1000)*byte_width;

[0270] Where (first_apts-first_vpts) / 90 represents the time of the video frame loss.

[0271] Scenario 6: When the interval between the first audio frame and the first video frame in the bitstream is greater than a preset threshold, and the first video frame is positioned before the first audio frame in the bitstream, the timestamp of the first audio frame is equal to the timestamp of the first video frame.

[0272] In this embodiment of the invention, for the situation described in scenario six, when the timestamp of the first frame of the audio is equal to the timestamp of the first frame of the video, the audio and the video are played directly.

[0273] Reference Figure 8 The diagram shown is a schematic representation of an audio and video synchronization method in an embodiment of the present invention, where the timestamp of the first audio frame is equal to the timestamp of the first video frame. Figure 8 As shown, the audio and video buffers some data, and at the first_apts time, the first frame a0 to the (n+1)th frame a of the audio are played directly. n+1 and the first frame v0 to the (n+1)th frame v of the video n .

[0274] In embodiments of the present invention, such as Figure 3 As shown in neutron diagram (b), when the interval between the distribution positions of the first frame audio and the first frame video in the bitstream is greater than a preset threshold, and the distribution position of the first frame audio in the bitstream is before the first frame video, in order to synchronize the audio and video, the audio can be played, and the current display timestamp of the audio and the display timestamp of the first frame of the video can be extracted. Based on the current display timestamp of the audio and the display timestamp of the first frame of the video, the video is processed using a corresponding synchronization strategy so that the processed audio and the video are synchronized.

[0275] Specifically, when the timestamp of the first frame of the video is equal to the current timestamp of the audio, the video is played directly.

[0276] In this embodiment of the invention, when it is determined that the current display timestamp of the audio and the display timestamp of the first frame of the current video are not aligned, and the display timestamp of the first frame of the video is greater than the current display timestamp of the audio, it indicates that the current video frame is behind the first frame of the audio. At this time, the time difference between the time corresponding to the display timestamp of the first frame of the video and the time corresponding to the current timestamp of the audio is determined, and the video is played after the time difference to achieve synchronized playback of audio and video.

[0277] When it is determined that the current display timestamp of the audio and the display timestamp of the first frame of the current video are not aligned, and the display timestamp of the first frame of the video is less than the current display timestamp of the audio, the video is subjected to frame dropping until the display timestamp of the video matches the display timestamp of the audio, at which point the video is played.

[0278] In specific implementation, the number of frames dropped in the video is:

[0279] drop_size=((ai_apts-first_vpts) / 90)*(samplate / 1000)*byte_width;

[0280] Where samplate represents the audio sampling rate, byte_width represents the number of channels, ai_apts represents the current display timestamp of the audio, and i is an integer greater than or equal to 0.

[0281] In this embodiment of the invention, when the interval between the distribution positions of the first frame audio and the first frame video in the bitstream is greater than a preset threshold, and the distribution position of the first frame video in the bitstream precedes that of the first frame audio, such as Figure 3 As shown in neutron diagram (b), the display timestamp of the first frame of the video can be extracted first, and then the display timestamp of the first frame of the audio can be extracted. Based on the display timestamps of the first frame of the video and the first frame of the audio, the video and the audio are processed using corresponding synchronization strategies to synchronize the video and the audio.

[0282] The timestamp of the first frame of the video and the timestamp of the first frame of the audio are generated based on the same system clock.

[0283] In specific implementation, the timestamp of the first audio frame and the timestamp of the first video frame are related as follows: the timestamp of the first audio frame is equal to the timestamp of the first video frame; the timestamp of the first audio frame is less than the timestamp of the first video frame; and the timestamp of the first audio frame is greater than the timestamp of the first video frame.

[0284] The audio and video synchronization strategies in the above situations are described below.

[0285] Scenario 7: When the interval between the distribution positions of the first frame audio and the first frame video in the bitstream is greater than a preset threshold, and the distribution position of the first frame audio in the bitstream is before the first frame video, and the display timestamp of the first frame audio is less than the display timestamp of the first frame video.

[0286] In this embodiment of the invention, for source files where the distribution position interval between the first frame audio and the first frame video in the bitstream is greater than a preset threshold, and the distribution position of the first frame audio in the bitstream is before the first frame video, and the display timestamp of the first frame audio is less than the display timestamp of the first frame video, this embodiment of the invention provides eight synchronization strategies to synchronize the audio and the video.

[0287] In specific implementation, when the timestamp of the first frame of the audio is less than the timestamp of the first frame of the video, the audio is played directly, and the initial value of the system clock is determined by the timestamp of the first frame of the audio; when the system clock increases to match the timestamp of the first frame of the video, the video is played.

[0288] Continue to refer to Figure 6a The diagram shown is a schematic representation of an audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6a As shown, at the first_apts time, the audio can be played directly, and the initial value of the system clock PCR, first_pcr, is determined by the timestamp first_apts of the first frame of the audio, where first_apts = first_apts - buf_threshold.

[0289] In this embodiment of the invention, buf_threshold represents the buffer threshold.

[0290] As a specific example, buf_threshold can be equal to 300ms * 90, where 300ms is the buffer time and 90 is the conversion unit for timestamps. During the period from first_apts to first_vpts, the first frame a0 and the second frame a1 of the audio are played. Since the timestamp first_vpts of the first frame v0 of the video is greater than the current timestamp of the system clock PCR, the video is not played.

[0291] At time first_vpts, since the current timestamp of the system clock PCR increases to be the same as the display timestamp first_vpts of the first frame v0 of the video, the video is played from the first frame v0 to the (n+1)th frame v. nMeanwhile, the audio continues to play from the third frame a2 to the (n+1)th frame a. n .

[0292] Since the initial value of the system clock is determined by the timestamp of the first audio frame, and the output of the first video frame is determined by the current timestamp of the system clock and the timestamp of the first audio frame, the audio display timestamp and the video display timestamp are both associated with the same system clock, and the video and the audio can be synchronized at the first_vpts time.

[0293] It should be noted that, in order to avoid data fluctuations in the link, some audio data needs to be buffered before playback starts. Therefore, the initial value of the system clock PCR, frist_pcr, can be less than the first frame display timestamp of the audio, first_apts.

[0294] In practical implementation, for example, during channel switching, audio can be played directly, and the video can be played only when the system clock matches the timestamp displayed in the first frame of the video, thus achieving synchronization between audio and video.

[0295] However, in such Figure 6a During the first_apts to first_vpts period shown, there will be a period of black screen due to no video being played, which affects the visual experience.

[0296] To address this, the first frame of the video can be played directly and its display status maintained to eliminate the black screen issue. Specifically, when the timestamp of the first audio frame is less than the timestamp of the first video frame, both the audio and the first video frame are played directly, and the initial value of the system clock is determined by the timestamp of the first audio frame. The display status of the first video frame is maintained until the system clock increases to match the timestamp of the second video frame, at which point the remaining frames of the video are played.

[0297] Continue to refer to Figure 6b The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6b As shown, at the first_apts time, the audio and the first frame v0 of the video are played directly (i.e., the first frame v0 of the video is output directly without referencing the system clock PCR), and the initial value of the system clock PCR, first_pcr, is determined by the timestamp first_apts of the first audio frame, where first_pcr = first_apts - buf_threshold.

[0298] In this embodiment of the invention, buf_threshold represents the buffer threshold.

[0299] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0300] During the period from first_apts to v1_apts, the first frame a0, the second frame a1, and the third frame a2 of the audio are played. By controlling the refresh state of the second frame v1 of the video, the display state of the first frame v0 of the video is maintained. For example, the second frame v1 of the video can be not refreshed in order to maintain the display state of the first frame v0 of the video.

[0301] At time v1_apts, since the timestamp of the second frame v1 of the video is the same as the current timestamp of the system clock PCR, the second frame v1 of the video is played to the (n+1)th frame v. n Meanwhile, the audio continues to play from frame a3 to frame a (n+1) of the fourth frame. n .

[0302] Since the initial value of the system clock is determined by the display timestamp of the first audio frame, and the output of the second video frame is determined by the current display timestamp of the system clock and the display timestamp of the first audio frame, the audio display timestamp and the video display timestamp are both associated with the same system clock, and the video and the audio can be synchronized at v1_apts time.

[0303] By employing the two synchronization strategies described above, audio and video can be synchronized through appropriate synchronization processing. Furthermore, since no frame dropping is performed during playback, complete audio and video information can be preserved, improving visual integrity.

[0304] In this embodiment of the invention, during the playback of the source file, the audio can be muted and the video can be processed accordingly to achieve audio and video synchronization.

[0305] In a specific implementation, when the timestamp of the first frame of the audio is less than the timestamp of the first frame of the video, the audio is played silently, and the initial value of the system clock is determined by the timestamp of the first frame of the audio; when the system clock increases to match the timestamp of the first frame of the video, the video and the audio are played simultaneously.

[0306] Continue to refer to Figure 6cThe diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6c As shown, due to the muting of the audio, the first_pcr of the system clock PCR is very close to the first_apts of the first frame a0 of the audio.

[0307] Specifically, first_pcr = first_apts - buf_threshold, where buf_threshold represents the buffer threshold.

[0308] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0309] During the time period from first_apts to first_vpts, the audio from the first frame a0 to the mth frame a... m-1 Playback is muted, as shown in the "mute" diagram. Since the display timestamp `first_vpts` of the first frame (v0) of the video is greater than the timestamp of the system clock `PCR`, the video is not played.

[0310] At time first_vpts, since the current timestamp of the system clock PCR increases to match the display timestamp first_vpts of the first frame v0 of the video, the (m+1)th frame a of the audio is played. m Up to frame n+1 a n Simultaneously playing the first frame v0 to the (n+1)th frame v of the video. n .

[0311] In this embodiment of the invention, to achieve synchronized output of the audio and video at the timestamp displayed in the first frame of the video, the number of frames in which the audio is muted can be controlled based on the timestamps of the first video frame (first_vpts), the timestamps of the first audio frame (first_apts), the audio sampling rate (samplate), and the number of channels (byte_width). Specifically, the number of frames in which the audio is played muted is:

[0312] mute-size=((first_vpts-first_apts) / 90)*(samplate / 1000)*byte_width;

[0313] Where (first_vpts-first_apts) / 90 represents the time during which the audio is muted.

[0314] In this embodiment of the invention, to eliminate the black screen phenomenon at the start of playback, the first frame of the video can be output directly without referencing the system clock PCR. Specifically, when the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is muted and the first video frame is played, with the initial value of the system clock determined by the timestamp of the first audio frame; the display state of the first video frame is maintained until the system clock increases to be greater than or equal to the timestamp of the second video frame, at which point the audio and video are played.

[0315] Continue to refer to Figure 6d The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6d As shown, due to the muting of the audio, the first_pcr of the system clock PCR is very close to the first_apts of the first frame a0 of the audio.

[0316] Specifically, first_pcr = first_apts - buf_threshold, where buf_threshold represents the buffer threshold.

[0317] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0318] During the time period from first_apts to v1_vpts, the audio from the first frame a0 to the mth frame a... m-1 Mute playback is performed as shown in the figure, and the first frame v0 of the video is played (that is, the first frame v0 of the video is output directly without referencing the system clock PCR). At the same time, the refresh state of the second frame v1 of the video is controlled to maintain the display state of the first frame v0 of the video. For example, the second frame v1 of the video can be not refreshed in order to maintain the display state of the first frame v0 of the video.

[0319] At time v1_vpts, since the current timestamp of the system clock PCR increases to match the display timestamp of the second frame v1 of the video, the (m+1)th frame a of the audio is played. m Up to frame n+1 a n And play the second frame v1 to the (n+1)th frame v of the video. n .

[0320] Since the initial value of the system clock is determined by the display timestamp of the first audio frame, and the output of the second video frame is determined by the current display timestamp of the system clock and the display timestamp of the first audio frame, the audio display timestamp and the video display timestamp are both associated with the same system clock, and the video and the audio can be synchronized at v1_vpts.

[0321] In practice, directly performing frame drop (DF) processing on the audio will complete the audio data processing faster than outputting a silence frame. Therefore, embodiments of the present invention can also shorten the time required for audio and video synchronization by performing frame drop processing on the audio.

[0322] Specifically, when the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is dropped, and the initial value of the system clock is determined by the timestamp of the first audio frame after the audio is dropped; when the system clock increases to match the timestamp of the first video frame, the audio and the video are played simultaneously.

[0323] Continue to refer to Figure 6e The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6e As shown, by performing frame dropping processing on the audio, the current display timestamp of the audio can be made the same as the display timestamp of the first frame of the video, first_vpts.

[0324] Specifically, by controlling the number of frames dropped in the audio, the first audio 'a' after the audio frame drop can be... m The corresponding display timestamp is the same as the display timestamp first_vpts of the first frame of the video, and is generated by the first audio 'a' after the audio frame is dropped. m The corresponding display timestamp a m _vpts determines the initial value of the system clock PCR, first_vpt, as follows: Figure 6e As shown, a m _vpts=first_vpt, first_pcr=a m _vpts-buf_threshold, where buf_threshold represents the buffer threshold;

[0325] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0326] During the time period from first_apts to first_vpts, the first frame a0 to the mth frame a of the audio... m-1 Frame dropping is performed, as shown in DF in the figure. Since the timestamp first_vpts of the first frame v0 of the video is greater than the initial value first_pcr of the system clock PCR, the video is not played.

[0327] At time first_vpts, the audio frame dropping process is completed. The display timestamp first_vpts of the first video frame v0 is equal to the current timestamp of the system clock PCR, which is equal to the display timestamp a corresponding to the first audio am after the audio frame dropping. m _vpts, simultaneously playing the video and audio, that is, playing the first frame v0 to the (n+1)th frame v of the video. n Simultaneously play the (m+1)th frame a of the audio. m Up to frame n+1 a n .

[0328] In this embodiment of the invention, to achieve synchronized output of audio and video after frame dropping at the timestamp displayed in the first frame of the video, the number of audio frames dropped can be controlled based on the timestamps of the first video frame (first_vpts), the timestamps of the first audio frame (first_apts), the audio sampling rate (samplate), and the number of channels (byte_width). Specifically, the number of audio frames dropped is as follows:

[0329] drop-size=((first_vpts-first_apts) / 90)*(samplate / 1000)*byte_width;

[0330] Where (first_vpts-first_apts) / 90 represents the time for dropping frames in the audio.

[0331] In this embodiment of the invention, to eliminate the black screen phenomenon at the start of playback, the first frame of the video can be output directly without referencing the system clock PCR. Specifically, when the display timestamp of the first audio frame is less than the display timestamp of the first video frame, the audio is dropped, and the first video frame is played. The initial value of the system clock is determined by the display timestamp corresponding to the first audio frame after the audio frame drop. The display state of the first video frame is maintained until the system clock increases to be greater than or equal to the display timestamp of the second video frame, at which point the audio and the video are played simultaneously.

[0332] Continue to refer to Figure 6fThe diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6f As shown, by performing frame dropping processing on the audio, the display timestamp of the audio can be made very close to the initial value first_pcr of the system clock PCR;

[0333] Specifically, first_pcr = a m _apts-buf_threshold, where buf_threshold represents the buffer threshold.

[0334] As a specific example, buf_threshold can be equal to 150ms*90, where 150ms is the buffer time and 90 represents the timestamp conversion unit.

[0335] During the time interval from first_apts to v1_apts, the first frame a0 to the mth frame a of the audio... m-1 Frame dropping is performed, and the first frame v0 of the video is played (i.e., the first frame v0 of the video is output directly without referencing the system clock PCR), while the refresh state of the second frame v1 of the video is controlled to maintain the display state of the first frame v0 of the video; for example, the second frame v1 of the video can be not refreshed in order to maintain the display state of the first frame v0 of the video.

[0336] At time v1_apts, since the current timestamp of the system clock PCR increases to be greater than or equal to the display timestamp of the second frame v1 of the video, the second frame v1 of the video is played up to the (n+1)th frame v. n Meanwhile, continue playing the (m+1)th frame a of the audio. m Up to frame n+1 a n .

[0337] In this embodiment of the invention, to achieve synchronized playback of the audio and video after frame dropping at the timestamp displayed on the first frame of the video, the number of frames dropped for the audio can be controlled based on the timestamps of the first video frame (first_vpts), the timestamps of the first audio frame (first_apts), the sampling rate of the audio (samplate), and the number of channels (byte_width). Specifically, the number of frames dropped for the audio is as follows:

[0338] drop-size=((first_vpts-first_apts) / 90)*(samplate / 1000)*byte_width;

[0339] Where (first_vpts-first_apts) / 90 represents the time for dropping frames in the audio.

[0340] In practice, when playing a video source file, the playback link may contain a lot of audio decoded pulse code data that has already been buffered. If played directly, the difference between the display timestamps of the audio and video will become larger and larger, resulting in audio and video being out of sync. Therefore, the audio and video can be synchronized by discarding the buffered audio data in the playback link.

[0341] Specifically, when the timestamp of the first audio frame is less than the timestamp of the first video frame, the number of frames in the audio buffer is obtained, and the number of dropped audio frames is calculated. The number of frames in the audio buffer is compared with the number of dropped audio frames. If the number of dropped audio frames is greater than the number of audio frames in the audio buffer, the number of dropped audio frames is processed by dropping frames in the audio buffer, and the first video frame is played. The initial value of the system clock is determined by the timestamp corresponding to the first audio after the dropped audio frames. The audio is played at the timestamp corresponding to the first audio after the dropped audio frames. Furthermore, when the system clock is inconsistent with the current timestamp of the video, the video is played in slow motion until the system clock is consistent with the current timestamp of the video, at which point the video is played at normal speed.

[0342] In this embodiment of the invention, the frame number bf of the audio buffer can be obtained, and the frame number df of the audio frame drops can be calculated based on the display timestamp first_vpts of the first video frame, the display timestamp first_apts of the first audio frame, the sampling rate of the audio, and the number of channels byte_width. The frame number bf of the audio buffer is compared with the frame number df of the audio frame drops. If bf is greater than df, it can be processed according to... Figure 6f The diagram shown illustrates synchronized audio and video playback. The corresponding processing of the audio and video is not detailed here. If bf is less than df, the number of frames df in the audio buffer of the playback link can be discarded, and then proceeded according to... Figure 6g The diagram illustrates the audio and video synchronization method, which performs corresponding processing on the audio and video.

[0343] Continue to refer to Figure 6g The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6g As shown, in first_apts to a m-1 During the _apts period, the number of frames a0 to a10 of the audio buffer. m-2 Perform frame dropping processing while playing the first frame v0 of the video (i.e., the first frame v0 of the video is output directly without referencing the system clock PCR);

[0344] In a m-1 _apts to a m During the _apts period, continue processing the audio buffer for the remaining frames a. m-1 Frame dropping is performed, and the video is played in slow motion by controlling the video display duration. For example, the display duration of the video frame can be set from the original 50ms to 60ms to achieve slow playback of the video.

[0345] In a m At time _apts, the frame dropping processing of the audio is completed, and the (m+1)th frame a of the audio is played. m And the first audio a after the audio frame was dropped. m The corresponding display timestamp determines the initial value of the system clock PCR, first_pcr, where first_pcr = a m _apts-buf_threshold, continue playing the video in slow motion;

[0346] In this embodiment of the invention, buf_threshold represents the buffer threshold.

[0347] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0348] It should be noted that since the number of frames in the audio buffer (bf) is less than the number of dropped audio frames (df), therefore, in a m At the _apts time, the audio and video are not synchronized, and the current display timestamp of the audio is slightly smaller than the display timestamp of the video.

[0349] In a m _apts to v i During the _apts period, since the current display timestamp of the video is greater than the initial value first_pcr of the system clock PCR, the audio plays normally, and the video continues to play slowly, where i is an integer greater than 1 and less than n;

[0350] In v i At time _apts, the current timestamp of the system clock PCR is equal to the current display timestamp v of the video. i _apts, plays the video at normal speed, that is, plays the m-th frame of the video at normal speed. m Up to frame n v n .

[0351] Since the initial value of the system clock is determined by the display timestamp corresponding to the first audio frame after the audio frame is dropped, and the timestamp of the video playing at normal speed is determined by the display timestamp of the first audio frame and the current display timestamp of the system clock, both the audio display timestamp and the video display timestamp are associated with the same system clock. Therefore, the video and audio can play at normal speed. i _apts is synchronized at all times.

[0352] It should be noted that because it takes some time for a video to switch from normal speed to slow speed, the first frame of the video will be displayed for a longer duration than the slow playback, making the video playback smoother when switching from normal speed to slow speed.

[0353] In this embodiment of the invention, when executing a channel switching command and receiving the bitstream of the next channel, in order to avoid the black screen phenomenon caused by waiting for the screen to display, the first frame of the video in the bitstream can be played directly after receiving the bitstream, and then the video can be played in slow speed to achieve synchronization of audio and video in the bitstream.

[0354] In specific implementation, when the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is played directly, and the first video frame is played. The initial value of the system clock is determined by the timestamp of the first audio frame. The video is played slowly until the system clock increases to match the current timestamp of the video, at which point the video is played at normal speed.

[0355] Continue to refer to Figure 6h The diagram shown illustrates another audio and video synchronization method in an embodiment of the present invention where the timestamp of the first audio frame is less than the timestamp of the first video frame. Figure 6h As shown, at the time first_apts, the bitstream of the next channel is received, and the initial value of the system clock PCR first_pcr is determined by the value of the timestamp first_apts of the first frame a0 of the audio in the bitstream, where first_pcr = first_apts - buf_threshold;

[0356] In this embodiment of the invention, buf_threshold represents the buffer threshold.

[0357] As a specific example, buf_threshold can be equal to 150ms * 90, where 150ms is the buffer time and 90 is the conversion unit for timestamps.

[0358] During the period from first_apts to a1_apts, the audio is played directly, and the first frame v0 of the video is played (i.e., the first frame v0 of the video is output directly without referencing the system clock PCR).

[0359] From a1_apts to v i During the _apts period, continue playing the audio from frame a1 to frame a+2. t+1 Simultaneously, by controlling the display duration of the video, the second frame v1 to the (m+1)th frame v of the video are... m To slow down the playback speed, for example, the display duration of video frames can be changed from 50ms to 60ms, thus achieving slow playback of the video. i _apts represents the current display timestamp of the video;

[0360] In v i At time _apts, the current timestamp of the system clock PCR is the same as the current display timestamp of the video. i _apts, plays the remaining frames of the video at normal speed, and continues playing the remaining frames of the audio, i.e., plays the (m+2)th frame of the video at normal speed. m+1 Up to frame n+1 of the video v n Simultaneously play the audio at frame a (t+2). t+1 Up to frame n+1 a n .

[0361] It should be noted that during slow-motion playback of the video, the audio and video are not completely synchronized. The audio and video can only achieve synchronized playback when the current display timestamp of the video matches the current display timestamp of the audio.

[0362] It should be noted that the above eight synchronization strategies are only illustrative examples. In practical applications, other synchronization strategies can also be used to process audio and video in the bitstream where the distribution position interval of the first frame audio and the first frame video is less than a preset threshold and the display timestamp of the first frame audio is less than the display timestamp of the first frame video, so as to achieve synchronization of the audio and video.

[0363] Scenario 8: When the interval between the first frame audio and the first frame video in the bitstream is greater than a preset threshold, and the first frame audio is positioned before the first frame video in the bitstream, the timestamp of the first frame audio is greater than the timestamp of the first frame video.

[0364] In this embodiment of the invention, when the interval between the distribution positions of the first frame audio and the first frame video in the bitstream is greater than a preset threshold, and the distribution position of the first frame video in the bitstream is before the first frame audio, and the source file whose first frame display timestamp is greater than the first frame display timestamp of the video, the audio and video can be processed according to the following two synchronization strategies.

[0365] Synchronization Strategy 1: When the timestamp of the first frame of the audio is greater than the timestamp of the first frame of the video, the video is played directly, and empty pulse-coded data is added to the audio. The initial value of the system clock is determined by the timestamp of the first frame of the video. When the system clock increases to match the timestamp of the first frame of the audio, the audio is played.

[0366] Continue to refer to Figure 7a The diagram shown is a schematic representation of an audio and video synchronization method in an embodiment of the present invention, where the timestamp of the first audio frame is greater than the timestamp of the first video frame. Figure 7a As shown, at the first_vpts time, the video can be played directly, and the initial value of the system clock PCR, first_pcr, is determined by the timestamp first_vpts of the first frame of the video, where first_pcr = first_vpts - buf_threshold.

[0367] In this embodiment of the invention, buf_threshold represents the buffer threshold.

[0368] As a specific example, buf_threshold can be equal to 300ms*90, where 300ms represents the buffer time and 90 represents the timestamp conversion unit.

[0369] During the period from first_vpts to first_apts, the first frame v0, the second frame v1, and the third frame v2 of the video are played. Since the timestamp first_apts of the first audio frame is greater than the current timestamp of the system clock PCR, empty pulse-coded data can be added to the audio, as shown in SF in the figure, that is, a silent frame (SF) is added before the first audio frame v0.

[0370] At time first_apts, after playing the empty burst encoded data added to the audio, the current timestamp of the system clock PCR is the same as the display timestamp first_apts of the first frame a0 of the audio. Then, the audio and video are played from the first frame a0 to the (n+1)th frame a. n .

[0371] In this embodiment of the invention, to achieve synchronized playback of audio and video with added empty pulse-coded data at the timestamp of the first frame of the audio, the number of frames in which empty pulse-coded data is added to the audio can be strictly controlled based on the timestamps of the first frame of the video (first_vpts), the timestamps of the first frame of the audio (first_apts), the sampling rate of the audio (samplate), and the number of channels (byte_width). Specifically, the number of frames in which empty pulse-coded data is added to the audio is as follows:

[0372] add-size=((first_apts-first_vpts) / 90)*(samplate / 1000)*byte_width;

[0373] Where (first_apts-first_vpts) / 90 represents the time for adding empty pulse-coded data to the audio.

[0374] It should be noted that, in order to avoid data fluctuations in the link, some audio data needs to be buffered before the start of playback. Therefore, the initial value of the system clock PCR, first_pcr, can be less than the first frame display timestamp of the video, first_vpts.

[0375] Synchronization Strategy 2: When the timestamp of the first audio frame is greater than the timestamp of the first video frame, the first video frame is played directly, and the display state of the first video frame is maintained. The initial value of the system clock is determined by the timestamp of the first audio frame. The video is processed by dropping frames until the current timestamp of the video matches the timestamp of the first audio frame, at which point the audio and the video are played.

[0376] Reference Figure 7b The diagram shown is a schematic representation of an audio and video synchronization method in an embodiment of the present invention, where the timestamp of the first audio frame is greater than the timestamp of the first video frame. Figure 7b As shown, at the first_vpts time, the first frame v0 of the video is played directly, and the display state of the first frame v0 of the video is maintained by controlling the refresh state of the second frame v1. For example, the second frame v1 of the video is not refreshed in order to maintain the display state of the first frame v0 of the video.

[0377] The initial value of the system clock PCR, first_pcr, is determined by the timestamp first_apts displayed in the first frame a0 of the audio. Specifically, first_pcr = first_apts - buf_threshold, where buf_threshold represents the buffer threshold.

[0378] As a specific example, buf_threshold can be equal to 150ms*90, where 150ms represents the buffer time and 90 represents the timestamp conversion unit;

[0379] During the time interval from first_vpts to first_apts, the first frame v0 of the video is played and its display state is maintained, while the second frame v1 to the m-th frame v... m-1 Frame dropping is performed, as shown in DF in the figure. Since the timestamp first_apts of the first audio frame a0 is greater than the timestamp of the system clock PCR, the audio is not played.

[0380] At time first_apts, the display timestamp v1_vpts of the first video frame v1 is the same as the display timestamp first_apts of the first audio frame a0, and the audio from the first frame a0 to the (n+1)th frame a is played. n Simultaneously play the (m+1)th frame v of the video. m Up to frame n+1 v n .

[0381] In this embodiment of the invention, to achieve synchronized playback of the video and audio after frame drops at the timestamp displayed on the first frame of the audio, the number of dropped video frames can be controlled based on the timestamps of the first video frame (first_vpts), the timestamps of the first audio frame (first_apts), the sampling rate of the audio (samplate), and the number of channels (byte_width). Specifically, the number of dropped audio frames is as follows:

[0382] drop-size=((first_apts-first_tpts) / 90)*(samplate / 1000)*byte_width;

[0383] Where (first_apts-first_vpts) / 90 represents the time of the video frame loss.

[0384] Scenario 9: When the interval between the first audio frame and the first video frame in the bitstream is greater than a preset threshold, and the first audio frame is positioned before the first video frame in the bitstream, the timestamp of the first audio frame is equal to the timestamp of the first video frame.

[0385] In this embodiment of the invention, for the situation described in scenario nine, when the timestamp of the first frame of the audio is equal to the timestamp of the first frame of the video, the audio and the video are played directly.

[0386] Continue to refer to Figure 8The diagram shown is a schematic representation of an audio and video synchronization method in an embodiment of the present invention, where the timestamp of the first audio frame is equal to the timestamp of the first video frame. Figure 8 As shown, the audio and video buffers some data, and at the first_apts time, the first frame a0 to the (n+1)th frame a of the audio are played directly. n+1 and the first frame v0 to the (n+1)th frame v of the video n .

[0387] In practice, the source file may contain only one data format, such as audio or video. When playing the source file, it can be played directly without any processing.

[0388] By analyzing the distribution of the first audio frame and the first video frame in the bitstream, different playback scenarios can be determined. Corresponding synchronization strategies can be adopted to process the audio and video for different playback scenarios, ensuring compatibility with audio and video synchronization under different playback scenarios and thus meeting different playback requirements.

[0389] This invention also provides a playback device corresponding to the above-described audio and video synchronized playback method. The following detailed description, with reference to the accompanying drawings, will illustrate specific embodiments. It should be understood that the playback device described below can be considered as a functional module required to implement the audio and video synchronized playback method provided in this invention; the playback device described below corresponds to the content of the audio and video synchronized playback method described above.

[0390] Reference Figure 9 The diagram shown illustrates the structure of a playback device according to an embodiment of the present invention. In this embodiment, the playback device 90 is adapted to synchronously play audio and video. The playback device 90 may include:

[0391] The first extraction unit 91 is adapted to extract the distribution position of the first frame audio in the bitstream;

[0392] The second extraction unit 92 is adapted to extract the distribution position of the first frame video in the bitstream;

[0393] The processing unit 93 is adapted to process the audio and the video using a corresponding synchronization strategy based on the distribution location characteristics of the first frame audio and the first frame video in the bitstream, so that the audio and the video are synchronized.

[0394] In specific implementation, the processing unit 93 may adopt the steps of the audio and video synchronous playback method described in any of the foregoing embodiments. For details, please refer to the foregoing embodiments and corresponding figures, which will not be repeated here.

[0395] This invention also provides corresponding playback devices, such as... Figure 10 As shown, the playback device 100 may include a memory 101, a processor 102, a display 103, and an audio player 104, wherein:

[0396] The memory 101 stores computer instructions that can be executed on the processor 102; when the processor 102 executes the computer instructions, it performs the steps of the audio and video synchronous playback method described in any of the foregoing embodiments. For details, please refer to the foregoing embodiments and corresponding figures, which will not be repeated here.

[0397] The display 103 is adapted to play the video, and the audio player 104 is adapted to play the audio.

[0398] In specific implementations, the playback device 100 may further include an expansion interface 105, suitable for connecting with other devices to achieve data interaction. For example... Figure 10 The playback device 100 can be connected to other electronic display devices through the expansion interface 105 to play synchronized audio and video.

[0399] In a specific implementation, the memory 101, processor 102, display 103, audio player 104 and expansion interface 105 can communicate with each other via bus 106.

[0400] In this embodiment of the invention, the playback device 100 can be a smart TV, personal computer, tablet, or other electronic devices that include a display device.

[0401] In a specific implementation, the memory 101 may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device.

[0402] The processor 102 can be implemented by a processing chip such as a central processing unit (CPU), a graphics processing unit (GPU), or a field programmable gate array (FPGA), or by an application specific integrated circuit (ASIC) or one or more integrated circuits configured to implement the embodiments of this specification.

[0403] This invention also provides a computer-readable storage medium storing computer instructions, which, when executed, can perform the steps of the audio and video synchronous playback method described in any of the above embodiments of this invention. The computer-readable storage medium can be any suitable readable storage medium such as an optical disc, a hard disk drive, or a solid-state drive. The instructions stored on the computer-readable storage medium execute the steps of the audio and video synchronous playback method described in any of the above embodiments; specific details can be found in the above embodiments and will not be repeated here.

[0404] While the embodiments of the present invention have been disclosed above, the present invention is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the present invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A method for synchronized audio and video playback, characterized in that, include: Extract the distribution position of the first frame of audio in the bitstream, and extract the distribution position of the first frame of video in the bitstream; Based on the distribution relationship between the first audio frame and the first video frame in the bitstream, a corresponding synchronization strategy is applied to process the audio and video to synchronize them, including: When the interval between the first audio frame and the first video frame in the bitstream is greater than a preset threshold, and the first video frame is positioned before the first audio frame in the bitstream, the video is played, and the current display timestamp of the video and the display timestamp of the first audio frame are extracted. Based on the current display timestamp of the video and the display timestamp of the first audio frame, the audio is processed using a corresponding synchronization strategy to synchronize the processed audio with the video. This includes: when the display timestamp of the first audio frame is less than the current display timestamp of the video, the audio is dropped until the display timestamp of the audio matches the display timestamp of the video, at which point the audio is played. When the interval between the first audio frame and the first video frame in the bitstream is less than the preset threshold, the display timestamps of the first audio frame and the first video frame are extracted, and the audio and video are processed using corresponding synchronization strategies based on the first audio frame display timestamps and the first video frame display timestamps to synchronize the audio and video. This includes: when the first audio frame display timestamp is less than the first video frame display timestamp, obtaining the number of frames in the audio buffer and calculating the number of dropped audio frames; comparing the number of frames in the audio buffer with the number of dropped audio frames, and determining if the number of dropped audio frames is greater than the first audio frame display timestamp. When the number of audio frames in the audio buffer is reached, frame dropping is performed on the audio buffer, and the first frame of the video is played. The initial value of the system clock is determined by the display timestamp corresponding to the first audio after frame dropping. The audio is played at the display timestamp corresponding to the first audio after frame dropping. Furthermore, when the system clock is inconsistent with the current display timestamp of the video, the video is played in slow motion until the system clock is consistent with the current display timestamp of the video, at which point the video is played at normal speed. The display timestamp of the first audio frame and the display timestamp of the first video frame are generated based on the same system clock.

2. The method for synchronized audio and video playback according to claim 1, characterized in that, The step of processing the audio and video using corresponding synchronization strategies based on the display timestamp of the first frame of the audio and the display timestamp of the first frame of the video to synchronize the audio and the video further includes: When the timestamp of the first frame of the audio is greater than the timestamp of the first frame of the video, the video is played directly, empty pulse-coded data is added to the audio, and the initial value of the system clock is determined by the timestamp of the first frame of the video. The audio is played when the system clock increases to match the timestamp displayed in the first frame of the audio.

3. The method for synchronized audio and video playback according to claim 1, characterized in that, The step of processing the audio and video using corresponding synchronization strategies based on the display timestamp of the first frame of the audio and the display timestamp of the first frame of the video to synchronize the audio and the video further includes: When the timestamp of the first audio frame is greater than the timestamp of the first video frame, the first video frame is played directly, and the display state of the first video frame is maintained. The initial value of the system clock is determined by the timestamp of the first audio frame. The video is subjected to frame dropping until the current display timestamp of the video matches the display timestamp of the first frame of the audio, at which point the audio and the video are played.

4. The method for synchronized audio and video playback according to claim 1, characterized in that, The step of processing the audio and video using corresponding synchronization strategies based on the display timestamp of the first frame of the audio and the display timestamp of the first frame of the video to synchronize the audio and the video further includes: When the timestamp of the first frame of the audio is equal to the timestamp of the first frame of the video, the audio and the video are played directly.

5. The method for synchronized audio and video playback according to claim 1, characterized in that, The step of processing the audio using a corresponding synchronization strategy based on the current display timestamp of the video and the display timestamp of the first frame of the audio, so that the processed audio is synchronized with the video, includes: When the timestamp of the first frame of the audio is equal to the current timestamp of the video, the audio is played directly.

6. The method for synchronized audio and video playback according to claim 1, characterized in that, The step of processing the audio using a corresponding synchronization strategy based on the current display timestamp of the video and the display timestamp of the first frame of the audio, so that the processed audio is synchronized with the video, includes: When the timestamp of the first frame of the audio is greater than the current timestamp of the video, the audio is muted until the timestamp of the audio matches the timestamp of the video, at which point the audio is played.

7. The method for synchronized audio and video playback according to claim 1, characterized in that, The step of processing the audio and video using a corresponding synchronization strategy based on the distribution position relationship characteristics of the first frame audio and the first frame video in the bitstream to synchronize the audio and video includes: When the interval between the first audio frame and the first video frame in the bitstream is greater than a preset threshold, and the first video frame is positioned before the first audio frame in the bitstream, the display timestamp of the first video frame is extracted, and then the display timestamp of the first audio frame is extracted. Based on the display timestamps of the first video frame and the first audio frame, the video and the audio are processed using corresponding synchronization strategies to synchronize the video and the audio. The display timestamps of the first video frame and the first audio frame are generated based on the same system clock.

8. The method for synchronized audio and video playback according to claim 7, characterized in that, The step of processing the video and audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the video and the display timestamp of the first frame of the audio, so that the video and the audio are synchronized, includes: When the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is played directly, and the initial value of the system clock is determined by the timestamp of the first audio frame. The video is played when the system clock increments to match the timestamp displayed in the first frame of the video.

9. The method for synchronized audio and video playback according to claim 7, characterized in that, The step of processing the video and audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the video and the display timestamp of the first frame of the audio, so that the video and the audio are synchronized, includes: When the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio and the first video frame are played directly, and the initial value of the system clock is determined by the timestamp of the first audio frame. The display state of the first frame of the video is maintained until the system clock increases to match the display timestamp of the second frame of the video, at which point the remaining frames of the video are played.

10. The method for synchronized audio and video playback according to claim 7, characterized in that, The step of processing the video and audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the video and the display timestamp of the first frame of the audio, so that the video and the audio are synchronized, includes: When the timestamp of the first frame of the audio is less than the timestamp of the first frame of the video, the audio is played silently, and the initial value of the system clock is determined by the timestamp of the first frame of the audio. When the system clock increases to match the timestamp displayed in the first frame of the video, the video and the audio are played simultaneously.

11. The method for synchronized audio and video playback according to claim 7, characterized in that, The step of processing the video and audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the video and the display timestamp of the first frame of the audio, so that the video and the audio are synchronized, includes: When the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is muted and the first video frame is played. The initial value of the system clock is determined by the timestamp of the first audio frame. The display state of the first frame of the video is maintained until the system clock increases to a timestamp greater than or equal to the display timestamp of the second frame of the video, at which point the audio and the video are played.

12. The method for synchronized audio and video playback according to claim 7, characterized in that, The step of processing the video and audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the video and the display timestamp of the first frame of the audio, so that the video and the audio are synchronized, includes: When the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is dropped, and the initial value of the system clock is determined by the timestamp of the first audio frame after the audio is dropped. When the system clock increases to match the timestamp displayed in the first frame of the video, the audio and the video are played simultaneously.

13. The method for synchronized audio and video playback according to claim 7, characterized in that, The step of processing the video and audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the video and the display timestamp of the first frame of the audio, so that the video and the audio are synchronized, includes: When the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is dropped, and the first video frame is played. The initial value of the system clock is determined by the timestamp of the first audio frame after the audio is dropped. The display state of the first frame of the video is maintained until the system clock increases to a timestamp greater than or equal to the display timestamp of the second frame of the video, at which point the audio and the video are played simultaneously.

14. The method for synchronized audio and video playback according to claim 7, characterized in that, The step of processing the video and audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the video and the display timestamp of the first frame of the audio, so that the video and the audio are synchronized, includes: When the timestamp of the first audio frame is less than the timestamp of the first video frame, obtain the number of frames in the audio buffer and calculate the number of dropped audio frames; The number of frames in the audio buffer is compared with the number of audio frames dropped. If the number of audio frames dropped is greater than the number of audio frames in the audio buffer, the number of frames in the audio buffer is dropped and the first frame of the video is played. The initial value of the system clock is determined by the display timestamp corresponding to the first audio after the audio frames are dropped. Play the audio at the timestamp corresponding to the first audio frame after the audio frame loss; Furthermore, when the system clock is inconsistent with the current display timestamp of the video, the video is played in slow speed until the system clock is consistent with the current display timestamp of the video, at which point the video is played at normal speed.

15. The method for synchronized audio and video playback according to claim 7, characterized in that, The step of processing the video and audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the video and the display timestamp of the first frame of the audio, so that the video and the audio are synchronized, includes: When the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is played directly, and the first video frame is played. The initial value of the system clock is determined by the timestamp of the first audio frame. The video is played in slow motion until the system clock increments to match the current display timestamp of the video, at which point the video is played at normal speed.

16. The method for synchronized audio and video playback according to claim 7, characterized in that, The step of processing the video and audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the video and the display timestamp of the first frame of the audio, so that the video and the audio are synchronized, includes: When the timestamp of the first frame of the audio is greater than the timestamp of the first frame of the video, the video is played directly, empty pulse-coded data is added to the audio, and the initial value of the system clock is determined by the timestamp of the first frame of the video. The audio is played when the system clock increases to match the timestamp displayed in the first frame of the audio.

17. The method for synchronized audio and video playback according to claim 7, characterized in that, The step of processing the video and audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the video and the display timestamp of the first frame of the audio, so that the video and the audio are synchronized, includes: When the timestamp of the first audio frame is greater than the timestamp of the first video frame, the first video frame is played directly, and the display state of the first video frame is maintained. The initial value of the system clock is determined by the timestamp of the first audio frame. The video is subjected to frame dropping until the current display timestamp of the video matches the display timestamp of the first frame of the audio, at which point the audio and the video are played.

18. The method for synchronized audio and video playback according to claim 7, characterized in that, The step of processing the video and audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the video and the display timestamp of the first frame of the audio, so that the video and the audio are synchronized, includes: When the timestamp of the first frame of the audio is equal to the timestamp of the first frame of the video, the audio and the video are played directly.

19. The method for synchronized audio and video playback according to claim 1, characterized in that, The step of processing the audio and video using a corresponding synchronization strategy based on the distribution position relationship characteristics of the first frame audio and the first frame video in the bitstream to synchronize the audio and video includes: When the interval between the first frame audio and the first frame video in the bitstream is greater than a preset threshold, and the first frame audio is positioned before the first frame video in the bitstream, the audio is played, and the current display timestamp of the audio and the display timestamp of the first frame video are extracted. Based on the current display timestamp of the audio and the display timestamp of the first frame video, the video is processed using a corresponding synchronization strategy to synchronize the processed video with the audio.

20. The method for synchronized audio and video playback according to claim 19, characterized in that, The step of processing the video using a corresponding synchronization strategy based on the current display timestamp of the audio and the display timestamp of the first frame of the video, so that the processed video is synchronized with the audio, includes: When the timestamp of the first frame of the video is equal to the current timestamp of the audio, the video is played directly.

21. The method for synchronized audio and video playback according to claim 19, characterized in that, The step of processing the video using a corresponding synchronization strategy based on the current display timestamp of the audio and the display timestamp of the first frame of the video, so that the processed video is synchronized with the audio, includes: When the timestamp of the first frame of the video is greater than the current timestamp of the audio, the time difference between the time corresponding to the timestamp of the first frame of the video and the time corresponding to the current timestamp of the audio is determined, and the video is played after the time difference.

22. The method for synchronized audio and video playback according to claim 19, characterized in that, The step of processing the video using a corresponding synchronization strategy based on the current display timestamp of the audio and the display timestamp of the first frame of the video, so that the processed video is synchronized with the audio, includes: When the timestamp of the first frame of the video is less than the current timestamp of the audio, the video is dropped until the timestamp of the video matches the timestamp of the audio, at which point the video is played.

23. The method for synchronized audio and video playback according to claim 1, characterized in that, The step of processing the audio and video using a corresponding synchronization strategy based on the distribution position relationship characteristics of the first frame audio and the first frame video in the bitstream to synchronize the audio and video includes: When the interval between the first audio frame and the first video frame in the bitstream is greater than a preset threshold, and the first audio frame is positioned before the first video frame in the bitstream, the first audio frame display timestamp is extracted, and then the first video frame display timestamp is extracted. Based on the first audio frame display timestamp and the first video frame display timestamp, the video and the audio are processed using a corresponding synchronization strategy to synchronize the video and the audio. The first audio frame display timestamp and the first video frame display timestamp are generated based on the same system clock.

24. The method for synchronized audio and video playback according to claim 23, characterized in that, The step of processing the video and the audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the audio and the display timestamp of the first frame of the video, so that the video and the audio are synchronized, includes: When the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is played directly, and the initial value of the system clock is determined by the timestamp of the first audio frame. The video is played when the system clock increments to match the timestamp displayed in the first frame of the video.

25. The method for synchronized audio and video playback according to claim 23, characterized in that, The step of processing the video and the audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the audio and the display timestamp of the first frame of the video, so that the video and the audio are synchronized, includes: When the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio and the first video frame are played directly, and the initial value of the system clock is determined by the timestamp of the first audio frame. The display state of the first frame of the video is maintained until the system clock increases to match the display timestamp of the second frame of the video, at which point the remaining frames of the video are played.

26. The method for synchronized audio and video playback according to claim 23, characterized in that, The step of processing the video and the audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the audio and the display timestamp of the first frame of the video, so that the video and the audio are synchronized, includes: When the timestamp of the first frame of the audio is less than the timestamp of the first frame of the video, the audio is played silently, and the initial value of the system clock is determined by the timestamp of the first frame of the audio. When the system clock increases to match the timestamp displayed in the first frame of the video, the video and the audio are played simultaneously.

27. The method for synchronized audio and video playback according to claim 23, characterized in that, The step of processing the video and the audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the audio and the display timestamp of the first frame of the video, so that the video and the audio are synchronized, includes: When the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is muted and the first video frame is played. The initial value of the system clock is determined by the timestamp of the first audio frame. The display state of the first frame of the video is maintained until the system clock increases to a timestamp greater than or equal to the display timestamp of the second frame of the video, at which point the audio and the video are played.

28. The method for synchronized audio and video playback according to claim 23, characterized in that, The step of processing the video and the audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the audio and the display timestamp of the first frame of the video, so that the video and the audio are synchronized, includes: When the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is dropped, and the initial value of the system clock is determined by the timestamp of the first audio frame after the audio is dropped. When the system clock increases to match the timestamp displayed in the first frame of the video, the audio and the video are played simultaneously.

29. The method for synchronized audio and video playback according to claim 23, characterized in that, The step of processing the video and the audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the audio and the display timestamp of the first frame of the video, so that the video and the audio are synchronized, includes: When the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is dropped, and the first video frame is played. The initial value of the system clock is determined by the timestamp of the first audio frame after the audio is dropped. The display state of the first frame of the video is maintained until the system clock increases to a timestamp greater than or equal to the display timestamp of the second frame of the video, at which point the audio and the video are played simultaneously.

30. The method for synchronized audio and video playback according to claim 23, characterized in that, The step of processing the video and the audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the audio and the display timestamp of the first frame of the video, so that the video and the audio are synchronized, includes: When the timestamp of the first audio frame is less than the timestamp of the first video frame, obtain the number of frames in the audio buffer and calculate the number of dropped audio frames; The number of frames in the audio buffer is compared with the number of audio frames dropped. If the number of audio frames dropped is greater than the number of audio frames in the audio buffer, the number of frames in the audio buffer is dropped and the first frame of the video is played. The initial value of the system clock is determined by the display timestamp corresponding to the first audio after the audio frames are dropped. Play the audio at the timestamp corresponding to the first audio frame after the audio frame loss; Furthermore, when the system clock is inconsistent with the current display timestamp of the video, the video is played in slow speed until the system clock is consistent with the current display timestamp of the video, at which point the video is played at normal speed.

31. The method for synchronized audio and video playback according to claim 23, characterized in that, The step of processing the video and the audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the audio and the display timestamp of the first frame of the video, so that the video and the audio are synchronized, includes: When the timestamp of the first audio frame is less than the timestamp of the first video frame, the audio is played directly, and the first video frame is played. The initial value of the system clock is determined by the timestamp of the first audio frame. The video is played in slow motion until the system clock increments to match the current display timestamp of the video, at which point the video is played at normal speed.

32. The method for synchronized audio and video playback according to claim 23, characterized in that, The step of processing the video and the audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the audio and the display timestamp of the first frame of the video, so that the video and the audio are synchronized, includes: When the timestamp of the first frame of the audio is greater than the timestamp of the first frame of the video, the video is played directly, empty pulse-coded data is added to the audio, and the initial value of the system clock is determined by the timestamp of the first frame of the video. The audio is played when the system clock increases to match the timestamp displayed in the first frame of the audio.

33. The method for synchronized audio and video playback according to claim 23, characterized in that, The step of processing the video and the audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the audio and the display timestamp of the first frame of the video, so that the video and the audio are synchronized, includes: When the timestamp of the first audio frame is greater than the timestamp of the first video frame, the first video frame is played directly, and the display state of the first video frame is maintained. The initial value of the system clock is determined by the timestamp of the first audio frame. The video is subjected to frame dropping until the current display timestamp of the video matches the display timestamp of the first frame of the audio, at which point the audio and the video are played.

34. The method for synchronized audio and video playback according to claim 23, characterized in that, The step of processing the video and the audio using a corresponding synchronization strategy based on the display timestamp of the first frame of the audio and the display timestamp of the first frame of the video, so that the video and the audio are synchronized, includes: When the timestamp of the first frame of the audio is equal to the timestamp of the first frame of the video, the audio and the video are played directly.

35. The method for synchronized audio and video playback according to claim 1, characterized in that, The step of processing the audio and video using a corresponding synchronization strategy based on the distribution position relationship characteristics of the first frame audio and the first frame video in the bitstream to synchronize the audio and video includes: If the bitstream contains only the audio or the video, then the audio or the video is played directly.

36. A playback device adapted to synchronously play audio and video, comprising: The first extraction unit is adapted to extract the distribution position of the first frame audio in the bitstream; The second extraction unit is adapted to extract the distribution position of the first frame of video in the bitstream; The processing unit is adapted to process the audio and video using a corresponding synchronization strategy based on the distribution position relationship characteristics of the first frame audio and the first frame video in the bitstream, so that the audio and video are synchronized. This includes: when the distribution position interval between the first frame audio and the first frame video in the bitstream is greater than a preset threshold, and the distribution position of the first frame video in the bitstream precedes the first frame audio, playing the video and extracting the current display timestamp of the video and the display timestamp of the first frame audio; processing the audio using a corresponding synchronization strategy based on the current display timestamp of the video and the display timestamp of the first frame audio, so that the processed audio is synchronized with the video, including: when the display timestamp of the first frame audio is less than the current display timestamp of the video, performing frame dropping processing on the audio until the display timestamp of the audio matches the display timestamp of the video, then playing the audio; when the distribution position interval between the first frame audio and the first frame video in the bitstream is less than the preset threshold, extracting the display timestamp of the first frame audio and the display timestamp of the first frame video. The system clock is synchronized with the audio and video using a corresponding synchronization strategy based on the timestamps of the first audio frame and the first video frame. This includes: when the timestamp of the first audio frame is less than the timestamp of the first video frame, obtaining the number of frames in the audio buffer and calculating the number of dropped audio frames; comparing the number of frames in the audio buffer with the number of dropped audio frames, and when the number of dropped audio frames is greater than the number of audio frames in the audio buffer, performing frame dropping on the audio buffer and playing the first video frame, with the initial value of the system clock determined by the timestamp corresponding to the first audio frame after the dropped audio frames; playing the audio at the timestamp corresponding to the first audio frame after the dropped audio frames; and, when the system clock is inconsistent with the current timestamp of the video, playing the video at slow speed until the system clock is consistent with the current timestamp of the video, then playing the video at normal speed; wherein the timestamps of the first audio frame and the first video frame are generated based on the same system clock.

37. A playback device, comprising: A memory, a processor, a display, and an audio player, wherein the memory stores computer instructions executable on the processor, characterized in that the processor, when executing the computer instructions, performs the steps of the audio and video synchronized playback method according to any one of claims 1 to 35.

Images