Audio and video processing methods, apparatus, and storage media

Abstract

This application provides an audio and video processing method, apparatus, and storage medium. The method acquires video and audio data frames, determines a video sequence number jump value based on the timestamp of the currently received video data frame and the timestamps of the previous two received video data frames, and then uses this video sequence number jump value to determine the expected playback time of the video. Similarly, it determines the audio sequence number jump value and the expected audio playback time using the same principle. Finally, it achieves audio and video synchronization by utilizing the correspondence between the expected audio playback time and the expected video playback time. By using video and audio sequence number jump thresholds to determine whether a jump has occurred, the sequence number jump thresholds can be used as the basis for determining the expected playback time. Furthermore, because the expected audio playback time and the expected video playback time are consistent, synchronization of audio and video data is achieved.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of communication technology, and in particular to an audio and video processing method, apparatus, and storage medium. Background Technology

[0002] There are many existing audio and video synchronization technologies. However, these methods generally use frame dropping or repeated frames to synchronize video and audio streams. In practical applications, the following scenario may occur: if the camera on the other side of the network restarts or experiences network jitter, causing the camera's RTP (Real-time Transport Protocol) timestamp to change at a certain moment, but the video or sound is continuous in time with the image or sound before the change, continuing to use the existing methods for audio and video synchronization in this scenario will result in repeated or dropped frames for a long time, seriously affecting the user's viewing experience. Summary of the Invention

[0003] This application provides an audio and video processing method, apparatus, and storage medium that can reduce the phenomenon of duplicate or dropped frames caused by timestamp changes.

[0004] In a first aspect, embodiments of this application provide an audio and video processing method, the audio and video processing method comprising:

[0005] Acquire video and audio data frames;

[0006] The video sequence number jump value is determined based on the timestamp sequence number of the video data frame received this time and the timestamp sequence numbers of the video data frames received in the previous two times.

[0007] The expected playback time of the video is determined based on the video sequence number jump value, and the expected playback time of the video represents the playback time corresponding to the video data frame.

[0008] The audio sequence number jump value is determined based on the timestamp sequence number of the audio data frame received this time and the timestamp sequence numbers of the audio data frames received in the previous two times.

[0009] The expected audio playback time is determined based on the audio sequence number jump value. The expected audio playback time represents the playback time corresponding to the audio data frame, wherein the expected audio playback time is consistent with the expected video playback time.

[0010] The audio data frames and the video data frames are synchronized according to the expected audio playback time and the expected video playback time.

[0011] Secondly, embodiments of this application also provide an audio and video processing apparatus, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the audio and video processing method as described above.

[0012] Thirdly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for performing the audio and video processing method described above.

[0013] This application's embodiments include: acquiring video data frames and audio data frames; determining a video sequence number jump value based on the timestamp sequence number of the currently received video data frame and the timestamp sequence numbers of the two previously received video data frames; determining the expected video playback time based on the video sequence number jump value, where the expected video playback time represents the playback time corresponding to the video data frame; determining an audio sequence number jump value based on the timestamp sequence number of the currently received audio data frame and the timestamp sequence numbers of the two previously received audio data frames; determining the expected audio playback time based on the audio sequence number jump value, where the expected audio playback time represents the playback time corresponding to the audio data frame, wherein the expected audio playback time is consistent with the expected video playback time; and performing synchronization processing on the audio data frames and video data frames based on the expected audio playback time and the expected video playback time. By using the timestamp sequence number of video data frames, the video sequence number jump threshold is determined, thus revealing the continuity of the video and identifying any jumps. This video sequence number jump threshold is then used to determine the expected playback time of the received video data frame, preventing prolonged frame repetition or frame drops during playback. This also ensures accurate playback of the video data frame within its expected playback time. Similarly, when receiving audio data frames, the timestamp sequence number is used to determine the audio sequence number jump threshold, revealing the continuity of the audio and identifying any jumps. This audio sequence number jump threshold is then used to determine the expected playback time of the received audio data frame, preventing prolonged frame repetition or frame drops during playback. This also ensures accurate playback of the audio data frame within its expected playback time. Finally, because the expected playback time of the audio and video data is consistent, the received audio and video data frames can be accurately matched in time, achieving synchronization of the audio and video data. Attached Figure Description

[0014] Figure 1 This is a flowchart of an audio / video processing method provided in one embodiment of this application;

[0015] Figure 2This is a flowchart of determining the video sequence number transition value provided in one embodiment of this application;

[0016] Figure 3 This is a flowchart for determining the audio sequence number transition value provided in one embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0019] This application provides an audio and video processing method, apparatus, and storage medium. Upon receiving a video data frame, the timestamp number of the video data frame can be used to determine the video sequence number jump threshold, thereby understanding the continuity of the video and determining whether a jump has occurred. The video sequence number jump threshold can then be used as a basis for determining the expected playback time of the received video data frame, avoiding prolonged repeated frames or dropped frames during playback. This also ensures accurate playback of the video data frame within the expected playback time. Similarly, upon receiving an audio data frame, the timestamp number of the audio data frame can be used to determine the audio sequence number jump threshold, thereby understanding the continuity of the audio and determining whether a jump has occurred. This can then be used as a basis for determining the expected playback time of the received audio data frame, avoiding prolonged repeated frames or dropped frames during playback. This also ensures accurate playback of the audio data frame within the expected playback time. Ultimately, because the expected playback time of the audio and the expected playback time of the video are consistent, the received audio and video data frames can be accurately matched in time, thus achieving synchronization of the audio and video data.

[0020] like Figure 1 As shown, Figure 1 This is a flowchart of an audio and video processing method provided in one embodiment of this application.

[0021] like Figure 1 As shown, the audio and video processing method includes steps S100, S200, S300, S400, S500, and S600.

[0022] Step S100: Acquire video data frames and audio data frames;

[0023] Step S200: Determine the video sequence number jump value based on the timestamp sequence number of the video data frame received this time and the timestamp sequence numbers of the two previous video data frames;

[0024] Step S300: Determine the expected playback time of the video based on the video sequence number jump value. The expected playback time of the video represents the playback time corresponding to the video data frame.

[0025] Step S400: Determine the audio sequence number jump value based on the timestamp sequence number of the audio data frame received this time and the timestamp sequence numbers of the two previous audio data frames received;

[0026] Step S500: Determine the expected audio playback time based on the audio sequence number jump value. The expected audio playback time represents the playback time corresponding to the audio data frame. The expected audio playback time is consistent with the expected video playback time.

[0027] Step S600: Synchronize the audio data frames and video data frames according to the expected audio playback time and the expected video playback time.

[0028] In this embodiment, upon receiving a video or audio data frame, the timestamp number carried by the video or audio data frame can be obtained. Therefore, the video or audio sequence number jump value can be determined directly by comparing the timestamp number of the currently received video or audio data frame with the timestamp numbers of the previous two received video or audio data frames. This allows for the determination of whether a timestamp number jump has occurred. Based on the audio sequence number jump value, the expected video playback time or expected audio playback time can be determined, ensuring that a prolonged period of frame loss or frame replacement is avoided due to significant timestamp number changes. Furthermore, since the expected video playback time and the expected audio playback time are time-consistent, ensuring that each received video data frame corresponds to the expected video playback time and each received audio data frame corresponds to the expected audio playback time is sufficient to guarantee audio-video synchronization.

[0029] Specifically, in scenarios involving live streaming or real-time conferencing, received video data frames carry timestamps. Because of the continuous transmission of video data frames by the audio / video data sender, the timestamps of the video data frames should normally be consecutive. If the timestamps are discontinuous, it indicates a problem with the video stream transmission. Timestamp jumps can cause significant changes in the timestamp sequence. Therefore, it is necessary to calculate the timestamp jump value using the timestamps of three consecutive received video data frames. By analyzing this value, it can be determined whether a jump has occurred. This jump value is then used to determine the expected playback time of the video under the jump state, ensuring that the video data frame can be played on time.

[0030] Similarly, the audio data frames received by the audio / video data processing end also carry timestamp sequences. Since the audio / video data sending end transmits video data frames continuously, the timestamp sequences of the audio data frames should also be continuous under normal circumstances. If the timestamp sequences are discontinuous, it indicates a problem with the audio stream transmission. Timestamp sequence jumps can cause significant changes in the timestamp sequence. Therefore, it is necessary to use the timestamp sequences of three consecutive received audio data frames to calculate the audio sequence jump value. By analyzing the audio sequence jump value, it can be determined whether a jump has occurred. Then, the audio sequence jump value is used to determine the expected playback time of the audio in the jump state, thus determining the expected playback time of the corresponding audio data frame and ensuring that the audio data frame can be played on time.

[0031] To better explain the audio-video synchronization principle of the audio-video processing method of this application, a brief description is provided here. Each received video data frame corresponds to an expected video playback time. After processing, the video data frame will be played precisely at this expected playback time. This is equivalent to setting different video data frames at different moments on a timeline; simply follow the pre-set process to play the video data frames. Similarly, different audio data frames can also be set at different moments on a timeline. In this case, as long as the expected audio playback time corresponds to the expected video playback time, their timeline alignment can be guaranteed. Therefore, after determining the expected audio playback time of the first audio data frame and the expected video playback time of the first video data frame, subsequent steps only require setting the corresponding expected playback time for each audio or video data frame according to the audio-video processing method of this application.

[0032] like Figure 2 As shown, Figure 2This is a flowchart of determining the video sequence number transition value according to an embodiment of this application, which further explains step S200. Step S200 includes, but is not limited to, steps S210, S220, and S230.

[0033] Step S210: Determine the first sequence number difference based on the timestamp sequence numbers of the two previously received video data frames;

[0034] Step S220: Determine the second sequence number difference based on the timestamp sequence number of the currently received video data frame and the timestamp sequence number of the previously received video data frame;

[0035] Step S220: Calculate the video sequence number jump value based on the difference between the second sequence number and the difference between the first sequence number.

[0036] This approach introduces the first two timestamp sequences to collaboratively determine the degree of timestamp sequence jump. The timestamp sequences of the three received video data frames are used to determine the video sequence jump value, which is the ratio of the difference between the first and second sequences. This allows for a direct and effective understanding of the degree of timestamp sequence jump, facilitating subsequent judgment on whether a timestamp sequence jump has occurred.

[0037] Specifically, the timestamp numbers of the three video data frames are x k-1 x k x k+1 For example, the constraint formula for calculating the video sequence number transition value u can be found in the following formula:

[0038]

[0039] In some embodiments, step S300 is further described, and step S300 includes, but is not limited to, the following steps: when the video sequence number jump value is greater than the preset video sequence number jump threshold, add a first time interval to the expected video playback time corresponding to the previously received video data frame, determine the expected video playback time, and obtain the first expected video playback time, wherein the first time interval represents the time interval between two received video data frames.

[0040] Considering that the timestamp sequence number change is relatively large during a jump, the video sequence number jump threshold also needs to be set relatively large to distinguish it from short-term timestamp sequence number loss caused by packet loss. After determining the video sequence number jump value of the received video data frame, it is necessary to judge the video sequence number jump value. When the video sequence number jump value is greater than the video sequence number jump threshold, it can be determined that a timestamp sequence number jump has occurred. In this case, directly using the timestamp sequence number to determine the expected playback time of the video can easily lead to discontinuous expected playback times. Furthermore, considering that the video data frame will be sent normally when a timestamp sequence number jump occurs, the expected playback time of the received video data frame can be determined based on the first time interval. Therefore, after determining that a timestamp sequence number jump has occurred, simply adding the first time interval to the expected playback time of the video corresponding to the previously received video data frame is sufficient to determine the expected playback time of the received video data frame. At the same time, this expected playback time will also be recorded as the first expected playback time, used for subsequent updates to the initial video timestamp sequence number and the initial expected playback time.

[0041] In some embodiments, the audio and video processing method further includes: when the video sequence number jump value is less than the video sequence number jump threshold, determining the expected video playback time based on the initial video timestamp sequence number, the initial expected video playback time, and the timestamp sequence number of the video data frame to obtain a second expected video playback time, wherein the initial video timestamp sequence number is obtained based on the timestamp sequence number of the currently received video data frame; and the initial expected video playback time is obtained based on the expected video playback time of the currently received video data frame.

[0042] When there is no timestamp sequence number jump, the timestamp sequence number change will not be significant. During normal playback, the timestamp sequence number remains continuous. Even when network fluctuations cause packet loss and missing video data frames, the missing timestamp sequence number will not be excessive. In this case, the timestamp sequence number of the currently received video data frame can be directly used to determine the expected playback time of the video. Furthermore, if the video sequence number jump value corresponding to the currently received video data frame is less than the preset video sequence number jump threshold, it indicates that the currently received video data frame is in a normal playback state or a state of network packet loss, with no jump. In this case, based on the preset initial video timestamp sequence number, the initial expected video playback time, and the timestamp sequence number of the currently received video data frame, the expected playback time of this frame can be quickly determined. Specifically, the playback time interval between every two video data frames is fixed. Therefore, by determining the difference between the currently received video data frame and the initial video timestamp sequence number, the expected playback time of the currently received video data frame can be determined using this time interval based on the initial expected video playback time.

[0043] Specifically, the difference between the timestamp of the current video data frame and the initial video timestamp can be used to determine the number of timestamps in the interval. Then, multiply this number by the playback time interval between each two video data frames to determine the video playback time difference from the initial expected playback time. Finally, the expected playback time of the current received video data frame can be determined by adding the video playback time difference to the initial expected playback time.

[0044] In some embodiments, the initial video timestamp number and the initial expected video playback time are obtained by the following steps: when the video data frame is the first received video data frame, the timestamp number of the first video data frame is determined as the initial video timestamp number, and the expected video playback time corresponding to the first video data frame is determined as the initial expected video playback time.

[0045] After receiving the first video data frame, initialization begins. At this time, the timestamp sequence number of the first video data frame is used to initialize the initial video timestamp sequence number, and the expected playback time of the first video data frame is used to initialize the initial video expected playback time.

[0046] Specifically, audio and video processing methods also include:

[0047] When the video sequence number jump value is greater than the video sequence number jump threshold, the initial video timestamp sequence number is updated according to the timestamp sequence number of the video data frame received this time.

[0048] Update the expected playback time of the initial video based on the expected playback time of the first video.

[0049] When a timestamp sequence number changes in a video data frame, the timestamp sequence numbers of all subsequent received video data frames are assigned based on the changed timestamp sequence number. Therefore, the initial video timestamp sequence number and initial expected playback time of the first video data frame cannot be used to calculate the expected playback time of subsequent frames. Furthermore, the initial video timestamp sequence number and initial expected playback time of the previous frame cannot be used after each change. To address this, when a timestamp sequence number changes, the timestamp sequence number and the first expected playback time of the currently received video data frame are first determined. Then, these timestamp sequence number and the first expected playback time are directly used to update the initial video timestamp sequence number and the initial expected playback time. Subsequent expected playback times are calculated based on the updated initial timestamp sequence number and the initial expected playback time, thus ensuring the accuracy and smoothness of the entire video data frame playback on the timeline.

[0050] In some embodiments, when the video data frame is the first received video data frame, the audio and video processing method further includes: when the time of receiving the first video data frame is earlier than or equal to the time of receiving the first audio data frame, setting the expected video playback time to a preset time value.

[0051] When initializing the initial video timestamp sequence number and the initial expected video playback time, the initial video timestamp sequence number can be directly obtained from the first video data frame, but the initial expected video playback time cannot be obtained directly. In this case, a preset time value can be defined as the start time, i.e., the initial expected video playback time. In some embodiments, the preset time value can be directly set to 0 seconds.

[0052] In some embodiments, when the video data frame is the first received video data frame, the audio and video processing method further includes: when the time of receiving the first video data frame is later than the time of receiving the first audio data frame, determining the time interval between receiving the first video data frame and receiving the first audio data frame as the expected video playback time.

[0053] In practice, due to various reasons, there may be a certain time interval between the first video data frame and the first audio data frame when they are sent for the first time. In this case, in order to continue to maintain the correspondence between the audio data frame and the video data frame, it is necessary to keep a fixed time interval between them initially. Therefore, when the video data frame is the first frame and is later than the first audio data frame, the expected playback time of the video cannot be directly determined to be 0 seconds, and a time delay of a time interval is required.

[0054] Specifically, to better illustrate the constraint relationship between the expected video playback time, timestamp sequence number, initial video timestamp sequence number, and initial expected video playback time, please refer to the following constraint formula:

[0055]

[0056] In the formula, x start x is the initial video timestamp sequence number. k For the k-th video data frame, t x abs(x) is the time interval between two playback of video data frames. start ) represents the initial expected playback time of the video, where 0 represents 0 seconds, and Δ represents the time interval between the first video data frame and the first audio data frame.

[0057] When the video data frame is the first frame, Equation (1) is used to calculate the expected video playback time, and the initial video timestamp sequence number and the initial expected video playback time can be determined. When there is no jump in the transmission of the video data frame, Equation (2) is used to determine the expected video playback time. When there is a jump in the video data frame, Equation (3) is used to determine the expected video playback time, and the initial video timestamp sequence number and the initial expected video playback time are updated at the same time.

[0058] In some embodiments, the audio / video processing method further includes:

[0059] If no video data frame is received within the preset video frame interpolation time threshold, copy the previously received video data frame.

[0060] The expected playback time of the copied video data frame is determined based on the first time interval and the expected playback time of the video corresponding to the previous received video data frame, wherein the first time interval represents the time interval between two received video data frames.

[0061] During video data transmission, if packet loss occurs, there may be periods without video data frames. In such cases, performing frame interpolation after determining the video sequence number jump value would cause a delay. Therefore, a video interpolation time threshold can be used to directly copy the previous frame to compensate for the missing video data frame. If no video data frame is received for each interpolation time threshold, the previous frame is used to complete the interpolation. This continues until a normal video data frame is received, at which point the desired playback time of the video is determined.

[0062] like Figure 3 As shown, Figure 3 This is a flowchart of determining the audio sequence number transition value according to an embodiment of this application, which further explains step S400. Step S400 includes, but is not limited to, steps S410, S420, and S430.

[0063] Step S410: Determine the difference in the third sequence number based on the timestamp sequence numbers of the first two received audio data frames;

[0064] Step S420: Determine the fourth sequence number difference based on the timestamp sequence number of the currently received audio data frame and the timestamp sequence number of the previously received audio data frame;

[0065] Step S430: Calculate the audio sequence number jump value based on the difference between the third and fourth sequence numbers.

[0066] This approach introduces the timestamp sequence numbers from the first two timestamps to collaboratively determine the degree of timestamp sequence number jumps. The audio sequence number jump value is determined using the timestamp sequence numbers from three received audio data frames; that is, the ratio of the difference between the third and fourth timestamp sequence numbers. This directly and effectively reveals the degree of timestamp sequence number jump, facilitating subsequent determination of whether a timestamp sequence number jump has occurred. The constraint formula for calculating the audio sequence number jump value can refer to the constraint formula for calculating the video sequence number jump value.

[0067] In some embodiments, step S600 is further described, and step S600 includes, but is not limited to, the following steps: when the audio sequence number jump value is greater than the preset audio sequence number jump threshold, add a second time interval to the audio expected playback time corresponding to the last received audio data frame, determine the audio expected playback time, and obtain the first audio expected playback time, wherein the second time interval represents the time interval between two received audio data frames.

[0068] Considering that the timestamp sequence number changes significantly during transitions, the audio sequence number transition threshold also needs to be set relatively high to distinguish it from short-term timestamp sequence number loss caused by packet loss. After determining the audio sequence number transition value of the received audio data frame, it needs to be judged. When the audio sequence number transition value is greater than the audio sequence number transition threshold, it can be determined that a timestamp sequence number transition has occurred. In this case, directly using the timestamp sequence number to determine the expected audio playback time can easily lead to discontinuous expected audio playback times. Furthermore, considering that the audio data frame will be sent normally when a timestamp sequence number transition occurs, the expected audio playback time of the received audio data frame can be determined based on the second time interval. Therefore, after determining that a timestamp sequence number transition has occurred, simply adding the second time interval to the expected audio playback time corresponding to the previously received audio data frame is sufficient to determine the expected audio playback time of the received audio data frame. This expected audio playback time will also be recorded as the first expected audio playback time, used for subsequent updates to the initial audio timestamp sequence number and the initial expected audio playback time.

[0069] In some embodiments, the audio and video processing method further includes: when the audio sequence number jump value is less than the audio sequence number jump threshold, determining the expected audio playback time based on the initial audio timestamp sequence number, the initial expected audio playback time, and the timestamp sequence number of the audio data frame, to obtain a second expected audio playback time, wherein the initial audio timestamp sequence number is obtained based on the timestamp sequence number of the currently received audio data frame, and the initial expected audio playback time is obtained based on the expected audio playback time of the currently received audio data frame.

[0070] When there is no timestamp sequence number jump, the timestamp sequence number will not change significantly. During normal playback, the timestamp sequence number remains continuous. Even when network fluctuations cause packet loss and audio data frames are missing, the number of missing timestamp sequences will not be excessive. In this case, the timestamp sequence number of the currently received audio data frame can be directly used to determine the expected playback time of the audio. Furthermore, if the audio sequence number jump value corresponding to the currently received audio data frame is less than the preset audio sequence number jump threshold, it indicates that the currently received audio data frame is in a normal playback state or a state of network packet loss, with no jump. In this case, the expected playback time of the current audio frame can be quickly determined based on the preset initial audio timestamp sequence number, the initial expected audio playback time, and the timestamp sequence number of the currently received audio data frame. Specifically, the playback time interval between every two audio data frames is fixed. Therefore, by determining the difference between the currently received audio data frame and the initial audio timestamp sequence number, the expected playback time of the current audio data frame can be determined using this time interval based on the initial expected audio playback time.

[0071] Specifically, the difference between the timestamp of the current audio data frame and the initial audio timestamp can be used to determine the number of timestamps in the interval. Then, multiply this number by the playback time interval between each two audio data frames to determine the audio playback time difference from the initial expected audio playback time. Finally, the expected audio playback time of the current received audio data frame can be determined by adding the audio playback time difference to the initial expected audio playback time.

[0072] In some embodiments, the initial audio timestamp number and the initial expected audio playback time are obtained by the following steps: when the audio data frame is the first received audio data frame, the timestamp number of the first audio data frame is determined as the initial audio timestamp number, and the expected audio playback time corresponding to the first audio data frame is determined as the initial expected audio playback time.

[0073] After receiving the first audio data frame, initialization begins. At this time, the timestamp sequence number of the first audio data frame is used to initialize the initial audio timestamp sequence number, and the expected audio playback time of the first audio data frame is used to initialize the expected audio playback time.

[0074] Specifically, audio and video processing methods also include:

[0075] When the audio sequence number jump value is greater than the audio sequence number jump threshold, the initial audio timestamp sequence number is updated according to the timestamp sequence number of the audio data frame received this time.

[0076] Update the initial expected playback time of the audio based on the expected playback time of the first audio.

[0077] When an audio data frame experiences a timestamp sequence change, the timestamps of all subsequent received audio data frames are assigned based on the changed timestamp. Therefore, the initial audio timestamp and initial expected playback time of the first audio data frame cannot be used to calculate the expected playback time of subsequent frames. Furthermore, the previous initial audio timestamp and expected playback time cannot be used after each timestamp. Instead, when a timestamp sequence change occurs, the timestamp and first expected playback time of the currently received audio data frame are first determined. These timestamps and the first expected playback time are then used to update the initial audio timestamp and expected playback time. Subsequent expected playback times are calculated based on the updated initial audio timestamp and expected playback time, thus ensuring the accuracy and smoothness of the entire audio data frame playback on the timeline.

[0078] In some embodiments, when the audio data frame is the first received audio data frame, the audio-video processing method further includes: when the time of receiving the first audio data frame is earlier than or equal to the time of receiving the first video data frame, setting the expected audio playback time to a preset time value.

[0079] When initializing the initial audio timestamp sequence number and the initial expected audio playback time, the initial audio timestamp sequence number can be directly obtained from the first audio data frame, but the initial expected audio playback time cannot be obtained directly. In this case, a preset time value can be defined as the start time, i.e., the initial expected audio playback time. In some embodiments, the preset time value can be directly set to 0 seconds.

[0080] In some embodiments, when the audio data frame is the first received audio data frame, the audio-video processing method further includes: when the time of receiving the first audio data frame is later than the time of receiving the first video data frame, determining the time interval between receiving the first audio data frame and receiving the first video data frame as the expected audio playback time.

[0081] In practice, due to various reasons, there may be a certain time interval between the first video data frame and the first audio data frame when they are sent for the first time. In this case, in order to continue to maintain the correspondence between the audio data frame and the video data frame, it is necessary to keep a fixed time interval between them initially. Therefore, when the audio data frame is the first frame and is later than the first video data frame, the expected audio playback time cannot be directly determined to be 0 seconds. A time delay of a time interval is required.

[0082] Specifically, the constraints on the expected audio playback time, timestamp sequence number, initial audio timestamp sequence number, and initial expected audio playback time can be referenced from the constraints on the expected video playback time, video data frame timestamp sequence number, initial video timestamp sequence number, and initial expected video playback time.

[0083] In some embodiments, the audio / video processing method further includes:

[0084] If no audio data frame is received within the preset audio interpolation time threshold, copy the previously received audio data frame.

[0085] The expected playback time of the copied audio data frame is determined based on the second time interval and the expected playback time of the audio data frame corresponding to the previous received audio data frame, wherein the second time interval represents the time interval between the two received audio data frames.

[0086] During audio data transmission, if packet loss occurs, there may be periods without audio data frames. In such cases, performing frame replacement after determining the audio sequence number jump value would cause a delay. Therefore, in this situation, the previous frame can be copied directly using an audio frame replacement time threshold to compensate for the missing audio data frame. If no audio data frame is received for each threshold period, the previous frame is used to complete the frame replacement. This process continues until a normal audio data frame is received, at which point the desired playback time of the audio is determined.

[0087] To more clearly illustrate the processing flow of the audio and video processing method provided in the embodiments of this application, specific examples are given below.

[0088] The audio and video processing method includes the following steps:

[0089] The system acquires the first audio data frame and the first video data frame, and determines the initial expected playback time and initial video timestamp number of the video data stream based on the first video data frame. It also determines the initial expected playback time and initial audio timestamp number of the audio data stream based on the first audio data frame. During the first initialization, the initial video timestamp number and the initial audio timestamp number are usually consistent, while the initial expected playback time of the video and the initial expected playback time of the audio are determined according to an NTP (Network Time Protocol) time interval.

[0090] It continuously receives audio and video data frames and continuously records the timestamps of the previous two received audio and video data frames. After receiving the current video or audio data frame, it can use the timestamps of the previous two records and the current timestamp to determine the video or audio sequence number jump value.

[0091] By comparing the video sequence number jump value with a preset video sequence number jump threshold or the audio sequence number jump value with a preset audio jump threshold, it can be determined whether a timestamp sequence number jump has occurred.

[0092] When it is determined that there is no change in the timestamp sequence number of the video data frame or audio data frame, the expected playback time of the video can be determined directly using the initial video timestamp sequence number, the initial expected playback time of the video, and the timestamp sequence number of the video data frame. The expected playback time of the audio can be determined using the initial audio timestamp sequence number, the initial expected playback time of the audio, and the timestamp sequence number of the audio data frame.

[0093] After determining that a timestamp sequence number has changed in a video or audio data frame, a first time interval is added to the expected playback time of the video corresponding to the last received video data frame to determine the expected playback time of the video after the change. At the same time, the first expected playback time of the video is obtained, and the initial video timestamp sequence number can be updated according to the first expected playback time of the video. Alternatively, the audio data frame can be processed according to the same principle as video processing to obtain the expected playback time of the audio after the change, and the initial audio timestamp sequence number can be updated.

[0094] In addition, when network packet loss is detected, the previous video or audio data frame can be copied directly, and then the settings can be completed in the normal way of determining the expected playback time of the video or audio, thereby ensuring the integrity and smoothness of video and audio playback.

[0095] The audio and video processing method of this application directly constructs a timeline using the expected video playback time and the expected audio playback time. By determining the corresponding expected video playback time and expected audio playback time for each video data frame and audio data frame, each video data frame and audio data frame is unique on the timeline, thereby enabling the entire media stream to achieve audio and video synchronization simply and clearly.

[0096] Furthermore, one embodiment of this application also provides a computer-readable storage medium storing computer-executable instructions for performing the audio-visual processing method described above. For example, the instructions can be executed by a processor in an embodiment of the audio-visual processing apparatus described above, causing the processor to perform the information processing method described above, such as executing the methods described above. Figure 1 The methods in Figure 2 The methods and Figure 3 The method in the middle.

[0097] In addition, one embodiment of this application provides an audio and video processing apparatus, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the audio and video processing method as described above.

[0098] The non-transient software program and instructions required to implement the audio and video processing method of the above embodiments are stored in memory. When executed by a processor, the audio and video processing method in the above embodiments is executed, for example, the method described above. Figure 1 The methods in Figure 2 The methods and Figure 3 The method in the middle.

[0099] Those skilled in the art will understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage unit technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

Claims

1. An audio / video processing method, comprising: Acquire video and audio data frames; The video sequence number jump value is determined based on the timestamp sequence number of the video data frame received this time and the timestamp sequence numbers of the video data frames received in the previous two times. The video sequence number jump value indicates the continuous state of the video data frame; the video sequence number jump value being less than the preset video sequence number jump threshold indicates that the video data frame is in a normal playback state or a network packet loss state. The expected playback time of the video is determined based on the video sequence number jump value, and the expected playback time of the video represents the playback time corresponding to the video data frame. The audio sequence number jump value is determined based on the timestamp sequence number of the audio data frame received this time and the timestamp sequence numbers of the audio data frames received in the previous two times. The audio sequence number jump value indicates the continuous state of the audio data frame; the audio sequence number jump value being less than the preset audio sequence number jump threshold indicates that the audio data frame is in a normal playback state or a network packet loss state. The expected audio playback time is determined based on the audio sequence number jump value. The expected audio playback time represents the playback time corresponding to the audio data frame, wherein the expected audio playback time is consistent with the expected video playback time. The audio data frames and the video data frames are synchronized according to the expected audio playback time and the expected video playback time.

2. The audio and video processing method according to claim 1, characterized in that, The step of determining the video sequence number transition value based on the timestamp sequence number of the currently received video data frame and the timestamp sequence numbers of the two previously received video data frames includes: The first sequence number difference is determined based on the timestamp sequence numbers of the two previously received video data frames; The second sequence number difference is determined based on the timestamp sequence number of the video data frame received this time and the timestamp sequence number of the video data frame received last time; The video sequence number jump value is calculated based on the difference between the second sequence number and the difference between the first sequence number.

3. The audio and video processing method according to claim 1, characterized in that, The step of determining the expected video playback time based on the video sequence number jump value includes: When the video sequence number jump value is greater than the preset video sequence number jump threshold, a first time interval is added to the expected playback time of the video corresponding to the previously received video data frame to determine the expected playback time of the video and obtain the first expected playback time of the video. The first time interval represents the time interval between two received video data frames.

4. The audio and video processing method according to claim 3, characterized in that, The step of determining the expected video playback time based on the video sequence number jump value includes: When the video sequence number jump value is less than the video sequence number jump threshold, the expected video playback time is determined based on the initial video timestamp sequence number, the initial expected video playback time, and the timestamp sequence number of the video data frame, to obtain the second expected video playback time. The initial video timestamp sequence number is obtained based on the timestamp sequence number of the video data frame received this time; the initial expected video playback time is obtained based on the expected video playback time of the video data frame received this time.

5. The audio and video processing method according to claim 4, characterized in that, The initial video timestamp number and the initial expected playback time of the video are obtained through the following steps: When the video data frame is the first received video data frame, the timestamp number of the first video data frame is determined as the initial video timestamp number, and the expected playback time of the video corresponding to the first video data frame is determined as the initial expected playback time of the video.

6. The audio and video processing method according to claim 4 or 5, characterized in that, The audio and video processing method further includes: When the video sequence number jump value is greater than the video sequence number jump threshold, the initial video timestamp sequence number is updated according to the timestamp sequence number of the video data frame received this time; The initial expected playback time of the video is updated based on the expected playback time of the first video.

7. The audio and video processing method according to claim 1, characterized in that, When the video data frame is the first received video data frame, the audio and video processing method further includes: When the time of receiving the first video data frame is earlier than or equal to the time of receiving the first audio data frame, the expected playback time of the video is set to a preset time value.

8. The audio and video processing method according to claim 1, characterized in that, When the video data frame is the first received video data frame, the audio and video processing method further includes: When the time of receiving the first video data frame is later than the time of receiving the first audio data frame, the time interval between receiving the first video data frame and receiving the first audio data frame is determined as the expected playback time of the video.

9. The audio and video processing method according to claim 1, characterized in that, The audio and video processing method further includes: If no video data frame is received within the preset video interpolation time threshold, the previously received video data frame is copied. The expected playback time of the copied video data frame is determined based on the first time interval and the expected playback time of the video corresponding to the last received video data frame, wherein the first time interval represents the time interval between two received video data frames.

10. The audio and video processing method according to claim 1, characterized in that, The step of determining the audio sequence number transition value based on the timestamp sequence number of the currently received audio data frame and the timestamp sequence numbers of the two previously received audio data frames includes: The difference in the third sequence number is determined based on the timestamp sequence numbers of the first two received audio data frames; The fourth sequence number difference is determined based on the timestamp sequence number of the audio data frame received this time and the timestamp sequence number of the audio data frame received last time. The audio sequence number jump value is calculated based on the difference between the third and fourth sequence numbers.

11. The audio and video processing method according to claim 1, characterized in that, The step of determining the expected audio playback time based on the audio sequence number jump value includes: When the audio sequence number jump value is greater than the preset audio sequence number jump threshold, a second time interval is added to the expected audio playback time corresponding to the last received audio data frame to determine the expected audio playback time and obtain the first expected audio playback time, wherein the second time interval represents the time interval between two received audio data frames.

12. The audio and video processing method according to claim 11, characterized in that, The audio and video processing method further includes: When the audio sequence number jump value is less than the audio sequence number jump threshold, the expected audio playback time is determined based on the initial audio timestamp number, the initial expected audio playback time, and the timestamp number of the audio data frame, to obtain the second expected audio playback time. The initial audio timestamp number is obtained based on the timestamp number of the audio data frame received this time, and the initial expected audio playback time is obtained based on the expected audio playback time of the audio data frame received this time.

13. The audio and video processing method according to claim 12, characterized in that, The initial audio timestamp number and the initial expected audio playback time are obtained through the following steps: When the audio data frame is the first received audio data frame, the timestamp number of the first audio data frame is determined as the initial audio timestamp number, and the expected audio playback time corresponding to the first audio data frame is determined as the initial expected audio playback time.

14. The audio and video processing method according to claim 12 or 13, characterized in that, The audio and video processing method further includes: When the audio sequence number jump value is greater than the audio sequence number jump threshold, the initial audio timestamp sequence number is updated according to the timestamp sequence number of the audio data frame received this time; The initial expected audio playback time is updated based on the first expected audio playback time.

15. The audio and video processing method according to claim 1, characterized in that, When the audio data frame is the first received audio data frame, the audio and video processing method further includes: When the time of receiving the first audio data frame is earlier than or equal to the time of receiving the first video data frame, the expected audio playback time is set to a preset time value.

16. The audio and video processing method according to claim 1 or 15, characterized in that, When the audio data frame is the first received audio data frame, the audio and video processing method further includes: When the time of receiving the first audio data frame is later than the time of receiving the first video data frame, the time interval between receiving the first audio data frame and receiving the first video data frame is determined as the expected audio playback time.

17. The audio and video processing method according to claim 1, characterized in that, The audio and video processing method further includes: If no audio data frame is received within a preset audio interpolation time threshold, the previously received audio data frame is copied. The expected playback time of the copied audio data frame is determined based on the second time interval and the expected playback time of the audio data frame corresponding to the last received audio data frame, wherein the second time interval represents the time interval between the two received audio data frames.

18. An audio / video processing apparatus, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the audio and video processing method as described in any one of claims 1 to 17.

19. A computer-readable storage medium storing computer-executable instructions, said computer-executable instructions being executed by a processor to implement the audio / video processing method according to any one of claims 1 to 17.

Images