Audio and video data synchronization method and device, computer equipment and storage medium

Abstract

The invention provides an audio and video synchronization method and device, computer equipment and a storage medium so as to improve user experience in a complex network environment. Firstly, audio and video data packets are received and cached in a playing buffer area; calculating data packet receiving time through an adaptive filtering algorithm to obtain basic delay evaluation data; and the particle filter identifies network fluctuation and delay jitter data to obtain real-time network state evaluation data. Based on these data, the system determines overall delay assessment data for historical periods of time to more accurately predict and adjust synchronization; the system dynamically adjusts the size of a playing buffer area and the cache amount of data packets, and ensures that enough cache data exist in audio and video synchronous processing. And finally, performing synchronization processing on the data packets in the playing buffer area to obtain and play target audio and video data. Generally speaking, the scheme effectively deals with network changes, reduces audio and video asynchronization and improves user experience by dynamically adjusting the size of the buffer area and the buffer amount.

Description

Technical Field

[0001] The present invention relates to the technical field of synchronous processing of audio and video data, and in particular to a method, device, computer equipment and storage medium for synchronizing audio and video data. Background Art

[0002] In modern multimedia systems, audio and video synchronization is one of the key technologies to ensure user experience. The quality of audio and video synchronization directly affects the user's experience when watching videos, especially in complex and changeable network environments, the requirements for audio and video synchronization technology are more stringent. Existing audio and video synchronization technologies usually rely on calculating the reception time of audio and video data packets and adjusting the playback speed to achieve synchronization. However, these methods often have certain shortcomings when faced with complex network environments and sudden delays.

[0003] Specifically, existing audio and video synchronization methods mainly rely on accelerating or decelerating playback to adjust synchronization, but this method may not work well in the face of sudden large delays, and it is difficult to respond and adjust quickly, resulting in audio and video being out of sync, affecting user experience. In addition, during the synchronization process, simply accelerating or decelerating the playback of audio data may result in a decrease in audio quality or unnatural playback effects. Although this method can achieve audio and video synchronization to a certain extent, it often affects the frequency and quality of the audio, and users may feel obvious discomfort during viewing. Summary of the invention

[0004] The main purpose of the present invention is to provide a method, device, computer equipment and storage medium for synchronizing audio and video data, so as to solve the problem that the existing audio and video synchronization methods are not effective and affect the user experience in modern multimedia systems.

[0005] To achieve the above-mentioned purpose, the present invention provides a method for synchronizing audio and video data, comprising the following steps: receiving audio data packets and video data packets of audio and video, and caching the audio data packets and video data packets in a playback buffer; calculating and processing the reception time of the data packets according to an adaptive filtering algorithm to obtain basic delay evaluation data of each data packet; identifying network fluctuation data and delay jitter data of the transmission network of the audio data packet through a particle filter to obtain real-time network status evaluation data; determining the overall delay evaluation data of a historical time period with a preset length from the current moment according to the basic delay evaluation data and the real-time network status evaluation data; dynamically adjusting the playback buffer size and the cached data volume of the audio data packet and / or video data packet based on the overall delay evaluation data; performing audio and video synchronization processing on the audio data packets and video data packets stored in the playback buffer to obtain and play the target audio and video data.

[0006] Furthermore, based on the basic delay evaluation data and the real-time network status evaluation data, the overall delay evaluation data of a historical time period of a preset time length from the current moment is determined, including: determining the basic delay evaluation data of the data packets that should theoretically be received within the historical time period of a preset time length from the current moment; determining the average delay time, total delay time and maximum delay time of the audio data packets, and the average delay time, total delay time and maximum delay time of the video data packets within the historical time period based on the determined basic delay evaluation data, to obtain first delay evaluation data; determining the network fluctuation data and delay jitter data of the transmission network within the historical time period of a preset time length from the current moment; calculating the overall network evaluation value of the transmission network within the historical time period based on the determined network fluctuation data and delay jitter data, and using the overall network evaluation value as the second delay evaluation data; integrating the first delay evaluation data and the second delay evaluation data to obtain the overall delay evaluation data.

[0007] Further, according to the determined network fluctuation data and delay jitter data, calculating the overall network evaluation value of the transmission network in the historical time period, including: using a first formula to calculate the determined network fluctuation data and delay jitter data to obtain the overall network evaluation value N, wherein the first formula is: , wherein the is the mean of network fluctuation data, is the variance of network fluctuation data, is the skewness of network volatility data, is the kurtosis of network fluctuation data, is the mean value of the delay jitter data, is the variance of the delay jitter data, The skewness of the delay jitter data, the is the kurtosis of the delay jitter data, and stated is the preset weight coefficient, and stated is the preset index parameter.

[0008] Further, dynamically adjusting the playback buffer size based on the overall delay evaluation data includes: determining whether the overall network evaluation value in the overall delay evaluation data is lower than a first threshold; if the overall network evaluation value is lower than the first threshold, expanding the playback buffer according to a second formula; if the overall network evaluation value is not lower than the first threshold, shrinking the playback buffer according to a third formula, wherein the second formula is: , wherein the third formula is: , Indicates the new playback buffer size, Indicates the current playback buffer size. It represents the sum of the average delay time of the audio data packet and the average delay time of the video data packet. represents the first delay threshold, Represents the expansion weight value, Represents the shrinkage weight value.

[0009] Further, dynamically adjusting the cached data volume of the audio data packet based on the overall delay evaluation data includes: based on the total value of delayed data packets and the maximum delay time of the audio data packets in the historical time period in the overall delay evaluation data, and using a fourth formula to calculate the total value of delayed data packets and the maximum delay time to obtain the current cached data volume of the audio data packet; wherein the fourth formula is: , Indicates the new audio data packet buffer size. Indicates the current audio data packet buffer size. Indicates the current audio data packet delay. Indicates the expected audio data packet delay. Indicates the maximum delay time, Indicates the audio packet adjustment factor.

[0010] Further, dynamically adjusting the cached data volume of the video data packet based on the overall delay evaluation data includes: based on the total value of delayed data packets and the maximum delay time of the video data packets in the historical time period in the overall delay evaluation data, and using a fourth formula to calculate the total value of delayed data packets and the maximum delay time to obtain the current cached data volume of the video data packet; wherein the fourth formula is: , Indicates the new video data packet buffer size, Indicates the current video data packet cache size, Indicates the current video data packet delay. Indicates the expected video packet delay, Indicates the maximum delay time, Indicates the video packet adjustment factor.

[0011] Furthermore, audio and video synchronization processing is performed on the audio data packets and video data packets stored in the playback buffer to obtain and play target audio and video data, including: performing timestamp extraction processing on the audio data packets to obtain timestamp information of the audio data packets; performing timestamp extraction processing on the video data packets to obtain timestamp information of the video data packets; performing timestamp matching processing on the audio data packets and the video data packets to obtain audio and video synchronization timestamp alignment information; performing time stretching processing on the audio data packets to adjust the audio playback speed without changing the audio frequency to obtain stretched audio data packets; performing smoothing processing on the stretched audio data packets to obtain smoothed audio data packets; based on the audio and video synchronization timestamp alignment information, decoding processing is performed on the smoothed audio data packets and the video data packets to obtain and play the target audio and video data.

[0012] The present invention also provides a synchronization device for audio and video data, comprising: a receiving unit, used to receive audio data packets and video data packets of audio and video, and cache the audio data packets and video data packets in a playback buffer; a calculation unit, used to calculate and process the receiving time of the data packet according to an adaptive filtering algorithm, and obtain basic delay evaluation data of each data packet; an identification unit, used to identify network fluctuation data and delay jitter data of the transmission network of the audio data packet through a particle filter, and obtain real-time network status evaluation data; a first evaluation unit, used to determine the overall delay evaluation data of a historical time period of a preset length from the current moment according to the basic delay evaluation data and the real-time network status evaluation data; a second evaluation unit, used to dynamically adjust the playback buffer size and the cached data volume of the audio data packet and / or video data packet based on the overall delay evaluation data; a synchronization unit, used to perform audio and video synchronization processing on the audio data packet and video data packet stored in the playback buffer, and obtain and play the target audio and video data.

[0013] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0014] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0015] The method, device, computer equipment and storage medium for synchronizing audio and video data provided by the present invention are intended to improve audio and video synchronization technology to enhance the user experience when watching videos, especially in the case of complex and changeable network environments. It includes the following steps: First, audio data packets and video data packets of audio and video are received and cached in a playback buffer. This ensures that both audio and video data can be effectively managed and synchronized during playback. Then, the reception time of the processing data packet is calculated by an adaptive filtering algorithm to obtain the basic delay evaluation data of each data packet. This step helps to establish a basic timeline for subsequent delay evaluation and synchronization processing of data packets. Then, through a particle filter, the network fluctuation data and delay jitter data of the transmission network of the audio data packet are identified to obtain real-time network status evaluation data. This step allows the system to monitor changes in network status in real time, so as to better adapt to the changing network environment. Then, based on the basic delay evaluation data and the real-time network status evaluation data, the overall delay evaluation data of the historical time period preset from the current moment is determined. In this way, the system can obtain the overall trend of delay so as to more accurately predict and adjust synchronization. Subsequently, the size of the playback buffer and the cached data volume of the audio and video data packets are dynamically adjusted according to the overall delay evaluation data. By dynamically adjusting the buffer and the amount of cached data for audio and video packets, the system can better cope with different delay situations and maintain audio and video synchronization. Finally, the audio and video packets stored in the playback buffer are processed for audio and video synchronization to obtain and play the target audio and video data. This step ensures that the audio and video content presented to the user is effectively synchronized, thereby providing a better viewing experience.

[0016] In general, this technical solution achieves better response to changes in the network environment and reduces audio and video asynchrony by comprehensively utilizing the overall delay evaluation data obtained based on the basic delay evaluation data and the real-time network status evaluation data, and dynamically adjusting the playback buffer size and the cached data volume of the audio and video data packets in a timely manner, thereby improving the user experience quality when watching videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 1 is a schematic diagram of the steps of a method for synchronizing audio and video data in one embodiment of the present invention; Figure 2 is a structural block diagram of a synchronization device for audio and video data in one embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0018] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0020] It should be noted that the existing technology is widely used in audio and video data synchronization, but there are still many limitations. The existing method usually relies on calculating the delay of audio data for synchronization adjustment, and achieves audio and video synchronization by accelerating or decelerating audio playback. The basic principle of this method is to calculate the expected value of the reception time interval of adjacent audio packets and compare it with the actual reception time to determine the delay of the current audio and video pair. In the specific implementation, the system calculates the sum of the target delay of the previous pair of audio and video corresponding to the current audio and video pair and the relative delay of the most recently received audio and video packet to obtain the audio and video relative delay of the current audio and video pair. Based on this relative delay, the system calculates the minimum playback delay of the audio, and determines it and the maximum value of the expected audio delay as the audio target delay of the current audio and video pair. Finally, the system compares the actual audio delay with the audio target delay, and accelerates or decelerates the audio data according to the comparison result to achieve synchronous playback of audio data and video data. This method of the prior art can achieve basic audio and video synchronization in many cases, but its effect and stability are often limited when facing complex network environments and sudden delays.

[0021] Specifically, the existing technology has exposed some technical problems in practical applications, which limits its effect and user experience in audio and video synchronization. First, when faced with sudden large delays, the existing method has a slow response speed and is difficult to adjust quickly, resulting in audio and video synchronization, affecting the user's viewing experience. Secondly, the simple method of accelerating or decelerating the playback of audio data will directly affect the frequency and quality of the audio. Users may feel that the audio playback effect is unnatural and the sound quality is reduced during the viewing process. In addition, the existing method lacks the ability to respond to network fluctuations in real time, mainly relying on the calculation of delays for adjustment, and fails to fully consider the real-time fluctuations and delay jitter of the transmission network, resulting in poor synchronization when the network condition is unstable. Furthermore, the existing technology usually uses a fixed-size buffer to cache audio and video data, which cannot dynamically adapt to changes in network conditions. When the network fluctuates greatly, the fixed buffer may not be sufficient to cope with the change in delay, thereby affecting the synchronization effect and causing audio and video synchronization. Finally, due to the lack of real-time evaluation of the network status, the existing technology has limited adjustment capabilities when dealing with network jitter and delay fluctuations, and it is difficult to maintain good synchronization effects in a rapidly changing network environment. These problems show that the existing technology has many deficiencies in processing audio and video synchronization, and further improvements are needed to solve these technical problems.

[0022] To this end, the embodiment of the present invention provides a method for synchronizing audio and video data. Figure 1 , specifically including the following steps: S1, receiving an audio data packet and a video data packet of audio and video, and buffering the audio data packet and the video data packet in a play buffer.

[0023] S2, calculating and processing the receiving time of the data packet according to the adaptive filtering algorithm to obtain basic delay evaluation data of each data packet.

[0024] S3, identifying the network fluctuation data and delay jitter data of the transmission network of the audio data packet through a particle filter, and obtaining real-time network status evaluation data.

[0025] S4, determining overall delay evaluation data of a historical time period of a preset length from the current moment according to the basic delay evaluation data and the real-time network status evaluation data.

[0026] S5, dynamically adjusting the playback buffer size and the cached data volume of the audio data packet and / or video data packet based on the overall delay evaluation data.

[0027] S6, performing audio and video synchronization processing on the audio data packet and the video data packet stored in the play buffer to obtain and play the target audio and video data.

[0028] That is, this technical solution aims to improve audio and video synchronization technology to enhance the user experience when watching videos, especially in complex and changeable network environments. It includes the following steps: First, the audio data packets and video data packets of the audio and video are received and cached in the playback buffer. This ensures that both audio and video data can be effectively managed and synchronized during playback. Then, the reception time of the processing data packet is calculated by an adaptive filtering algorithm to obtain the basic delay evaluation data of each data packet. This step helps to establish a basic timeline for subsequent delay evaluation and data packet synchronization processing. Then, through the particle filter, the network fluctuation data and delay jitter data of the transmission network of the audio data packet are identified to obtain real-time network status evaluation data. This step allows the system to monitor changes in network status in real time, so as to better adapt to the changing network environment. Then, based on the basic delay evaluation data and the real-time network status evaluation data, the overall delay evaluation data of the historical time period preset from the current moment is determined. In this way, the system can obtain the overall trend of delay in order to more accurately predict and adjust synchronization. Subsequently, the playback buffer size and the cached data volume of the audio and video data packets are dynamically adjusted according to the overall delay evaluation data. By dynamically adjusting the buffer and the amount of cached data for audio and video packets, the system can better cope with different delay situations and maintain audio and video synchronization. Finally, the audio and video packets stored in the playback buffer are processed for audio and video synchronization to obtain and play the target audio and video data. This step ensures that the audio and video content presented to the user is effectively synchronized, thereby providing a better viewing experience.

[0029] In general, this technical solution achieves better response to changes in the network environment and reduces audio and video asynchrony by comprehensively utilizing the overall delay evaluation data obtained based on the basic delay evaluation data and the real-time network status evaluation data, and dynamically adjusting the playback buffer size and the cached data volume of the audio and video data packets in a timely manner, thereby improving the user experience quality when watching videos.

[0030] In one example, the overall delay evaluation data of a historical time period of a preset time length from the current moment is determined based on the basic delay evaluation data and the real-time network status evaluation data, including: determining the basic delay evaluation data of the data packets that should theoretically be received within the historical time period of the preset time length from the current moment; determining the average delay time, total delay time and maximum delay time of the audio data packets, and the average delay time, total delay time and maximum delay time of the video data packets within the historical time period based on the determined basic delay evaluation data, to obtain first delay evaluation data; determining the network fluctuation data and delay jitter data of the transmission network within the historical time period of the preset time length from the current moment; calculating the overall network evaluation value of the transmission network within the historical time period based on the determined network fluctuation data and delay jitter data, and using the overall network evaluation value as the second delay evaluation data; integrating the first delay evaluation data and the second delay evaluation data to obtain the overall delay evaluation data.

[0031] In this example, by analyzing the basic delay evaluation data and the real-time network status evaluation data, the following steps are performed: First, the basic delay evaluation data of the data packets that should be received within the historical time period of the preset time from the current moment is determined. This means determining the basic delay of the data packets that should be received theoretically during this period. Then, according to the determined basic delay evaluation data, the delay statistics of the audio and video data packets in the historical time period are calculated, including the average delay time, the total number of delayed data packets, and the maximum delay time. In this way, the first delay evaluation data can be obtained, which provides detailed information about the historical delay situation. Then, the network fluctuation data and delay jitter data of the transmission network in the historical time period of the preset time from the current moment are determined. This step helps to understand the fluctuations and instability of the network in the past period of time. Then, the determined network fluctuation data and delay jitter data are used to calculate the overall network evaluation value of the transmission network. This provides an evaluation of the overall performance of the network, including the comprehensive impact of fluctuations and jitters. Finally, the first delay evaluation data and the second delay evaluation data are integrated to obtain the overall delay evaluation data. In this way, factors such as basic delay, network fluctuations, and delay jitter can be comprehensively considered to provide a more accurate reference for subsequent audio and video synchronization adjustments. The entire technical solution takes into account factors such as basic delay and network fluctuations, as well as statistics on historical delays, to achieve an accurate assessment of the overall delay situation, laying a good foundation for the subsequent adjustment of the cache size of audio and video data packets, thereby helping to more effectively maintain audio and video synchronization and enhance the user experience when watching videos.

[0032] In one example, according to the determined network fluctuation data and delay jitter data, calculating the overall network evaluation value of the transmission network in the historical time period includes: using a first formula to calculate the determined network fluctuation data and delay jitter data to obtain the overall network evaluation value N, wherein the first formula is: , wherein the is the mean of network fluctuation data, is the variance of network fluctuation data, is the skewness of network volatility data, is the kurtosis of network fluctuation data, is the mean value of the delay jitter data, is the variance of the delay jitter data, The skewness of the delay jitter data, the is the kurtosis of the delay jitter data, and stated is the preset weight coefficient, and stated is the preset index parameter.

[0033] In this example, first, the first formula is used to calculate the determined network fluctuation data and delay jitter data to obtain the overall network evaluation value N. This formula comprehensively considers various factors that affect network transmission conditions, and formulates an evaluation system for various factors that affect network transmission conditions, thereby comprehensively evaluating the overall performance of the transmission network. By calculating the overall network evaluation value N, the stability and reliability of the transmission network in the historical time period can be more accurately understood. This helps the system better adapt to the ever-changing network environment and provides a more reliable basis for subsequent audio and video synchronization adjustments.

[0034] In one example, dynamically adjusting the playback buffer size based on the overall delay evaluation data includes: determining whether the overall network evaluation value in the overall delay evaluation data is lower than a first threshold; if the overall network evaluation value is lower than the first threshold, expanding the playback buffer according to a second formula; if the overall network evaluation value is not lower than the first threshold, shrinking the playback buffer according to a third formula, wherein the second formula is: , wherein the third formula is: , Indicates the new playback buffer size, Indicates the current playback buffer size. It represents the sum of the average delay time of the audio data packet and the average delay time of the video data packet. represents the first delay threshold, Represents the expansion weight value, Represents the shrinkage weight value.

[0035] In this example, the playback buffer size is dynamically adjusted based on the overall delay evaluation data, and the steps are as follows: First, determine whether the overall network evaluation value is lower than the first threshold. If the overall network evaluation value is lower than the first threshold, perform an expansion adjustment; otherwise, perform a contraction adjustment. In this way, the system can flexibly adjust the buffer size according to the real-time situation of network performance. For the expansion adjustment, the second formula is used to calculate the new playback buffer size. This formula comprehensively considers the current playback buffer size, the sum of the average delay time of the audio data packet and the video data packet, and the expansion weight value to ensure that there is enough buffer to cope with potential delay fluctuations when the network performance is good. For the contraction adjustment, the third formula is used to calculate the new playback buffer size. This formula similarly comprehensively considers the current playback buffer size, the sum of the average delay time of the audio data packet and the video data packet, and the contraction weight value to appropriately reduce the buffer size when the network performance decreases to avoid playback delays caused by over-buffering. The entire technical solution achieves more precise control of audio and video synchronization by dynamically adjusting the playback buffer size according to network performance. This can better cope with delay changes in different network environments, thereby improving the user experience quality when watching videos.

[0036] In one example, dynamically adjusting the amount of cached data of the audio data packet based on the overall delay evaluation data includes: based on the total value of delayed data packets and the maximum delay time of the audio data packets in the historical time period in the overall delay evaluation data, and using a fourth formula to calculate the total value of delayed data packets and the maximum delay time to obtain the current amount of cached data of the audio data packet; wherein the fourth formula is: , Indicates the new audio data packet buffer size. Indicates the current audio data packet buffer size. Indicates the current audio data packet delay. Indicates the expected audio data packet delay. Indicates the maximum delay time, Indicates the audio packet adjustment factor.

[0037] In this example, the steps of dynamically adjusting the amount of cached data of the audio data packet based on the overall delay evaluation data are as follows: First, based on the total value of the delayed data packets of the audio data packet in the historical time period and the maximum delay time in the overall delay evaluation data, the fourth formula is used to calculate the current amount of cached data of the audio data packet. This step takes into account the historical delay situation, as well as the expected audio data packet delay and the maximum delay time, so as to dynamically adjust the amount of cached audio data packets. The fourth formula comprehensively considers the current audio data packet delay and the maximum delay time, as well as the preset audio data packet adjustment coefficient, and calculates the new amount of cached audio data packets. This ensures that the amount of cached audio data packets can be adjusted in time under different delay conditions to maintain the stability of audio and video synchronization and user experience. The entire technical solution realizes flexible control of audio data packet transmission and caching by dynamically adjusting the amount of cached audio data packets according to the overall delay evaluation data. This helps the system better cope with delay changes in different network environments, improves the accuracy of audio and video synchronization and the quality of experience of users when watching videos.

[0038] In one example, dynamically adjusting the cached data volume of the video data packet based on the overall delay evaluation data includes: based on the total value of delayed data packets and the maximum delay time of the video data packets in the historical time period in the overall delay evaluation data, and using a fourth formula to calculate the total value of delayed data packets and the maximum delay time to obtain the current cached data volume of the video data packet; wherein the fourth formula is: , Indicates the new video data packet buffer size, Indicates the current video data packet cache size, Indicates the current video data packet delay. Indicates the expected video packet delay, Indicates the maximum delay time, Indicates the video packet adjustment factor.

[0039] In this example, the steps for dynamically adjusting the cached data volume of the video data packet based on the overall delay evaluation data are as follows: First, based on the total value of the delayed data packets of the video data packet in the historical time period and the maximum delay time in the overall delay evaluation data, the fourth formula is used to calculate the current cached data volume of the video data packet. This step takes into account the historical delay situation, as well as the expected video data packet delay and maximum delay time, so as to dynamically adjust the cached volume of the video data packet. The fourth formula comprehensively considers the current video data packet delay and maximum delay time, as well as the preset video data packet adjustment coefficient, and calculates the new video data packet cache volume. This ensures that the cached volume of the video data packet can be adjusted in time under different delay conditions to maintain the stability of audio and video synchronization and user experience. The entire technical solution realizes flexible control of video data packet transmission and caching by dynamically adjusting the cached data volume of the video data packet according to the overall delay evaluation data. This helps the system better cope with delay changes in different network environments, improves the accuracy of audio and video synchronization and the quality of experience of users when watching videos.

[0040] In one example, audio and video synchronization processing is performed on the audio data packets and video data packets stored in the playback buffer to obtain and play target audio and video data, including: performing timestamp extraction processing on the audio data packets to obtain timestamp information of the audio data packets; performing timestamp extraction processing on the video data packets to obtain timestamp information of the video data packets; performing timestamp matching processing on the audio data packets and the video data packets to obtain audio and video synchronization timestamp alignment information; performing time stretching processing on the audio data packets to adjust the audio playback speed without changing the audio frequency to obtain stretched audio data packets; performing smooth transition processing on the stretched audio data packets to obtain smoothed audio data packets; based on the audio and video synchronization timestamp alignment information, decoding processing is performed on the smoothed audio data packets and the video data packets to obtain and play the target audio and video data.

[0041] In this example, the steps of performing audio and video synchronization processing on the audio data packets and video data packets stored in the playback buffer are as follows: First, the audio data packets are subjected to timestamp extraction processing to obtain the timestamp information of the audio data packets. Then, the video data packets are subjected to similar timestamp extraction processing to obtain the timestamp information of the video data packets. These two steps ensure that both the audio and video data are correctly marked with time. Next, the timestamp matching processing of the audio data packets and the video data packets is performed to obtain the audio and video synchronization timestamp alignment information. This ensures that the time axes of the audio and video are aligned during playback, avoiding the situation where the audio and video are out of sync. Subsequently, the audio data packets are subjected to time stretching processing to adjust the audio playback speed without changing the audio frequency. This ensures that the audio pitch remains unchanged while matching it with the playback speed of the video data packets. Then, the stretched audio data packets are subjected to smooth transition processing to ensure the continuity and smoothness of the audio playback. This prevents obvious breaks or jumps during audio playback. Finally, based on the audio and video synchronization timestamp alignment information, the audio data packets and video data packets after the smooth transition are decoded to obtain and play the target audio and video data. This ensures that the audio and video content presented to users is effectively synchronized, providing a better viewing experience. The entire technical solution achieves accurate synchronization of audio and video by extracting, matching and processing the timestamps of audio and video data packets, and improves the quality of user experience when watching videos through smooth transition processing.

[0042] It should be noted that the existing technology compares the actual audio delay with the audio target delay, and accelerates or decelerates the audio data according to the comparison result to achieve synchronous playback of audio data and video data. However, this method often causes changes in audio frequency when faced with delay adjustment, resulting in problems such as reduced sound quality and unnatural playback effects.

[0043] The technical solution provided by the embodiment of the present invention adjusts the audio playback speed without changing the audio frequency by time stretching the audio data packet, thereby obtaining a stretched audio data packet. This method avoids frequency changes caused by accelerated or decelerated playback, ensures the naturalness and authenticity of audio playback, and avoids problems such as pitch deviation or sound quality distortion. Furthermore, smoothing the transition of the stretched audio data packet can effectively reduce the audio mutation and unnaturalness caused by time stretching, so that the audio data remains stable and consistent during the adjusted playback process.

[0044] In addition, during the research on audio and video synchronization through accelerated or decelerated playback, it was also found that this method may not guarantee the ideal synchronization effect when dealing with sudden large delays, and there are potential technical problems. Specifically, when there is a sudden large delay, relying solely on accelerated or decelerated playback for synchronization may not be enough to effectively solve the problem of audio and video asynchrony. This is because a large delay will cause the gap between the audio and video tracks to be too large, and simple speed adjustment cannot bridge this gap in a short period of time, which may cause the synchronization state of audio and video to fluctuate continuously, thus seriously affecting the user experience. To this end, this application provides the following technical solutions: In one example, after the playback buffer is expanded and adjusted according to the second formula, the synchronization method further includes: adding a preset audio cache amount based on the current audio data packet cache amount, and directly adding a preset video cache amount based on the current video data packet cache amount; after the playback buffer is contracted and adjusted according to the third formula, the synchronization method further includes: reducing the preset audio cache amount based on the current audio data packet cache amount, and directly reducing the preset video cache amount based on the current video data packet cache amount.

[0045] In this example, it is found that only relying on accelerated or decelerated playback cannot effectively solve the synchronization problem caused by sudden large delays. To this end, the following technical solutions are proposed: First, after the playback buffer is expanded and adjusted according to the second formula, a preset audio buffer will be added on the basis of the current audio data packet buffer, and a preset video buffer will be added on the basis of the current video data packet buffer. In this way, the buffer capacity can be directly increased on the basis of the extended buffer to better cope with sudden delays, thereby improving the effect of audio and video synchronization. Secondly, after the playback buffer is contracted and adjusted according to the third formula, the preset audio buffer will be reduced on the basis of the current audio data packet buffer, and the preset video buffer will be directly reduced. In this way, when the network condition returns to normal, the size of the buffer can be reduced in time to reduce the playback delay and return to the ideal audio and video synchronization state as soon as possible. Considering that the playback buffer size has been dynamically adjusted based on the overall delay evaluation data, it means that there is a problem with the network state. For the sake of conservatism, this application first directly adds a preset video buffer based on the current video data packet buffer, and then dynamically adjusts it in combination with the formula calculation to ensure normal playback effect.

[0046] Overall, this technical solution optimizes for sudden large delays by dynamically adjusting the amount of audio and video cache, combined with expansion and contraction adjustment strategies. This helps maintain the stability of audio and video synchronization in the face of network fluctuations, thereby improving the quality of user viewing experience.

[0047] In addition, a large delay will also cause the misalignment and loss of audio and video data packets, and simple speed adjustment cannot make up for the loss of these data packets, which may cause the audio and video to be out of sync, such as screen freezes or sound distortion, seriously affecting the user experience. To this end, this application provides the following technical solutions: In one example, after calculating the reception time of the data packets according to the adaptive filtering algorithm to obtain the basic delay evaluation data of each data packet, the synchronization method further includes: determining the data packets whose delay time exceeds a second delay threshold based on the basic delay evaluation data of each data packet; and retransmitting the data packets whose delay time exceeds the second delay threshold.

[0048] In this example, the following technical solution is proposed to solve the problem of audio and video data packet misalignment and loss caused by large delays: First, the receiving time of the data packet is calculated and processed according to the adaptive filtering algorithm to obtain the basic delay evaluation data of each data packet. Then, based on the basic delay evaluation data of each data packet, the data packet whose delay time exceeds the second delay threshold is determined. This step is to identify data packets with large delays for subsequent processing. Next, the data packets whose delay time exceeds the second delay threshold are retransmitted. By resending data packets with large delays, the loss or misalignment of data packets can be compensated, thereby reducing the situation of audio and video asynchrony, such as picture freeze or sound distortion, and further improving the user experience. Overall, this technical solution solves the problem of audio and video asynchrony caused by large delays by identifying and retransmitting data packets with large delays. This helps to maintain the stability of audio and video synchronization, reduce the unpleasant experience that users may encounter when watching videos, and improve the quality of the user's viewing experience.

[0049] Reference Figure 2 , an embodiment of the present invention provides a synchronization device for audio and video data, including: The receiving unit 1 is used to receive audio data packets and video data packets of audio and video, and cache the audio data packets and video data packets in a play buffer.

[0050] The calculation unit 2 is used to calculate the receiving time of the data packet according to the adaptive filtering algorithm to obtain the basic delay evaluation data of each data packet.

[0051] The identification unit 3 is used to identify the network fluctuation data and delay jitter data of the transmission network of the audio data packet through a particle filter to obtain real-time network status evaluation data.

[0052] The first evaluation unit 4 is used to determine the overall delay evaluation data of a historical time period of a preset time length from the current moment according to the basic delay evaluation data and the real-time network status evaluation data.

[0053] The second evaluation unit 5 is used to dynamically adjust the playback buffer size and the cached data volume of the audio data packet and / or video data packet based on the overall delay evaluation data.

[0054] The synchronization unit 6 is used to perform audio and video synchronization processing on the audio data packets and video data packets stored in the play buffer to obtain and play the target audio and video data.

[0055] In this embodiment, for the specific implementation of each unit in the above device embodiment, please refer to the above method embodiment, which will not be repeated here.

[0056] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0057] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0058] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0059] In summary, the existing methods cannot accurately evaluate the reception delay of audio and video data packets when facing complex network environments, resulting in inaccurate audio and video synchronization adjustment. The present application calculates the reception time of the data packet through an adaptive filtering algorithm to obtain basic delay evaluation data. This step establishes a basic timeline, allowing the system to more accurately evaluate the delay of each data packet. This can provide a reliable foundation for subsequent synchronization processing, ensure more accurate synchronization adjustment, and effectively improve the effect of audio and video synchronization.

[0060] The existing technology lacks effective monitoring of real-time network fluctuations and delay jitter, resulting in poor audio and video synchronization when the network condition is unstable. This application uses a particle filter to identify the transmission network fluctuation data and delay jitter data of audio data packets, and the system can obtain network status evaluation data in real time. This real-time monitoring mechanism enables the system to better adapt to the ever-changing network environment, adjust the synchronization strategy in time, reduce the phenomenon of audio and video asynchrony caused by network fluctuations, and improve the user viewing experience.

[0061] The fixed-size buffer in the existing technology cannot adapt to changes in network conditions. As a result, when the network fluctuates greatly, the buffer may not be sufficient to cope with delay changes, affecting the synchronization effect. Based on the overall delay evaluation data, the system can dynamically adjust the playback buffer size and the cache amount of audio and video data packets. This can more flexibly respond to different delay situations and ensure that audio and video synchronization is maintained under various network conditions. Through dynamic adjustment, the system can effectively reduce the situation of audio and video asynchrony and improve the user experience when watching videos.

[0062] In addition, in the prior art, simply accelerating or decelerating the playback of audio data will cause changes in the audio frequency, resulting in a decrease in sound quality and unnatural playback effects. The audio data packet is time-stretched to adjust the audio playback speed without changing the audio frequency, ensuring the naturalness and authenticity of the audio playback. The stretched audio data packet is then smoothed to reduce the audio mutations and unnaturalness caused by time stretching. Through these processes, the system can maintain audio quality while adjusting the playback speed, providing a more natural and smooth audio playback effect, thereby effectively improving the user experience of audio and video synchronization.

[0063] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0064] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0065] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for synchronizing audio and video data, characterized in that: The following steps are involved: Receive audio data packets and video data packets of audio and video, and cache the audio data packets and video data packets in a play buffer; The receiving time of the data packet is calculated and processed according to the adaptive filtering algorithm to obtain the basic delay evaluation data of each data packet; Identify network fluctuation data and delay jitter data of the transmission network of the audio data packet through a particle filter to obtain real-time network status evaluation data; Determine overall delay evaluation data of a historical time period of a preset length from the current moment according to the basic delay evaluation data and the real-time network status evaluation data; Dynamically adjusting the playback buffer size and the cached data volume of the audio data packet and / or the video data packet based on the overall delay evaluation data; Perform audio and video synchronization processing on the audio data packet and the video data packet stored in the play buffer to obtain and play the target audio and video data.

2. The synchronization method according to claim 1, characterized in that: Determining overall delay evaluation data of a historical time period of a preset length from the current moment according to the basic delay evaluation data and the real-time network status evaluation data includes: Determine basic delay evaluation data of data packets that should theoretically be received within a historical time period of a preset length from the current moment; Determine, according to the determined basic delay evaluation data, the average delay time of the audio data packets, the total value of the delayed data packets, and the maximum delay time, as well as the average delay time of the video data packets, the total value of the delayed data packets, and the maximum delay time, to obtain first delay evaluation data; Determine network fluctuation data and delay jitter data of the transmission network within a historical time period of a preset length from the current moment; Calculate the overall network evaluation value of the transmission network in the historical time period according to the determined network fluctuation data and delay jitter data, and use the overall network evaluation value as the second delay evaluation data; The first delay assessment data and the second delay assessment data are integrated to obtain the overall delay assessment data.

3. The synchronization method according to claim 1, characterized in that: Calculating the overall network evaluation value of the transmission network in the historical time period according to the determined network fluctuation data and delay jitter data, including: The first formula is used to calculate the determined network fluctuation data and delay jitter data to obtain the overall network evaluation value N, Among them, the first formula is: , Among them, the is the mean of network fluctuation data, is the variance of network fluctuation data, is the skewness of network volatility data, is the kurtosis of network fluctuation data, is the mean value of the delay jitter data, is the variance of the delay jitter data, The skewness of the delay jitter data, the is the kurtosis of the delay jitter data, and stated is the preset weight coefficient, and stated is the preset index parameter.

4. The synchronization method according to claim 1, characterized in that: Dynamically adjusting the playback buffer size based on the overall delay evaluation data includes: Determining whether the overall network evaluation value in the overall delay evaluation data is lower than a first threshold; If the overall network evaluation value is lower than the first threshold, the playback buffer is expanded and adjusted according to the second formula; if the overall network evaluation value is not lower than the first threshold, the playback buffer is contracted and adjusted according to the third formula. Wherein, the second formula is: , Wherein, the third formula is: , Indicates the new playback buffer size, Indicates the current playback buffer size. It represents the sum of the average delay time of the audio data packet and the average delay time of the video data packet. represents the first delay threshold, Represents the expansion weight value, Represents the shrinkage weight value.

5. The synchronization method according to claim 1, characterized in that: Dynamically adjusting the buffered data amount of the audio data packet based on the overall delay evaluation data includes: Based on the total value of delayed data packets and the maximum delay time of the audio data packets in the historical time period in the overall delay evaluation data, and using a fourth formula to calculate the total value of delayed data packets and the maximum delay time, a current cached data volume of the audio data packets is obtained; Wherein, the fourth formula is: , Indicates the new audio data packet buffer size. Indicates the current audio data packet buffer size. Indicates the current audio data packet delay. Indicates the expected audio data packet delay. Indicates the maximum delay time, Indicates the audio packet adjustment factor.

6. The synchronization method according to claim 1, characterized in that: Dynamically adjusting the buffered data amount of the video data packet based on the overall delay evaluation data includes: Based on the total value of delayed data packets and the maximum delay time of the video data packets in the historical time period in the overall delay evaluation data, and using the fourth formula to calculate the total value of delayed data packets and the maximum delay time, the current cached data volume of the video data packets is obtained; Wherein, the fourth formula is: , Indicates the new video data packet buffer size, Indicates the current video data packet cache size, Indicates the current video data packet delay. Indicates the expected video packet delay, Indicates the maximum delay time, Indicates the video packet adjustment factor.

7. The synchronization method according to claim 1, characterized in that: Performing audio and video synchronization processing on the audio data packet and the video data packet stored in the play buffer to obtain and play the target audio and video data includes: Performing timestamp extraction processing on the audio data packet to obtain timestamp information of the audio data packet; Performing timestamp extraction processing on the video data packet to obtain timestamp information of the video data packet; Performing timestamp matching processing on the audio data packet and the video data packet to obtain audio and video synchronization timestamp alignment information; Performing time stretching processing on the audio data packet, adjusting the audio playback speed without changing the audio frequency, to obtain a stretched audio data packet; Performing smoothing processing on the stretched audio data packet to obtain a smoothed audio data packet; Based on the audio and video synchronization timestamp alignment information, the smoothed audio data packet and the video data packet are decoded to obtain and play the target audio and video data.

8. A synchronization device for audio and video data, characterized in that: include: A receiving unit, used for receiving audio data packets and video data packets of audio and video, and caching the audio data packets and video data packets in a play buffer; A calculation unit, used to calculate and process the receiving time of the data packet according to the adaptive filtering algorithm to obtain basic delay evaluation data of each data packet; An identification unit, used to identify network fluctuation data and delay jitter data of the transmission network of the audio data packet through a particle filter to obtain real-time network status evaluation data; A first evaluation unit, configured to determine overall delay evaluation data of a historical time period of a preset length from a current moment according to the basic delay evaluation data and the real-time network status evaluation data; A second evaluation unit, configured to dynamically adjust the playback buffer size and the cached data volume of the audio data packet and / or the video data packet based on the overall delay evaluation data; The synchronization unit is used to perform audio and video synchronization processing on the audio data packet and the video data packet stored in the play buffer to obtain and play the target audio and video data.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images