Buffer area adaptive adjustment method and system for audio and video synchronization

Abstract

The invention discloses a buffer adaptive adjustment method and system for audio and video synchronization. The method comprises the following steps: calculating a delay value of each media data source to a media server in each time period; all delay values of each media data source in the current time period are extracted from the data table to determine the maximum delay reference value and the delay standard deviation of each media data source in the current time period, so that the smoothing coefficient of each media data source in the current time period is obtained; setting a first time window and a second time window, extracting all delay values of each media data source in the first time window and the second time window from the data table so as to determine maximum delay reference values of the media data source in the first time window and the second time window, and predicting maximum delay prediction values of the media data sources in the next time period; the buffer size of each media data source is adjusted. The invention relates to the technical field of computers, and can realize self-adaptive adjustment of the size of a media buffer area in a delay violent fluctuation scene, and shorten network delay while ensuring audio and video synchronous experience.

Description

Technical Field

[0001] The present invention relates to a buffer zone adaptive adjustment method and system for audio and video synchronization, and relates to the technical field of computers. Background Art

[0002] Currently, in the online precision operation scenarios that integrate air, space, and land, the complexity of the network itself, the rapid changes in user locations, the strong fluctuation of delays, and the extreme susceptibility to cosmic rays lead to increased network delays and the risk of packet loss.

[0003] To address the aforementioned situation, dynamic buffers are typically set in scenarios with drastic latency fluctuations and audio and video synchronization requirements to ensure a synchronized audio and video playback experience. However, existing dynamic buffer setting strategies are mostly based on simple algorithmic fitting implementations. These strategies fail to account for complex and volatile network conditions and the characteristics of independent networks, resulting in inflexible buffer sizes. For example, if the satellite audio and aircraft video transmission channels are not on the same path, and the network latency of the video link suddenly increases compared to the audio link, improper buffer resizing can lead to data overflow due to the difference in data accumulation between the two parties, resulting in an audio buffer full and an audio and video out-of-sync experience. For example, if the packet loss rate of satellite video transmission is suddenly increased due to cosmic rays, failure to adjust the buffer in a timely manner can also lead to buffer overflow, resulting in data loss and audio and video out-of-sync. Another example is if the network latency and packet loss rate of a large number of aircraft and satellites suddenly increase, failure to adjust the buffer in a timely manner can also lead to long periods of high latency, significantly impacting precision operations. This shows that incorrect and untimely buffer size settings will lead to inconsistent audio and video playback and high latency problems, and therefore cannot meet the low latency and high synchronization requirements in scenarios with drastic latency fluctuations and audio and video synchronization requirements.

[0004] Therefore, how to achieve adaptive adjustment of the audio and video media buffer size in scenarios with drastic delay fluctuations, thereby minimizing network delay while ensuring a synchronized audio and video experience, has become a key technical issue that technical personnel are focusing on. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a buffer adaptive adjustment method and system for audio and video synchronization, which can realize adaptive adjustment of the audio and video media buffer size in scenarios with drastic delay fluctuations, thereby shortening network delay as much as possible while ensuring the audio and video synchronization experience.

[0006] In order to achieve the above object, the present invention provides a buffer adaptive adjustment method for audio and video synchronization, comprising:

[0007] Step 1: Calculate the delay value from each media data source to the media server in each time period and save it in the data table;

[0008] Step 2: At the end of each time period, extract all delay values ​​stored by each media data source in the current time period from the data table to determine its maximum delay reference value and delay standard deviation in the current time period, thereby obtaining a smoothing coefficient for each media data source in the current time period;

[0009] Step 3: Set a first time window and a second time window, extract all delay values ​​stored in the first time window and the second time window for each media data source from the data table to determine its maximum delay reference value in the first time window and the second time window, and then predict the maximum delay prediction value of each media data source in the next time period based on the smoothing coefficient of the media data source in the current time period;

[0010] Step 4: Based on the maximum delay prediction value of the media data source in the next time period, adjust the buffer size of each media data source.

[0011] In order to achieve the above object, the present invention further provides a buffer adaptive adjustment system for audio and video synchronization, comprising:

[0012] A delay calculation device is used to calculate the delay value of each media data source to the media server in each time period and save it in a data table;

[0013] a smoothing coefficient calculation device for extracting, at the end of each time period, from a data table all delay values ​​stored for each media data source in the current time period, to determine its maximum delay reference value and delay standard deviation in the current time period, thereby obtaining a smoothing coefficient for each media data source in the current time period;

[0014] The maximum delay prediction device is configured to set a first time window and a second time window, extract all delay values ​​stored by each media data source in the first time window and the second time window from a data table to determine its maximum delay reference value in the first time window and the second time window, and then predict the maximum delay prediction value of each media data source in the next time period based on the smoothing coefficient of the media data source in the current time period;

[0015] The buffer adjustment device is used to adjust the buffer size of each media data source based on the maximum delay prediction value of the media data source in the next time period.

[0016] In order to achieve the above object, the present invention further provides a computing device, comprising:

[0017] memory and processor;

[0018] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the buffer adaptive adjustment method for audio and video synchronization are implemented.

[0019] In order to achieve the above object, the present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the buffer adaptive adjustment method for audio and video synchronization.

[0020] Compared with the prior art, the present invention has the following advantages: when the network environment delay increases and the packet loss rate increases, the present invention can quickly increase the buffer zone to ensure the synchronous experience of audio and video playback of each media data source. When the network environment becomes better, the buffer zone can be reduced to reduce the delay of audio and video playback of each media data source. In the present invention, the media server periodically passively measures the network delay size of the node or transit plane and stores it in a database. The computing server extracts this data to calculate the quartiles, interquartile range, etc. to filter out abnormal delay values, and then dynamically adjusts the smoothing coefficient according to the reference value of the current delay to dynamically calculate the expected value of the delay size. Finally, the media server takes a multiple of the expected value of the delay size as the current buffer size. The present invention also dynamically calculates the buffer adjustment amount at the current moment based on the physical model of the tuned mass damper to dynamically adjust the corresponding buffer size according to the network status of the specific node. The present invention is suitable for online precision operation scenarios that integrate air, space, and ground. The server uses real-time statistics of the network status and combines historical and current network conditions to intelligently decide the buffer size, thereby minimizing the delay while ensuring the synchronous experience of audio and video. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The figure is a flow chart showing a method for adaptively adjusting a buffer for audio and video synchronization according to an exemplary embodiment of the present invention.

[0022] Figure 2 It is a specific flow chart showing an exemplary embodiment of the present invention, which divides each time period into multiple time periods, and then adjusts the buffer size of the next time period at the end of each time period based on the tuned mass damper physical model and the delay change of the current time period.

[0023] Figure 3 The figure is a structural diagram of a buffer adaptive adjustment system for audio and video synchronization according to an exemplary embodiment of the present invention.

[0024] Figure 4 It is a structural diagram of a computer device according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0026] like Figure 1 As shown, the present invention provides a buffer adaptive adjustment method for audio and video synchronization, comprising:

[0027] Step 1: Calculate the delay value from each media data source to the media server in each time period and save it in the data table;

[0028] The time period can be set to 10ms;

[0029] Step 2: At the end of each time period, extract all delay values ​​stored by each media data source in the current time period from the data table to determine its maximum delay reference value and delay standard deviation in the current time period, thereby obtaining a smoothing coefficient for each media data source in the current time period;

[0030] The time period can be set to 1 minute;

[0031] Step 3: Set a first time window and a second time window, extract all delay values ​​stored in the first time window and the second time window for each media data source from the data table to determine its maximum delay reference value in the first time window and the second time window, and then predict the maximum delay prediction value of each media data source in the next time period based on the smoothing coefficient of the media data source in the current time period;

[0032] The first time window and the second time window represent short-term and long-term windows, respectively. For example, the first time window can be set to a historical time period within 5 minutes from the current time, and the second time window can be set to a historical time period within 1 hour from the current time.

[0033] Step 4: Based on the maximum delay prediction value of the media data source in the next time period, adjust the buffer size of each media data source.

[0034] In step 1, to ensure the accuracy of the buffer size prediction, abnormal delay values ​​in the data table can also be deleted, including:

[0035] All delay values ​​of each media data source within a time period are periodically retrieved from the data table. The 25th percentile Q1 and the 75th percentile Q3 are extracted, and the corresponding interquartile range (IQR) is calculated: IQR = Q3 - Q1. This determines the normal value range of the delay value of each media data source within the time period: [Q1 - 1.5 * IQR, Q3 + 1.5 * IQR]. Delay values ​​that are not within the normal value range within the time period are deleted from the data table.

[0036] In step 2 or step 3, determining the maximum delay reference value in the current time period (or the first time window, or the second time window) may further include:

[0037] The 99th percentile digit is selected from all delay values ​​of the media data source in the current time period (or the first time window, or the second time window) as the maximum delay reference value.

[0038] In step 2, the calculation formula for obtaining the smoothing coefficient of each media data source in the current time period can be as follows: Among them, lag t and σ are the maximum delay reference value and delay standard deviation of each media data source in the current time period, respectively. t , α t-1 They are the smoothing coefficients of each media data source in the current time period and the previous time period, δ is the single smoothing coefficient adjustment, and γ(lag t ) is the smoothing coefficient adjustment coefficient. When the maximum delay reference value exceeds 2 times the delay standard deviation, the smoothing coefficient is increased to enhance the sensitivity to recent data. When the maximum delay reference value is lower than 0.5 times the delay standard deviation, the smoothing coefficient is reduced to improve the prediction stability.

[0039] The standard deviation is the arithmetic mean of the squared deviations from the mean (i.e., the variance). It is calculated by first squaring each difference from the mean, then adding up all the squared differences and dividing by the total, and finally taking the square root of the result.

[0040] In step 3, the calculation formula for predicting the maximum delay prediction value of each media data source in the next time period can be: lag′ t+1 =α t *lag1+(1-α t )*lag2, lag1, lag2 are the maximum delay reference values ​​of each media data source in the first time window and the second time window respectively, lag′ t+1 is the maximum delay prediction value of each media data source in the next time period, α t It is the smoothing coefficient of each media data source in the current time period.

[0041] Step 4 may further include:

[0042] The buffer size of each media data source in the next time period is set to P times its maximum delay prediction value in the next time period. P can be set according to actual business needs, for example, P is 1.2.

[0043] like Figure 2 As shown, the present invention can further divide each time period (e.g., 1 minute) into multiple time periods (e.g., 10 ms). Then, at the end of each time period, the buffer size for the next time period is adjusted based on the tuned mass damper physical model and the delay variation of each media data source in the current time period. The method also includes:

[0044] Step A1: Calculate the delay variation of each media data source in the current time period, and calculate the TMD acceleration of each media data source in the current time period accordingly: Among them, y ‘ n ’ Indicates the TMD acceleration of each media data source in the current time period, y n 、y ’ n Respectively represent the TMD displacement and instantaneous speed of each media data source in the current time period. When n=1, y1, y ’ The value of 1 is 0, Fext is the delay change of each media data source in the current time period, c is the damping coefficient, its value can be set to 0.1, k is the spring stiffness, its value can be set to 0.3, M is the buffer size of each media data source in the current time period, and m is the TMD virtual mass, its value can be set to 1.5;

[0045] Step A2: Based on the TMD acceleration of each media data source in the current time period, predict the TMD displacement of each media data source in the next time period: y ‘ n+1 =y ‘ n +y ‘ n ’ *Δt,y n+1 =y n +y ‘ n+1 *Δt+0.5*y ‘ n ’ *(Δt) 2 , where y n+1 、y ‘ n+1are the TMD displacement and instantaneous velocity of each media data source in the next time period, respectively, and Δt is the time step corresponding to each time period;

[0046] Step A3: Calculate the buffer change of each media data source in the next time period based on the TMD displacement of each media data source in the next time period: Δx n+1 It is the amount of change in the buffer of each media data source in the next time period, and the buffer size of each media data source in the next time period is adjusted accordingly.

[0047] like Figure 3 As shown, the present invention provides a buffer adaptive adjustment system for audio and video synchronization, comprising:

[0048] A delay calculation device is used to calculate the delay value of each media data source to the media server in each time period and save it in a data table;

[0049] a smoothing coefficient calculation device for extracting, at the end of each time period, from a data table all delay values ​​stored for each media data source in the current time period, to determine its maximum delay reference value and delay standard deviation in the current time period, thereby obtaining a smoothing coefficient for each media data source in the current time period;

[0050] The maximum delay prediction device is configured to set a first time window and a second time window, extract all delay values ​​stored by each media data source in the first time window and the second time window from a data table to determine its maximum delay reference value in the first time window and the second time window, and then predict the maximum delay prediction value of each media data source in the next time period based on the smoothing coefficient of the media data source in the current time period;

[0051] The buffer adjustment device is used to adjust the buffer size of each media data source based on the maximum delay prediction value of the media data source in the next time period.

[0052] See also Figure 4 , Figure 4 4 is a block diagram of a computing device 400 according to an exemplary embodiment of this specification. Components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.

[0053] The computing device 400 also includes an access device 440 that enables the computing device 400 to communicate via one or more networks 460. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 402.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and the like.

[0054] In one embodiment of the present specification, the above components of the computing device 400 and Figure 4 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 4 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0055] Computing device 400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. Computing device 400 may also be a mobile or stationary server or cloud server, etc.

[0056] The processor 420 is configured to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned buffer adaptive adjustment method for audio and video synchronization.

[0057] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the aforementioned method for adaptively adjusting the buffer for audio and video synchronization are based on the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the aforementioned method for adaptively adjusting the buffer for audio and video synchronization.

[0058] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned buffer adaptive adjustment method for audio and video synchronization.

[0059] The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the aforementioned method for adaptively adjusting the buffer for audio and video synchronization. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the aforementioned method or system for adaptively adjusting the buffer for audio and video synchronization.

[0060] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is instructed to execute the steps of the above-mentioned buffer adaptive adjustment method for audio and video synchronization.

[0061] The above is an illustrative solution of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the aforementioned method for adaptively adjusting the buffer for audio and video synchronization are based on the same concept. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the aforementioned method or system for adaptively adjusting the buffer for audio and video synchronization.

[0062] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0063] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium.

[0064] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0065] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A buffer adaptive adjustment method for audio and video synchronization, characterized in that: Includes: Step 1: Calculate the delay value from each media data source to the media server in each time period and save it in the data table; Step 2: At the end of each time period, extract all delay values ​​stored by each media data source in the current time period from the data table to determine its maximum delay reference value and delay standard deviation in the current time period, thereby obtaining a smoothing coefficient for each media data source in the current time period; Step 3: Set a first time window and a second time window, extract all delay values ​​stored in the first time window and the second time window for each media data source from the data table to determine its maximum delay reference value in the first time window and the second time window, and then predict the maximum delay prediction value of each media data source in the next time period based on the smoothing coefficient of the media data source in the current time period; Step 4: Based on the maximum delay prediction value of the media data source in the next time period, adjust the buffer size of each media data source.

2. The method according to claim 1, characterized in that Step one also includes: All delay values ​​of each media data source within a time period are periodically retrieved from the data table. The 25th percentile Q1 and the 75th percentile Q3 are extracted, and the corresponding interquartile range (IQR) is calculated: IQR = Q3 - Q1. This determines the normal value range of the delay value of each media data source within the time period: [Q1 - 1.5 * IQR, Q3 + 1.5 * IQR]. Delay values ​​that are not within the normal value range within the time period are deleted from the data table.

3. The method according to claim 1, characterized in that In step 2 or step 3, determining a maximum delay reference value in the current time period or the first time window or the second time window includes: The 99th percentile bit is selected from all delay values ​​of the media data source in the current time period or the first time window or the second time window as the maximum delay reference value.

4. The method according to claim 1, wherein In step 2, the calculation formula for obtaining the smoothing coefficient of each media data source in the current time period is as follows: Among them, lag t and σ are the maximum delay reference value and delay standard deviation of each media data source in the current time period, respectively. t , α t-1 They are the smoothing coefficients of each media data source in the current time period and the previous time period, δ is the single smoothing coefficient adjustment, and γ(lag t ) is the smoothing coefficient adjustment factor.

5. The method according to claim 1, wherein In step 3, the calculation formula for predicting the maximum delay prediction value of each media data source in the next time period is: lag′ t+1 =α t *lag1+(1-α t )*lag2, lag1, lag2 are the maximum delay reference values ​​of each media data source in the first time window and the second time window respectively, lag′ t+1 is the maximum delay prediction value of each media data source in the next time period, α t It is the smoothing coefficient of each media data source in the current time period.

6. The method according to claim 1, characterized in that Step 4 includes: The buffer size of each media data source in the next time period is set to P times its maximum delay prediction value in the next time period.

7. The method according to claim 1, characterized in that Each time period is divided into multiple time periods. At the end of each time period, the buffer size for the next time period is adjusted based on the tuned mass damper physical model and the delay change of each media data source in the current time period. This includes: Step A1: Calculate the delay variation of each media data source in the current time period, and calculate the TMD acceleration of each media data source in the current time period accordingly: Among them, y ‘ n ’ Indicates the TMD acceleration of each media data source in the current time period, y n 、y ’ n Respectively represent the TMD displacement and instantaneous speed of each media data source in the current time period. When n=1, y1, y ’ The value of 1 is 0, Fext is the delay change of each media data source in the current time period, c is the damping coefficient, k is the spring stiffness, M is the buffer size of each media data source in the current time period, and m is the TMD virtual mass; Step A2: Based on the TMD acceleration of each media data source in the current time period, predict the TMD displacement of each media data source in the next time period: y ‘ n+1 =y ‘ n +y ‘ n ’* Δt,y n+1 =y n +y ‘ n+1 *Δt+0.5*y ‘ n ’ *(Δt) 2 , where y n+1 、y ‘ n+1 are the TMD displacement and instantaneous velocity of each media data source in the next time period, respectively, and Δt is the time step corresponding to each time period; Step A3: Calculate the buffer change of each media data source in the next time period based on the TMD displacement of each media data source in the next time period: Δx n+1 It is the amount of change in the buffer of each media data source in the next time period, and the buffer size of each media data source in the next time period is adjusted accordingly.

8. A buffer adaptive adjustment system for audio and video synchronization, characterized in that: Includes: A delay calculation device is used to calculate the delay value of each media data source to the media server in each time period and save it in a data table; a smoothing coefficient calculation device for extracting, at the end of each time period, from a data table all delay values ​​stored for each media data source in the current time period, to determine its maximum delay reference value and delay standard deviation in the current time period, thereby obtaining a smoothing coefficient for each media data source in the current time period; The maximum delay prediction device is configured to set a first time window and a second time window, extract all delay values ​​stored by each media data source in the first time window and the second time window from a data table to determine its maximum delay reference value in the first time window and the second time window, and then predict the maximum delay prediction value of each media data source in the next time period based on the smoothing coefficient of the media data source in the current time period; The buffer adjustment device is used to adjust the buffer size of each media data source based on the maximum delay prediction value of the media data source in the next time period.

9. A computing device, characterized in that include: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the buffer adaptive adjustment method for audio and video synchronization according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that It stores computer-executable instructions, which, when executed by a processor, implement the steps of the buffer adaptive adjustment method for audio and video synchronization as described in any one of claims 1 to 7.

Images