A method and apparatus for measuring audio and video playback environment delay

Abstract

This invention provides a method and apparatus for measuring latency in audio and video playback environments. The method includes: connecting test audio directly to an audio input port via an audio output port to obtain a recording of the test audio; performing loop analysis on the recording to calculate the sum of the audio output latency and the audio input latency; playing a test video and calculating the time difference between the played sound and the video to obtain the audio output latency; subtracting the audio output latency from the sum of the audio output latency and the audio input latency to obtain the audio input latency; directly measuring various latency parameters that can correct audio and video playback results, ultimately improving the user experience.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for measuring latency in audio and video playback environments. Background Technology

[0002] Embedded systems typically exhibit audio output latency, audio input latency, and video output latency. These latency issues significantly impact applications requiring synchronized audio and video output. For example, during real-time scoring of a user's performance, asynchronous audio and video playback can lead to inaccurate scoring and a poor user experience. Specifically, audio output latency causes sound to lag behind the video, affecting the user's ability to follow along with the audio. Audio input latency causes the scoring algorithm to receive the user's audio too late, resulting in comparisons with incorrect standard information and impacting scoring accuracy. Video output latency causes the video to lag behind the audio, affecting the user's ability to follow along with the video and significantly degrading the user experience.

[0003] One corrective measure is to introduce a delay calculation into the real-time scoring algorithm. That is, the recording sample is received at a specific time T, and the score is calculated after a delay of (T+a). The best matching value between the recording sample at time T and the standard pitch at times Ta to T+a is calculated, and the best matching value is used as the singing score at time T. This is feasible to some extent, but it introduces two problems: one is that it sacrifices the accuracy of the score, because the score calculated in this way is the best score; the other is that it sacrifices real-time performance, because the introduced delay time 'a' is generally greater than the sum of the system's audio output delay time and audio input delay time, which leads to a delay in the entire scoring process.

[0004] Another effective and correct correction method is to first measure the various delays in the system and compensate for them in the algorithm. For example, the compensation process for input delay is as follows: Audio input delay means that an audio sample sung by a user at time T, which the user believes is sung at the standard pitch at time T, will not be received by the algorithm until time T+a (where a is the audio input delay). Therefore, the algorithm needs to correct the audio sample received at time T+a, marking its timestamp as (T+ab), where b is the estimated system input delay. This correction method requires an effective estimation method to measure the various delays in the system. Existing common estimation methods generally start from chip parameters and the design principles of software modules, such as buffer size, to calculate the delay value. However, this method requires accurate acquisition of various parameters, which is often difficult to obtain in complex systems; furthermore, this method is also difficult to handle software systems with dynamic environments and dynamic loads. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and apparatus for measuring latency in audio and video playback environment, which can accurately measure the time of various latency that can correct the audio and video playback results, and ultimately improve the user experience.

[0006] In a first aspect, the present invention provides a method for measuring latency in an audio / video playback environment, comprising:

[0007] The test audio is directly connected to the audio input port through the audio output port to obtain the recording of the test audio. Loop analysis is performed on the recording to calculate the sum of the audio output delay time and the audio input delay time.

[0008] Play the test video, calculate the time difference between the sound and the picture, obtain the audio output delay time, and subtract the audio output delay time from the sum of the audio output delay time and the audio input delay time to obtain the audio input delay time.

[0009] Secondly, the present invention provides an audio / video playback environment delay measurement device, comprising:

[0010] The audio delay summation module is used to directly input the test audio through the audio output port to the audio input port, obtain the recording of the test audio, perform loop analysis on the recording, and calculate the sum of the audio output delay time and the audio input delay time.

[0011] The audio output / input delay module is used to play test videos, calculate the time difference between the played sound and the picture to obtain the audio output delay time, and subtract the audio output delay time from the sum of the audio output delay time and the audio input delay time to obtain the audio input delay time.

[0012] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages: The present invention can skip the details of specific embedded systems, directly simulate user behavior, measure the time of various delays that can correct audio and video playback results, and ultimately improve the user experience.

[0013] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0014] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0015] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention;

[0016] Figure 2 This is a schematic diagram illustrating the sum of the output delay time and the input delay time in this invention;

[0017] Figure 3 This is a schematic diagram of the test video used in this invention;

[0018] Figure 4 This is a schematic diagram of the pie chart rotation angle in this invention;

[0019] Figure 5 This is a schematic diagram of the audio output delay time in this invention;

[0020] Figure 6 This is a schematic diagram illustrating the video output delay time of the main screen and the secondary screen in this invention;

[0021] Figure 7 This is a schematic diagram of the device in Embodiment 3 of the present invention. Detailed Implementation

[0022] The overall concept of the technical solution in this application is as follows:

[0023] Connect the audio output to the audio input, record the specified audio, perform loop analysis, and calculate the sum of the audio output delay time and the audio input delay time;

[0024] The output latency of the main screen is very low and negligible. Therefore, based on the audio-visual synchronization difference, the audio output latency time can be calculated, and then the audio input latency time can be obtained.

[0025] For dual-screen devices, the secondary screen often experiences video output latency, while the primary screen generally has lower video output latency. In this case, the same clock source readings can be displayed on both the primary and secondary screens. By comparing the difference in clock values ​​on both screens at a certain moment, the video output latency of the secondary screen can be estimated.

[0026] This invention provides a method for measuring latency in audio and video playback environments, comprising:

[0027] The test audio is directly input to the audio input port through the audio output port to obtain the recording of the test audio. Loop analysis is performed on the recording to calculate the sum of the audio output delay time and the audio input delay time.

[0028] Play the test video, calculate the time difference between the sound and the picture, obtain the audio output delay time, and subtract the audio output delay time from the sum of the audio output delay time and the audio input delay time to obtain the audio input delay time.

[0029] In a preferred embodiment, the audio and video playback environment delay measurement method further includes: displaying a timer of a clock source on the screen to be tested, playing a timer of the same clock source on a reference screen, obtaining the numerical difference between the timer of the screen to be tested and the timer of the reference screen at a preset time, and obtaining the video output delay time of the screen to be tested.

[0030] In a preferred embodiment, the step of "displaying a timer from a clock source on the screen to be tested, playing a timer from the same clock source on a reference screen, obtaining the numerical difference between the timer on the screen to be tested and the timer on the reference screen at a preset time, and obtaining the video output delay time of the screen to be tested" specifically refers to:

[0031] A timer from a clock source is displayed on the screen to be tested, and a timer from the same clock source is played on the reference screen. The images of each screen at a preset time are captured by taking screenshots. The timer readings of each screen are obtained according to OCR. The video output delay time of the screen to be tested is obtained based on the difference between the timer readings of the reference screen and the timer readings of the screen to be tested.

[0032] In a preferred embodiment, the step of "directly inputting the test audio to the audio input port through the audio output port, obtaining a recording of the test audio, performing loop analysis on the recording, and calculating the sum of the audio output delay time and the audio input delay time" is further specified as follows:

[0033] Connect the audio output port directly to the audio input port;

[0034] Play the test audio, which is directly input to the audio input port through the audio output port, and start recording. The test audio contains pulses.

[0035] After playback is complete, locate the occurrence time of each pulse in the recording, calculate the time difference between the occurrence time of each pulse and the occurrence time of each pulse in the original test audio, and take the average of each time difference as the sum of the audio output delay time and the audio input delay time.

[0036] Furthermore, the test audio is set with pulses at uniform time intervals, the time intervals being greater than the estimated delay time.

[0037] In a preferred embodiment, the step "playing the test video, calculating the time difference between the played sound and the picture, and obtaining the audio output delay time" is further specified as follows:

[0038] Play a pre-set test video while simultaneously recording audio and video; the test video includes a pie chart, and the time it takes for the pie chart to rotate one full circle is equal to the time interval between two adjacent pulses;

[0039] Analyze the audio in the recording to find each pulse;

[0040] Analyze the video recording, calculate the rotation angle of the pie chart at the pulse arrival time, and calculate the audio output delay time based on the rotation angle of the pie chart.

[0041] Furthermore, the "calculation of the rotation angle of the pie chart at the pulse arrival time" specifically refers to:

[0042] Obtain the pie chart from the video recording corresponding to the pulse arrival time;

[0043] The pie chart image is binarized, and the edge image of the pie chart is extracted; the edge image contains a concentric sector, and the angle of the central angle of the sector is the rotation angle of the pie chart.

[0044] The method for calculating the central angle of the sector is as follows: Set a rectangle with the center point C of the pie chart as the center. The side length of the rectangle is less than the diameter of the pie chart. Calculate the intersection points of the rectangle and the sector to obtain two intersection points A and B. A and B are located on the two radii of the sector, respectively.

[0045] Construct a Cartesian coordinate system with the center C as the origin. Since line segment AC always overlaps with the y-axis, the value of ∠ACB can be calculated by calculating the slope of line segment BC; this value is the rotation angle of the pie chart.

[0046] Furthermore, the phrase "calculating the audio output delay time based on the rotation angle of the pie chart" specifically refers to:

[0047] The audio output delay time is calculated using the formula t = rotation angle / 360 * 1000, where t is the audio output delay time in milliseconds.

[0048] Example 1

[0049] like Figure 1 As shown, this embodiment provides a method for measuring latency in an audio / video playback environment, including:

[0050] Connect the audio output port directly to the audio input port; play the test audio, which is directly connected to the audio input port through the audio output port, and start recording. The test audio contains pulses. After playback is complete, find the occurrence time of each pulse in the recording, calculate the time difference between the occurrence time of each pulse and the occurrence time of each pulse in the original test audio, and take the average of the time differences as the sum of the audio output delay time and the audio input delay time. The test audio contains pulses with uniform time intervals, and the time interval is greater than the estimated delay time.

[0051] Play a pre-set test video while simultaneously recording audio and video. The test video displays a pie chart; the time it takes for the pie chart to rotate one full circle is equal to the time interval between two adjacent pulses. Analyze the audio recording to locate each pulse. Analyze the video recording to calculate the rotation angle of the pie chart at the pulse arrival time. Calculate the audio output delay time based on the pie chart's rotation angle. Subtract the audio output delay time from the sum of the audio output delay time and the audio input delay time to obtain the audio input delay time. Calculate the audio output delay time using the formula t = rotation angle / 360 * 1000, where t is the audio output delay time in milliseconds.

[0052] A timer from a clock source is displayed on the screen to be tested, and a timer from the same clock source is played on the reference screen. The images of each screen at a preset time are captured by taking screenshots. The timer readings of each screen are obtained according to OCR. The video output delay time of the screen to be tested is obtained based on the difference between the timer readings of the reference screen and the timer readings of the screen to be tested.

[0053] The calculated audio input delay time can be applied to the real-time scoring of the user's singing. The audio input delay means that the audio sample sung by the user at time T will not be received until time T+c (where c is the audio input delay mentioned above). Therefore, by directly comparing the audio sample received at time T+c with the standard pitch at time T, the user's real-time pitch score can be obtained.

[0054] When a device has an audio output delay, the output audio and the played video will be out of sync. In this case, the video can be delayed, meaning the video at time T is played at time T+d, where d is the audio output delay time, thus ensuring synchronization between the audio and video.

[0055] In dual-screen or multi-screen displays, the screen with the smaller display latency is used as the reference screen, and the remaining screens are used as the screens to be tested. For example, in general, in a dual-screen setup, the main screen has a smaller latency and is used as the reference screen, while the secondary screen is used as the screen to be tested. Alternatively, in a dual-screen setup, the screen with the larger latency, which is used for multi-layer rendering, is used as the screen to be tested, while the other screen is used as the reference screen. This approach can also be applied to situations with more than two screens.

[0056] When a device has a video output delay, the output audio and the played video will be out of sync. It's necessary to pre-render the output video, for example, rendering and playing the video at time T at time Te (playing it earlier), where e is the video output delay time, to ensure synchronization between the video and audio.

[0057] Additionally, when the device exhibits both video and audio output delays, it's necessary to determine which delay is more severe. If the video output delay *e* is greater than the audio output delay *d*, the video to be played at time *T* will be played at time *T-e+d*, effectively playing the video earlier. Conversely, if the video output delay *e* is less than the audio output delay *d*, the video to be played at time *T+de* will be played later, effectively delaying the video playback.

[0058] In applications that require synchronized audio and video output, such as real-time singing scoring applications, the above-calculated audio input delay time, audio output delay time, and video frame delay time can be used to ensure the accuracy of the scoring and the synchronization between the playback video and audio, greatly improving the user experience.

[0059] Example 2

[0060] like Figure 2 Based on Example 1, the method for calculating the sum of the measured audio output delay time and the audio input delay time is as follows:

[0061] 1. Connect the audio output of the embedded system to the audio input of the embedded system; for example, connect the audio output to the audio input, or by pointing the speaker towards the microphone;

[0062] 2. Play the set test audio and start recording; the test audio is set with pulses at uniform intervals, which are longer than the estimated delay time. For example, if the estimated delay time is 500ms, then the pulse interval must be longer than 500ms; an energy value is calculated every 20ms, and whether the pulse occurs is determined based on whether the energy value is greater than the preset threshold.

[0063] 3. After playback is complete, find the occurrence time of each pulse in the recorded audio, calculate the time difference between the occurrence time of each pulse and the pulse occurrence time of the originally set test audio, and take the average value. This value is the sum of the audio output delay and audio input delay of the embedded system, that is, the total delay time.

[0064] Measure audio output delay:

[0065] By playing a bipbop test video, audio and video recording can be performed simultaneously using a device (e.g., a camera);

[0066] Analyze the audio in the recording to find the points where each pulse occurs; calculate the energy value every 20ms, and determine whether a pulse occurs based on whether the energy value is greater than a preset threshold; the playback time of this test video does not need to be limited, and the playback time can be set to a few seconds;

[0067] The video frame corresponding to the moment the pulse occurs is obtained, and the rotation angle of the pie chart at this time is calculated. The time it takes for the pie chart to rotate one full turn is set as the time interval between two adjacent pulses. The audio output delay time can be calculated by combining the pie chart rotation angle. Audio output delay time = pie chart rotation angle / 360 * 1000, and the unit of audio output delay time is milliseconds.

[0068] Subtracting the audio output delay time from the sum of the audio output delay time and the audio input delay time, we get the audio input delay time as equal to 1.

[0069] like Figure 3 As shown, the video feed is a pie chart. It starts with a white fill, taking 1 second to fill a full circle (meaning the time interval between two pulses is 1 second), emitting a "bip" sound when it's completely filled. Then it fills with gray, taking 1 second to fill a full circle, emitting a "bop" sound when it's completely filled. Then it returns to a white fill state… and repeats continuously. This example could use a simpler video, emitting a single pulse sound with each full circle (no need for "bip" and "bop" distinctions, and no need to change the fill color; a fixed fill color is sufficient).

[0070] like Figure 4 As shown, obtain the rotation angle in the pie chart:

[0071] Obtain the pie chart from the video recording corresponding to the pulse arrival time;

[0072] The pie chart image is binarized, and the edge image of the pie chart is extracted; the edge image contains a concentric sector, and the angle of the central angle of the sector is the rotation angle of the pie chart.

[0073] Set a rectangle with the center point C of the pie chart as the center. The side length of the rectangle is smaller than the diameter of the pie chart. Calculate the intersection points of the rectangle and the sector, and obtain two intersection points A and B. A and B are exactly located on the two radii of the sector, respectively.

[0074] Construct a Cartesian coordinate system with the center C as the origin. Since line segment AC always overlaps with the y-axis, the value of ∠ACB can be calculated by calculating the slope of line segment BC; this value is the rotation angle of the pie chart.

[0075] like Figure 5 As shown, when the pulse occurs, the pie chart has already exceeded the 12 o'clock position by 30 degrees. Since there is no delay, the pie chart should be exactly at the 12 o'clock position. Therefore, the angles that exceed the position are all the angles of the delay portion. The audio output delay time can be calculated as: 30 / 360x1000 = 83 milliseconds.

[0076] like Figure 6As shown, in a dual-screen device, if the video output latency of the main screen is small and can be ignored, while the video output latency of the secondary screen is large, the calculation method for the video output latency of the secondary screen is as follows;

[0077] The embedded system time is displayed in real time on both the main and secondary screens; simultaneously, screenshots are taken of each screen at a specific moment, and OCR is used to obtain the time readings for each screen; the difference between these readings is calculated, which is the video output delay time of the secondary screen. For cases with two or more screens, the above method can be used to obtain the video output delay time of the corresponding screen with video output delay.

[0078] For example:

[0079] The implementation is as follows (taking Android as an example):

[0080] 1. Create an Activity, which will be displayed on the home screen;

[0081] 2. Place a Text control on the Activity;

[0082] 3. Create a presentation, which will be displayed on the secondary screen;

[0083] 4. Place a Text control on the Presentation;

[0084] 5. Set a timer on the main thread that triggers every 10ms;

[0085] 6. When the timer is triggered, read the current system time via the Java API;

[0086] 7. Set the read values ​​to the Activity Text control and Presentation Text control respectively.

[0087] The above program can also be adapted to the actual business scenario; for example, to achieve rich visual effects on Android, Virtual Display technology may be used, in which case the program implementation would be adjusted as follows:

[0088] 1. Create an Activity, which will be displayed on the home screen;

[0089] 2. Place a Text control on the Activity;

[0090] 3. Create a VirtualDisplay and display it on the secondary screen;

[0091] 4. Create a Presentation and specify that the Presentation should be displayed on a VirtualDisplay;

[0092] 5. Place a Text control on the Presentation;

[0093] 6. Set a timer on the main thread that triggers every 10ms;

[0094] 7. When the timer is triggered, read the current system time through the Java API; set the read value to the Activity Text control and the Presentation Text control respectively.

[0095] Based on the same inventive concept, this application also provides an apparatus corresponding to the method in Embodiment 1, as detailed in Embodiment 3.

[0096] Example 3

[0097] like Figure 7 As shown, this embodiment provides an audio / video playback environment delay measurement device, including:

[0098] The audio delay summation module is used to directly input the test audio through the audio output port to the audio input port, obtain the recording of the test audio, perform loop analysis on the recording, and calculate the sum of the audio output delay time and the audio input delay time.

[0099] The audio output / input delay module is used to play test videos, calculate the time difference between the played sound and the picture to obtain the audio output delay time, and subtract the audio output delay time from the sum of the audio output delay time and the audio input delay time to obtain the audio input delay time.

[0100] In a preferred embodiment, the audio and video playback environment delay measurement device further includes: a video delay module, used to display a timer of a clock source on the screen to be tested, play a timer of the same clock source on a reference screen, obtain the numerical difference between the timer of the screen to be tested and the timer of the reference screen at a preset time, and obtain the video output delay time of the screen to be tested.

[0101] In Example 3, the calculation of each delay time is the same as that of each delay time in Example 1.

[0102] Since the apparatus described in Embodiment 3 of this invention is the same apparatus used to implement the method of Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of the apparatus based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All apparatuses used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0103] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for measuring delay in an audio-visual playback environment, the method comprising: The method comprises the following steps: connecting the audio output port directly to the audio input port, playing a test audio, the test audio being inputted directly to the audio input port through the audio output port, and starting recording, the test audio being provided with pulses, playing being completed, finding the time of occurrence of each pulse in the recording, calculating the time difference between the time of occurrence of each pulse and the time of occurrence of each pulse in the original test audio, and taking the average of each time difference as the sum of the audio output delay time and the audio input delay time playing a test video, calculating the time difference between the sound and the picture during playing, obtaining the audio output delay time, and subtracting the audio output delay time from the sum of the audio output delay time and the audio input delay time to obtain the audio input delay time.

2. The method of claim 1, wherein, The method further comprises the following steps: displaying a timer of a clock source in a to-be-detected screen, playing the timer of the same clock source in a reference screen, obtaining the numerical difference between the timer of the to-be-detected screen and the timer of the reference screen at a preset time, and obtaining the video output delay time of the to-be-detected screen.

3. The method of claim 2, wherein, The step of "displaying a timer of a clock source in a to-be-detected screen, playing the timer of the same clock source in a reference screen, obtaining the numerical difference between the timer of the to-be-detected screen and the timer of the reference screen at a preset time, and obtaining the video output delay time of the to-be-detected screen" is specifically as follows: displaying a timer of a clock source in a to-be-detected screen, playing the timer of the same clock source in a reference screen, obtaining the picture of each screen at a preset time through screen capture, obtaining the reading of the timer of each screen according to OCR, and obtaining the video output delay time of the to-be-detected screen according to the numerical difference between the reading of the timer of the reference screen and the reading of the timer of the to-be-detected screen.

4. The method of claim 1, wherein, The test audio is provided with pulses with uniform time intervals, and the time intervals are greater than the estimated delay time.

5. The method of claim 1, wherein, The step of "playing a test video, calculating the time difference between the sound and the picture during playing, and obtaining the audio output delay time" is further specifically as follows: playing a set test video while recording and taking a picture; the test video plays a pie chart, and the time for the pie chart to rotate one circle is equal to the time interval between two adjacent pulses; analyzing the audio in the recording to find each pulse; analyzing the video in the recording to calculate the rotation angle of the pie chart at the time when the pulse arrives, and calculating the audio output delay time according to the rotation angle of the pie chart.

6. The method of claim 5, wherein, The step of "calculating the rotation angle of the pie chart at the time when the pulse arrives" is specifically as follows: obtaining the pie chart picture in the recording corresponding to the time when the pulse arrives; binarizing the pie chart picture to extract the edge image of the pie chart; the edge image contains a concentric sector, and the angle of the sector center angle is the rotation angle of the pie chart; setting a rectangle with the center point C of the pie chart as the center, and the side length of the rectangle being less than the diameter of the pie chart, calculating the intersection points of the rectangle and the sector to obtain two intersection points A and B; A and B are located on the two radii of the sector, respectively; constructing a Cartesian coordinate system with the center point C as the origin, since the line segment AC always overlaps with the y-axis, the value of ∠ACB can be calculated by calculating the slope of the line segment BC; the value is the rotation angle of the pie chart.

7. The method of claim 5 or 6, wherein, The step of "calculating the audio output delay time according to the rotation angle of the pie chart" is specifically as follows: The audio output delay time is calculated according to the formula t = rotation angle / 360*1000, wherein t is the audio output delay time in milliseconds.

8. A device for measuring the delay of audio and video playback environment, characterized in that, The application comprises: an audio delay summation module, which is used to connect the audio output port directly to the audio input port, play a test audio, input the test audio directly into the audio input port through the audio output port, start recording, set pulses in the test audio, complete playing, find the occurrence time of each pulse in the recording, calculate the time difference between the occurrence time of each pulse and the occurrence time of each pulse in the original test audio, and take the average of each time difference as the sum of the audio output delay time and the audio input delay time; an audio output input delay module, which is used to play a test video, calculate the time difference between the played sound and the picture, obtain the audio output delay time, subtract the audio output delay time from the sum of the audio output delay time and the audio input delay time to obtain the audio input delay time.

9. The apparatus for measuring delay of an audio / video playback environment according to claim 8, wherein, The application further comprises: a video delay module, which is used to display a timer of a clock source in a to-be-detected screen, play the timer of the same clock source in a reference screen, and obtain the value difference between the timer of the to-be-detected screen and the timer of the reference screen at a preset time to obtain the video output delay time of the to-be-detected screen.

Images