Method, device, computing device, and storage medium for evaluating audio stuttering rate

Abstract

The present invention provides a method, apparatus, computing device, and storage medium for evaluating audio stuttering. The method includes: setting an observation window for the audio to be evaluated, where the observation window includes one or more audio frames; determining, for each observation window within a unit of time, whether the observation window contains a stuttering window based on preset conditions, and counting the number of stuttering windows included; and evaluating the stuttering rate, where the stuttering rate is the ratio of the cumulative number of stuttering windows determined within a unit of time to the cumulative number of all observation windows within the unit of time. This method can quantify stuttering and accurately evaluate the stuttering rate.

Description

Technical Field

[0001] The present invention relates to audio communication technology, and in particular to a method, apparatus, computing device and storage medium for evaluating audio stuttering rate. Background Art

[0002] The development of internet communication technologies has led to the increasing use of Internet telephony, streaming media, and live audio and video broadcasts. Real-time audio communication (RTC) is an indispensable feature in these applications. All RTAC applications are internet-based. However, unstable network environments can lead to packet loss, latency, and jitter, which can cause abnormal audio data packets received by the receiving end, resulting in audio stuttering. Audio stuttering itself is a key indicator of the audio experience, and the severity of audio stuttering directly impacts the user experience.

[0003] In the related art, dynamic jitter buffer technology is usually used at the audio playback end to eliminate the impact of the above-mentioned network factors on audio fluency. However, after the audio data packet arrives at the receiving end, it will undergo a series of operations during the process of being transmitted from the jitter buffer of the receiving end to the sound card for playback, such as actively discarding audio frames, clearing or resetting the jitter buffer, etc. These operations may cause audio frame loss, resulting in audio jams. Therefore, it is necessary to evaluate and quantify the jams that may be caused by this process. The quantification of this jam can, for example, be used as a reference indicator to measure the pros and cons of the design of related modules in audio products, thereby promoting improvements in the smoothness experience of audio products. The quantification of this jam can also be used as a feedback so that the sending end and / or the receiving end can respond accordingly based on the feedback, thereby improving the jam condition and improving audio fluency.

[0004] Current methods for evaluating audio fluency primarily include subjective and objective assessment methods. Subjective assessment methods primarily assess fluency based on subjective auditory feedback from testers. However, prolonged subjective evaluations can easily lead to misjudgments by testers, and this method is inefficient and costly, making it unsuitable for widespread adoption. Objective assessment methods primarily focus on a holistic assessment of audio quality. This holistic assessment is primarily based on standards such as ITU-TP.862 PESQ and ITU-RBS.1387 PEAQ. However, these standards do not separately quantify the severity of audio stuttering, and the assessment of audio stuttering is only one component of the overall audio quality assessment. This objective assessment method cannot intuitively reflect the extent of audio stuttering and is therefore unsuitable for verifying and improving the smoothness experience of audio products. Summary of the Invention

[0005] The present invention proposes a method, apparatus, computing device, and storage medium for evaluating audio stuttering rate, which are used to more accurately evaluate the audio stuttering rate to solve one or more of the above-mentioned problems and other problems.

[0006] According to a first aspect of the present invention, a method for evaluating audio stuttering rate is provided. The method comprises:

[0007] Setting an observation window for the audio to be evaluated, where the observation window includes one or more audio frames;

[0008] For each observation window within a unit time, determine whether the observation window contains a stuck window according to preset conditions, and count the stuck windows included; and

[0009] A jam rate is evaluated, where the jam rate is a ratio of a cumulative number of jam windows determined within the unit time to a cumulative number of all observation windows within the unit time.

[0010] According to a second aspect of the present invention, a device for evaluating audio jam rate is provided. The device comprises:

[0011] a setting module, configured to set an observation window for the audio to be evaluated, wherein the observation window includes one or more audio frames;

[0012] a judgment module, configured to judge, for each observation window within a unit time, whether the observation window contains a stuck window according to preset conditions, and count the stuck windows contained; and

[0013] An evaluation module is used to evaluate a jam rate, wherein the jam rate is a ratio of a cumulative number of jam windows determined within the unit time to a cumulative number of all observation windows within the unit time.

[0014] According to a third aspect of the present invention, a computing device is provided. The computing device includes: a memory and a processor, wherein the memory is configured to store computer-executable instructions; and the processor is configured to execute the computer-executable instructions to perform any embodiment of the method for evaluating audio stuttering rate described above.

[0015] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which computer-executable instructions are stored. When the instructions are executed by a computer, the computer executes any embodiment of the above-mentioned method for evaluating audio stuttering rate.

[0016] The method, apparatus, computing device, and storage medium for evaluating audio stuttering rate of the present invention have the following advantages:

[0017] This paper identifies two main scenarios that cause stuttering: continuous empty frame insertion and forced frame drop. Based on an observation window model, it quantifies the subjective stuttering experience that these identified scenarios may cause as a stuttering window count. Identifying the main scenarios that cause stuttering improves the accuracy of stuttering rate assessment, and the use of stuttering windows simplifies the computational process for quantifying and assessing stuttering.

[0018] Furthermore, when the audio frame switches from an empty frame to a normal coded frame, the silence or DTX scenario of the transmitter is identified, and the interference of the empty frame introduced in the silence or DTX scenario on the quantification and evaluation of the jamming is eliminated, thereby further improving the accuracy of the jamming rate evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 An exemplary system for evaluating audio stuttering rate according to an embodiment of the present invention is shown.

[0020] Figure 2 An exemplary flow chart of a method for evaluating an audio jam rate according to an embodiment of the present invention is shown.

[0021] Figure 3A and 3B An example of a jam window according to an embodiment of the present invention and examples of audio frames in a scenario of continuous insertion of empty frames and a scenario of forced frame loss are shown.

[0022] Figure 4 Another exemplary flow chart of a method for evaluating an audio jam rate according to an embodiment of the present invention is shown.

[0023] Figure 5 Examples of audio frames in a normal play state and audio frames in a mute or DTX state according to an embodiment of the present invention are shown.

[0024] Figure 6 An exemplary flow chart of a process for determining whether a transmitting end is in a mute state according to an embodiment of the present invention is shown.

[0025] Figure 7 A specific exemplary application scenario of the method for evaluating the audio jam rate according to an embodiment of the present invention is shown.

[0026] Figure 8 An exemplary computing device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0027] In the following description, the present invention is further described in detail with reference to the accompanying drawings and exemplary embodiments. However, those skilled in the art will appreciate that the present invention is not limited to these exemplary embodiments.

[0028] The method, apparatus, computing device, and storage medium for evaluating the audio stuttering rate provided in this application can be applied to any application involving audio communication, such as live broadcast, anchor PK, voice chat rooms, etc.

[0029] See also Figure 1 , Figure 1 An exemplary system for evaluating audio jam rate is shown. A voice message containing audio frames is transmitted from a transmitting end to a receiving end through a network. As mentioned above, in the jitter buffer of the receiving end, the audio frames will undergo a series of processing, such as frame loss, frame insertion, etc. Then, the audio frames are transmitted to the decoding module for encoding and decoding, and the decoding module will decode (decode) or repair (recover) the audio data packets. Optionally, other processing modules can be set after the decoding module, such as Figure 1 The playback buffer (PlaybackBuffer) in the audio frame is used to perform other logical processing on the audio frame. The audio frame is finally transmitted to the sound card device for playback. During this series of operations, various processing of the audio frame may cause audio stuttering. Therefore, the characteristic information of the audio frame (such as sequence number, codec status, timestamp, etc.) can be analyzed after the audio frame is decoded and before it reaches the sound card to identify stuttering and quantify the severity of the stuttering.

[0030] The present invention envisions setting up a jamming assessment module after the jitter buffer and decoding module, so as to well cover various scenarios that cause jamming to occur, so as to more accurately identify and quantify jamming and evaluate the jamming rate.

[0031] According to one aspect of the present invention, a method 100 for evaluating audio freeze rate is provided. Figure 2 As shown. Method 100 can be performed by Figure 1 The method 100 may be performed by the jamming assessment module shown in FIG. 1 , or may be performed by other suitable hardware, software, or a combination of hardware and software. The method 100 includes steps 101 to 103.

[0032] In step 101 , an observation window is set for audio to be evaluated, where the observation window includes one or more audio frames.

[0033] In an embodiment of the present invention, the jamming assessment module may obtain audio frames per unit time from, for example, a playback buffer, and then identify and quantify jamming in an observation window manner.

[0034] The unit time can be set as needed, for example, to 30 seconds, 1 minute, 5 minutes, etc. The audio frames within the unit time can be acquired all at once or as a data stream during the method execution. The unit time can be considered as an evaluation period for the stutter rate. Typically, at the end of the evaluation period, the stutter evaluation module calculates the audio stutter rate for that evaluation period.

[0035] The acquired audio frames are allocated to a set observation window. Typically, the observation window has a fixed size and does not overlap. The size of the observation window can be pre-set based on the actual application.

[0036] Specifically, according to the known acoustic statistics theory, the normal speaking frequency of Chinese people is generally about 200 to 300 words per minute, that is, one word every 200 to 300 ms. The human ear will have a clear perception of the interruption of voice playback at the word level, and this perceptible voice interruption is regarded as voice jamming. Therefore, for example, the size of the observation window can be set to 200ms. When the length of an audio frame is 20ms, the observation window with a duration of 200ms includes 10 audio frames. In actual applications, the size of the observation window is set as needed. An observation window can include one or more audio frames, and is not limited to 10 audio frames, and the duration of the observation window is not limited to 200ms. For example, the size of the observation window can be determined according to the language (such as Chinese, English, etc.) and the length of the audio frame.

[0037] See also Figure 3A , Figure 3A An example of an observation window is shown. As shown in the figure, the size of the observation window is set to 200ms, and the audio frames are divided into windows such as window W0, window W1, window W2, etc. according to time, and each observation window includes 10 audio frames.

[0038] In the embodiment of the present invention, the use of the observation window can not only well quantify the jamming that can be perceived by the human ear from the dimension of "words", but also simplify the calculation process of jamming quantification and jamming rate evaluation.

[0039] In step 102, for each observation window within a unit time, it is determined according to a preset condition whether the observation window contains a jamming window, and the number of jamming windows included is counted.

[0040] In this embodiment of the present invention, audio frames are analyzed based on the characteristics of different scenarios that cause jamming to identify these scenarios. Based on these scenarios, corresponding jamming evaluation logic is used to calculate and update the number of jamming windows. A jamming window is an observation window in which audio frames contained therein may cause jamming.

[0041] From the perspective of audio frames, there are two main scenarios that cause lag: continuous insertion of empty frames and forced frame loss.

[0042] (1) Continuously inserting empty frame scenes

[0043] As mentioned above, the decoding module primarily performs two operations: decoding and repair. Normal audio frames containing audio content (also called normally decoded frames) are decoded, and the resulting audio data stream does not experience any lag. When no normal audio frames can be decoded, the decoding module performs a repair operation. This repair operation can occur for a variety of reasons, including when no normal audio frames that can be decoded are received within the specified timeframe or when audio frames that should have been decoded are discarded. This repair operation generates empty frames—audio frames devoid of any audio information. These empty frames are inserted into the audio data stream. Continuous insertion of these empty frames can cause lag. In actual audio communication, decoding and repairing occur alternately. Over a period of time, the more repair operations are performed, the more empty frames are inserted, and the more noticeable the resulting lag. This continuous insertion of empty frames is also the most common cause of lag.

[0044] (2) Forced frame loss scenario

[0045] Typically, the jitter buffer stores audio frames in ascending order of their sequence numbers (also known as Seq). For example, audio frames 999, 1000, and 1001 are stored in the jitter buffer in ascending order of their Seqs and are subsequently decoded and played in this order. However, in poor network conditions, significant packet loss or jitter can cause data packets to arrive later at the receiver. To accommodate newly arrived audio packets, the jitter buffer at the receiver forcibly discards some audio frames, resulting in discontinuities in the Seqs of two adjacent, normally decoded frames. When the number of discarded audio frames exceeds a certain limit, meaning the Seq difference between two adjacent audio frames is too large, audio lag can occur.

[0046] Therefore, it is necessary to identify the two main scenarios that cause jamming and calculate the corresponding jamming parameters, such as the number of jamming windows and the number of accumulated observation windows. For the sake of the layout of this application, the accumulated observation windows will be explained later.

[0047] In the embodiment of the present invention, first, a scene of continuously inserting empty frames is identified. Accordingly, the operation of determining whether the observation window includes a freeze window according to a preset condition in step 102 includes:

[0048] Determine whether the audio frame in the observation window meets the preset empty frame condition;

[0049] If so, the observation window is determined to be a jam window, and the number of jam windows is accumulated.

[0050] The preset empty frame condition refers to whether the number of empty frames in an observation window is greater than or equal to a preset empty frame threshold. The empty frame threshold can be pre-set based on the actual application. Typically, the empty frame threshold is set based on the size of the observation window, for example, it can be set to half the total number of audio frames in the observation window. Of course, the empty frame threshold can also be set to other values ​​based on actual needs.

[0051] If an observation window contains empty frames and the number of empty frames is greater than or equal to the empty frame threshold, it can be determined that a continuous empty frame insertion scenario has occurred in the observation window. The audio frames in the observation window will produce jamming when played, and the observation window is therefore considered a jamming window.

[0052] For example, if the length of an audio frame is 20ms and the observation window size is 200ms (i.e., 10 audio frames), the empty frame threshold can be set to 5. Therefore, if there are 5 or more empty frames in the observation window (i.e., empty frames with a total length of 100ms or more), it can be considered that the playback of the audio frames in the current observation window will cause perceptible lag.

[0053] See also Figure 3A In observation window W0, audio frames with sequence numbers 1-4 and 6 are decoded normally or can be rendered and played, while audio frames with sequence numbers 5 and 7-10 are empty frames with no audio information and cannot be rendered and played. Therefore, there are five empty frames in observation window W0, and the total number of empty frames is equal to the empty frame threshold, so observation window W0 is a stuck window.

[0054] After identifying the jamming windows, you need to count them. For scenarios with continuous empty frames, calculating the number of jamming windows is relatively simple. Each time the number of empty frames exceeds the empty frame threshold, a jamming window is identified. The jamming window count is then set to 1, and the number of jamming windows within the current evaluation period is incremented by 1.

[0055] Furthermore, in an embodiment of the present invention, a forced frame loss scenario is also identified. Accordingly, the operation of determining whether the observation window includes a freeze window according to a preset condition in step 102 includes:

[0056] Determine whether two adjacent normal audio frames in the observation window meet a preset frame loss condition;

[0057] If yes, the number of jamming windows and the number of observation windows to be accumulated under the observation window are determined according to a calculation method corresponding to the preset frame loss condition.

[0058] All observation windows within a unit time include actual observation windows within the unit time and all observation windows to be accumulated.

[0059] The frame drop condition mentioned above refers to the difference between the sequence numbers of two adjacent normal audio frames within an observation window being greater than or equal to the preset frame drop threshold. If the sequence numbers of two adjacent normal audio frames within an observation window are discontinuous and the difference between the sequence numbers is greater than or equal to the frame drop threshold, it can be determined that a forced frame drop scenario has occurred. The audio frames in this observation window will experience lag when played.

[0060] The frame loss threshold can be pre-set according to the actual application. For example, the frame loss threshold can be set to 20. If the number of forced discarded frames observed in an observation window reaches or exceeds 20, it can be considered that the audio frames in the observation window will be jammed when played. In the experiment, when the number of forced discarded frames observed in an observation window exceeds 20, the audio quality will be significantly reduced. According to the requirements of the specific application scenario for audio quality, the frame loss threshold can also be set to other values, such as 8, 9, 10, etc. The present invention is not limited by the specific number of discarded audio frames.

[0061] For example, see Figure 3B Normally, the frame following frame 1001 should be Seq 1002. However, due to forced frame loss, the next frame may be Seq 1200. The difference between the Seqs of the two normally decoded frames 1001 and 1200 is 199, indicating that 199 frames were discarded, and therefore 199 audio frames could not be decoded and played. At this point, the number of discarded audio frames exceeds the frame loss threshold, causing audio lag. For example, if the audio frame length is 20ms, the loss of 199 frames will cause approximately 4 seconds of lag.

[0062] After identifying a forced frame drop scenario, calculate the number of corresponding jam windows and the number of observation windows to be accumulated. The number of jam windows and the number of observation windows to be accumulated are the same, representing the number of observation windows corresponding to the dropped audio frames.

[0063] In step 102, determining the number of the jam windows and the number of the observation windows to be accumulated according to the calculation method includes: determining the number of the jam windows and the number of the observation windows to be accumulated as n, wherein,

[0064] n=ceil((FrameLength*SeqDiff) / WindowDuration),

[0065] Where FrameLength represents the length of a single audio frame, SeqDiff represents the difference between the sequence numbers of two adjacent normal audio frames, WindowDuration represents the duration of an observation window, and ceil() is a ceiling function whose output is an integer that is greater than or equal to the input parameter and is closest to the input parameter.

[0066] In this embodiment, the number of discarded audio frames is converted into the number of observation windows. These observation windows are all jam windows, and their number is n, so the number of jam windows in the current evaluation period is accumulated by n. At this time, these discarded audio frames that cause jams (if discarded) actually increase the number of observation windows to the same extent as the number of jam windows. Therefore, when calculating the relevant parameters of the jam rate, these discarded audio frames also need to be accumulated in the evaluation period, thereby increasing the total number of audio frames in the evaluation period. In other words, the number of original observation windows in the evaluation period will increase, and the amount of increase is the number of observation windows to be accumulated n.

[0067] In step 103, the jam rate is evaluated, where the jam rate is the ratio of the cumulative number of jam windows determined in the unit time to the cumulative number of all observation windows in the unit time. The jam rate can be expressed by the following formula:

[0068]

[0069] Among them, StuckPercent represents the stuck rate, StuckCount represents the number of stuck windows determined in unit time, and WindowCount represents the sum of the total number of actual observation windows in unit time and the number of all determined observation windows to be accumulated.

[0070] In an embodiment of the present invention, characteristic information such as the Sequence and codec information of an audio frame is analyzed to identify the main scenes causing jamming. Then, based on the observation window mode, the subjective jamming experience that may be caused by the identified jamming scenes is quantified as a jamming window count. Finally, the jamming rate is calculated based on the number of jamming windows and the number of observation windows. In an embodiment of the present invention, the use of jamming windows simplifies the process of jamming quantification and evaluation, and by identifying the main scenes causing jamming, the accuracy of jamming rate evaluation can be improved.

[0071] Furthermore, the transmitter may be muted for some reason while sending voice messages. For audio communications, any scenarios that cause lag in a muted state will not actually cause subjective lag. Therefore, the present invention further eliminates the interference that the muted state of the transmitter may cause in lag identification, thereby further improving the accuracy of lag rate assessment.

[0072] In end-to-end voice communication, if the sender is muted or in a pause between calls, to reduce the bandwidth occupied by voice packets, the sender typically does not send voice packets. Instead, it uses discontinuous transmission (DTX). During silence or pauses, DTX inserts a Silence Insertion Descriptor (SID) frame into the audio data packet. The SID frame contains background noise parameters. The receiver decodes the SID frame to generate background noise, which is known as comfort noise. Providing comfort noise prevents the receiver from mistakenly believing that the call has ended when the sender is muted or silent, thereby improving the user experience.

[0073] When the transmitter is silent (usually accompanied by DTX), the receiver, without receiving voice packets, introduces a large number of empty frames into its jitter buffer. However, empty frames in silent conditions do not cause stuttering and should not be included in the stutter rate calculation. Therefore, it is necessary to identify silent or DTX scenarios and eliminate the interference of empty frames introduced in silent or DTX scenarios on the identification and quantification of stuttering.

[0074] Accordingly, an embodiment of the present invention provides a method 200 for evaluating audio jam rate, such as Figure 4 As shown. Method 200 can be performed by Figure 1 The method 200 may be performed by the jamming assessment module shown in FIG. 2 or by other suitable hardware, software, or a combination of hardware and software. The method 200 includes steps 201 to 205.

[0075] Steps 201 and 202 are the same as steps 101 and 102 in method 100 and are not described again here.

[0076] In step 203, when the current audio frame is a normal audio frame and the previous audio frame is a null frame, a silence or DTX scene is detected.

[0077] As mentioned above, in silence, the receiver's jitter buffer introduces empty frames, which should not be considered when calculating the stutter rate. Therefore, when switching from an empty frame to a normal audio frame—that is, when the decoding module switches from recovering to decoding—it determines whether silence or DTX is present. If so, it is necessary to eliminate the interference of empty frames introduced by silence on the stutter rate evaluation.

[0078] In step 204, if the silence or DTX scenario is detected, the cumulative number of the jamming windows determined within the unit time and the cumulative number of all observation windows within the unit time are reset.

[0079] If the current audio frame is sent when the transmitting end is in a silent or quiet state, then the jamming scenes identified during the silent or quiet period should not be taken into account in the evaluation of the jamming rate. Therefore, in an embodiment of the present invention, when a silent or DTX scene is detected, the temporarily accumulated number of jamming windows and the number of observation windows are cleared to eliminate the interference of the silent or DTX scene on the final jamming rate calculation. After the reset operation, method 200 continues to identify subsequent audio frames, that is, returns to step 202 to continue to determine whether the next audio frame will cause jamming. The above method steps are executed in a loop until all audio frames in the current evaluation period have been analyzed, and then method 200 executes step 205.

[0080] In step 205 , a jam rate is evaluated, where the jam rate is a ratio of a cumulative number of jam windows determined within the unit time to a cumulative number of all observation windows within the unit time.

[0081] Step 205 is similar to step 103 in method 100 and will not be described in detail here. However, the calculation of the jam rate in method 200 has eliminated the interference of empty frames introduced by the silent state on the jam rate, and is therefore more accurate.

[0082] In the above implementation, after identifying the main scene causing the jamming, the present invention further detects the mute or DTX state. If the mute or DTX state is detected, the jamming window determined in the mute state is excluded, thereby further improving the accuracy of the jamming rate assessment.

[0083] In the embodiment of the present invention, the detection of the mute or DTX scene may be performed based on the transmission characteristics of the audio data packet in the mute state.

[0084] In one example, after switching from a normal voice state to a silent state, the payload portion of the first 1 to 2 audio data packets sent by the transmitter contains unencoded silent frames, and the silent frames contain a silent notification identifier. The receiver can determine whether the audio frame contains a silent notification identifier. If the audio frame contains a silent notification identifier, it indicates that the current transmitter has just entered a silent state. Due to entering the silent state, when the receiver receives the audio frame containing the silent notification identifier, a certain number of empty frames have been introduced into its jitter buffer. These empty frames should not be included in the evaluation of the jamming rate.

[0085] Accordingly, the detecting of the mute or DTX scenario in step 203 may include:

[0086] Step 2031, determining whether the current audio frame contains a mute notification flag;

[0087] Step 2034, if yes, determine whether a silence or DTX scene is detected.

[0088] In another example, in the case of normal speech, the timestamps of the audio data frames sent by the transmitter are incremented at a fixed time interval. For example, a data packet contains two audio frames. Figure 5 As shown in the figure, in normal speech, audio data packets are continuous, and the interval between the sending timestamps of each two audio frames is 20ms (i.e., the length of an audio frame). However, in the case of silence or DTX, the transmitter does not continuously send audio data packets. Instead, it sends data packets containing SID frames or empty packets without actual audio data at another fixed time interval. Therefore, the timestamps of the sent data packets will also increase according to the other fixed time interval, rather than the time interval in normal speech. See again Figure 5 In a silent or DTX scenario, the time interval between two consecutive data packets sent by the transmitter (the first audio packet containing audio frames 1 and 2, and the second audio packet containing audio frames 3 and 4) is 2000ms. If the difference in the timestamps of two adjacent normal audio packets (i.e., the difference in the timestamps between the last frame of the previous audio packet and the first frame of the next audio packet) is approximately equal to the time interval for sending audio packets in a silent scenario (e.g., 2000ms), it indicates that the transmitter is currently silent. On the receiving end, the interval between the first and second audio packets is also 2000ms.

[0089] Accordingly, the detecting of the mute or DTX scenario in step 203 may further include:

[0090] Step 2032: determine whether the difference (TimestampDiff) between the timestamp of the previous normal audio frame and the timestamp of the current audio frame is approximately equal to the first silence interval threshold;

[0091] Step 2034, if yes, determine whether a silence or DTX scene is detected.

[0092] The first silence interval threshold value needs to be determined according to the time interval for sending data packets in a silent scenario in actual applications. Figure 5 In the example shown, in the mute or DTX state, the interval between two audio packets is 2000ms. In this case, the first mute interval threshold can be set to 2000ms. The difference between the timestamps of the audio frame with Sequence 3 in the second audio data packet and the audio frame with Sequence 2 in the first audio data packet is 2000–20 = 1980ms, which is approximately equal to the first mute interval threshold. Therefore, it can be determined that the current transmitter is in mute mode.

[0093] In another example, during silence, although time is constantly accumulating, the encoder at the transmitter does not encode normal audio data packets as it would in a normal voice state. In other words, during silence, time is constantly accumulating, but the number of normal audio data packets sent from the transmitter is far less than the number of audio data packets that should be sent in a normal voice state. This causes the increment of the Seq of the audio frames in the audio data packets to be slower than in a normal voice state. Therefore, when switching from a silent state to a normal voice state, the timestamp of the first normal audio data packet (its first audio frame) sent by the transmitter will be much larger than the timestamp of the previous normal audio data packet (its last audio frame), while the difference in the sequence numbers of the two will not be very large. If an audio frame or audio data packet meets the above conditions, it indicates that the current transmitter has just switched from a silent state to a normal voice state.

[0094] Accordingly, the detecting of the mute or DTX scenario in step 203 may further include:

[0095] Step 2033: Calculate a first difference (SeqDiff) between the sequence number of the previous normal audio frame and the sequence number of the current audio frame; calculate a second difference (TimestampDiff) between the timestamp of the previous normal audio frame and the timestamp of the current audio frame; and determine whether the first difference (SeqDiff) and the second difference (TimestampDiff) satisfy the following conditions:

[0096] TimestampDiff–SeqDiff*FrameLength>second silence interval threshold,

[0097] Among them, FrameLength is the length of a single audio frame;

[0098] Step 2034, if yes, determine whether a silence or DTX scene is detected.

[0099] The specific value of the second silence interval threshold can be obtained through simulation experiments or empirical values, or can be set according to actual applications. For example, the second silence interval threshold can be set to 1500ms. Figure 5 In the example above, the length of a single audio frame is 20ms. The timestamp difference (TimestampDiff) between the third and second audio frames is 2000–20 = 1980ms, while the sequence number difference (SeqDiff) is only 1. At this point, 1980ms–1*20ms = 1960ms exceeds 1500ms, indicating that the transmitter has just switched from silence to normal speech. Of course, the second silence interval threshold can also be set to other values, such as 2500ms.

[0100] In the above three examples, it is possible to determine whether the state is mute or DTX. The difference lies in the stage of the mute state. In the first example, the transmitter has just switched from normal speech state to mute state; in the second example, the transmitter is in mute state; in the third example, the transmitter has just switched from mute state to normal speech state. In actual applications, such as Figure 6 The silence detection process shown can determine whether the sending end is in a silence or DXT scenario no matter which of the three phases the sending end is determined to be in.

[0101] In the above embodiment, method 200, based on identifying jamming scenarios such as continuous insertion of empty frames and forced frame loss, further determines whether the transmitter is in a silent or DTX state, thereby eliminating the interference of empty frames introduced in the silent or DTX scenarios on the quantification and evaluation of jamming, thereby further improving the accuracy of the jamming rate evaluation.

[0102] Now refer to Figure 7 A specific example of the jam rate determination logic of the present invention is described below to better understand the concept of the present invention. This example is only one embodiment of the present invention and the present invention is not limited thereto.

[0103] See also Figure 7 First, check the audio playback status. If it is in the state of continuous empty frame playback, perform temporary freeze statistics and continuously update the number of empty frames.

[0104] If the normal decoding audio frame state is continuously played, the judgment of whether it is a forced frame loss processing scenario is entered, and the specific judgment method is as described above. If it is judged that no forced frame loss occurs, the current state is normal playback, and it is not judged as a jam, and only the final jam rate denominator WindowCount, that is, the number of observation windows, is updated. If a forced frame loss scenario is judged, and the difference between the serial numbers of two adjacent normal audio frames in an observation window is greater than or equal to the preset frame loss threshold, it is judged as a jam. Since forced frame loss scenarios will not occur in the silent or DTX state, the judgment and exclusion of the silent or DTX state can be ignored at this time. At this time, the number of lost audio frames can be included in the accumulation of the numerator and denominator of the final jam rate using the calculation method described above.

[0105] When switching from the state of playing empty frames to the state of playing normal decoding frames, that is, when the decoding module changes from the recover operation to the decode operation, first determine whether the sending end is in the end mute or DTX state. If so, it means that the audio transmission and playback are in a silent state, and there is no need to include the numerator and denominator of the temporary stutter rate in the final stutter rate calculation, and reset the number of temporary empty frames and the stutter rate denominator. If the sending end is not in a silent or DTX state, determine whether the number of temporary empty frames exceeds the empty frame number threshold (for example, 5). If so, it is determined to be a stutter and included in the numerator of the final stutter rate, and the final numerator and denominator are updated, and the number of temporary empty frames is reset. If not, it is not determined to be a stutter, but only the denominator of the final stutter rate is updated, and the number of temporary empty frames is reset.

[0106] According to the above-mentioned jam rate judgment logic, it is possible to judge whether jamming occurs in the empty frame playing state and the normal audio frame playing state during the process of receiving and playing audio; and, when switching from the empty frame playing state to the normal audio frame playing state, it is also possible to identify silent scenes and eliminate their interference with the jam rate statistics, so as to timely and accurately judge whether jamming exists and calculate the jam rate in real time.

[0107] The stutter rate calculated using the aforementioned method can be fed back to the transmitter, serving as a basis for adjusting transmission parameters or selecting a transmission path to improve stuttering and enhance audio fluency. For example, when the transmitter is in an environment where a mobile network and a local area network overlap, the transmitter can dynamically select either the mobile network or the local area network based on parameters such as the stutter rate fed back by the receiver and the data transmission quality of the two communication networks to reduce the stutter rate. Based on the data transmission quality of the communication network, the transmitter can adjust the audio encoding method to reduce the stutter rate.

[0108] Although the present application describes the above exemplary method as a series of steps or operations, it should be understood that the present invention is not limited by the order of the steps or operations shown herein. One or more of the above steps can be performed in different orders, and some of the steps can also be performed simultaneously with other steps. In addition, when implementing the method of the present invention, it is not necessary to perform all the steps shown in the application.

[0109] According to another aspect of the present invention, a device for evaluating audio jam rate is provided. The device includes:

[0110] a setting module, configured to set an observation window for the audio to be evaluated, wherein the observation window includes one or more audio frames;

[0111] a judgment module, configured to judge, for each observation window within a unit time, whether the observation window contains a stuck window according to preset conditions, and count the stuck windows contained; and

[0112] An evaluation module is used to evaluate a jam rate, wherein the jam rate is a ratio of a cumulative number of jam windows determined within the unit time to a cumulative number of all observation windows within the unit time.

[0113] In some embodiments, the determination module includes:

[0114] an empty frame determination submodule, configured to determine whether the audio frames in the observation window meet a preset empty frame condition, wherein the preset empty frame condition is that the number of empty frames in the observation window is greater than or equal to a preset empty frame number threshold; and

[0115] The first accumulation submodule is used to determine that the observation window is a jam window when the empty frame judgment submodule determines that the audio frame in the observation window meets the preset empty frame condition, and to accumulate the number of jam windows.

[0116] In some embodiments, the determination module includes:

[0117] A frame loss judgment submodule, configured to judge whether two adjacent normal audio frames in the observation window meet a preset frame loss condition; and

[0118] a second accumulation submodule configured to determine, when the frame loss determination submodule determines that two adjacent normal audio frames in the observation window meet a preset frame loss condition, the number of jamming windows and the number of observation windows to be accumulated within the observation window according to a calculation method corresponding to the preset frame loss condition;

[0119] All observation windows within the unit time include actual observation windows within the unit time and all observation windows to be accumulated.

[0120] In some embodiments, the frame loss condition is that the difference between the sequence numbers of two adjacent normal audio frames in the observation window is greater than or equal to a preset frame loss threshold, and

[0121] The second accumulation submodule is further configured to:

[0122] The number of the freeze windows and the number of the observation windows to be accumulated are both determined as n,

[0123] n=ceil((FrameLength*SeqDiff) / WindowDuration)

[0124] Wherein, FrameLength represents the length of a single audio frame, SeqDiff represents the difference between the sequence numbers of the two adjacent normal audio frames, WindowDuration represents the duration of an observation window, and ceil() is a ceiling function.

[0125] In some embodiments, the apparatus further comprises:

[0126] a silence detection module, configured to detect a silence or DTX scenario when the current audio frame is a normal audio frame and the previous audio frame is a null frame; and

[0127] A reset module is used to reset the cumulative number of freeze windows determined in the unit time and the cumulative number of all observation windows in the unit time when the silence detection module detects a silence or DTX scene.

[0128] In some embodiments, the silence detection module includes a first silence detection submodule configured to:

[0129] Determining whether the current audio frame includes a mute notification flag;

[0130] If so, it is determined that a silence or DTX scene is detected.

[0131] In some embodiments, the silence detection module includes a second silence detection submodule configured to:

[0132] Determine whether a difference between a timestamp of a previous normal audio frame and a timestamp of the current audio frame is equal to a first silence interval threshold;

[0133] If so, it is determined that a silence or DTX scene is detected.

[0134] In some embodiments, the silence detection module includes a third silence detection submodule configured to:

[0135] Calculating a first difference between a sequence number of a previous normal audio frame and a sequence number of the current audio frame;

[0136] Calculating a second difference between the timestamp of the previous normal audio frame and the timestamp of the current audio frame;

[0137] Determine whether the first difference and the second difference satisfy:

[0138] TimestampDiff–SeqDiff*FrameLength>second silence interval threshold,

[0139] Wherein, TimestampDiff is the second difference, SeqDiff is the first difference, and FrameLength is the length of a single audio frame;

[0140] If so, it is determined that a silence or DTX scene is detected.

[0141] According to another aspect of the present invention, a computing device 300 is provided. Figure 8As shown. Computing device 300 may include one or more processors 301 and memory 302. Memory 302 stores computer-executable instructions that, when executed by processor 301, cause computing device 300 to perform any of the aforementioned embodiments of the method for evaluating audio stuttering rate. Computing device 300 may also include a communication interface 305.

[0142] The processor 300 may be any suitable processing device, such as a microprocessor, a microcontroller, an integrated circuit, or other suitable processing device. The memory 302 may include any suitable computing system or medium, including but not limited to non-transitory computer-readable media, random access memory (RAM), read-only memory (ROM), a hard disk, flash memory, or other memory devices. The memory 302 may store computer-executable instructions 303, which may be executed by the processor 301 to cause the computing device 300 to perform any embodiment of the above-mentioned method for evaluating audio stuttering rate. The memory 302 may also store data 304.

[0143] In the embodiment of the present invention, the processor 302 may execute various modules included in the instruction 303 to implement any embodiment of the above method for evaluating the audio stuttering rate. For example, the computing device 300 may implement various modules included in the above apparatus for evaluating the audio stuttering rate to execute Figure 2 and Figure 4 Either of methods 100 and 200 is shown.

[0144] According to another aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein computer-executable instructions are stored on the storage medium, and when the instructions are executed by a computer, the computer executes any embodiment of the method for evaluating audio stuttering rate.

[0145] It should be understood that the term "module" refers to computer logic for providing desired functionality. Thus, a module can be implemented by hardware, dedicated circuits, firmware, and / or software, and combinations thereof. In one embodiment, a module is a program code file stored on a storage device, loaded into a memory, and executed by a processor, or can be a computer program product (e.g., computer-executable instructions) stored in a tangible computer-readable storage medium such as RAM, a hard disk, or optical or magnetic media.

[0146] While some embodiments and features of the present invention have been described above, those skilled in the art will appreciate that various modifications or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. Furthermore, under the guidance of the present invention, these features and embodiments may be modified to suit specific circumstances without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein.

Claims

1. A method for evaluating an audio stuttering rate, comprising: Setting an observation window for the audio to be evaluated, where the observation window includes one or more audio frames; For each observation window within the unit time, determining whether the observation window includes a jam window according to a preset condition, and counting the jam windows included, wherein the preset condition includes a preset determination condition based on the number of empty frames in the audio frame; as well as A jam rate is evaluated, where the jam rate is a ratio of a cumulative number of jam windows determined within the unit time to a cumulative number of all observation windows within the unit time.

2. The method according to claim 1, wherein The determining, according to a preset condition, whether the observation window includes a jamming window includes: Determining whether the audio frames in the observation window meet a preset empty frame condition, wherein the preset empty frame condition is that the number of empty frames in the observation window is greater than or equal to a preset empty frame number threshold; If so, the observation window is determined to be a jam window, and the number of jam windows is accumulated.

3. The method according to claim 1 or 2, wherein: The preset condition also includes a preset judgment condition for the number of frame drops in the audio frame, and the judging whether the observation window includes a jamming window according to the preset condition includes: Determine whether two adjacent normal audio frames in the observation window meet a preset frame loss condition; If yes, determining the number of jamming windows and the number of observation windows to be accumulated under the observation window according to a calculation method corresponding to the preset frame loss condition; All observation windows within the unit time include actual observation windows within the unit time and all observation windows to be accumulated.

4. The method according to claim 3, wherein: The frame loss condition is that the difference between the sequence numbers of two adjacent normal audio frames in the observation window is greater than or equal to a preset frame loss threshold; and Determining the number of the jam windows and the number of the to-be-accumulated observation windows according to the calculation method includes: The number of the freeze windows and the number of the observation windows to be accumulated are both determined as n, n=ceil((FrameLength*SeqDiff) / WindowDuration) Wherein, FrameLength represents the length of a single audio frame, SeqDiff represents the difference between the sequence numbers of the two adjacent normal audio frames, WindowDuration represents the duration of an observation window, and ceil() is a ceiling function.

5. The method according to any preceding claim, further comprising: When the current audio frame is a normal audio frame and the previous audio frame is a null frame, detecting a silent or discontinuous transmission scenario; If the silence or discontinuous transmission scenario is detected, the cumulative number of the freeze windows determined within the unit time and the cumulative number of all observation windows within the unit time are reset.

6. The method according to claim 5, wherein: The detection of silence or discontinuous transmission scenarios includes: Determining whether the current audio frame includes a mute notification flag; If so, it is determined that a silence or discontinuous transmission scenario is detected.

7. The method according to claim 5, wherein: The detection of silence or discontinuous transmission scenarios includes: Determine whether a difference between a timestamp of a previous normal audio frame and a timestamp of the current audio frame is equal to a first silence interval threshold; If so, it is determined that a silence or discontinuous transmission scenario is detected.

8. The method according to claim 5, wherein The detection of silence or discontinuous transmission scenarios includes: Calculating a first difference between a sequence number of a previous normal audio frame and a sequence number of the current audio frame; Calculating a second difference between the timestamp of the previous normal audio frame and the timestamp of the current audio frame; Determine whether the first difference and the second difference satisfy: TimestampDiff–SeqDiff*FrameLength>second silence interval threshold, Wherein, TimestampDiff is the second difference, SeqDiff is the first difference, and FrameLength is the length of a single audio frame; If so, it is determined that a silence or discontinuous transmission scenario is detected.

9. A device for evaluating audio stuttering rate, comprising: a setting module, configured to set an observation window for the audio to be evaluated, wherein the observation window includes one or more audio frames; a determination module, configured to determine, for each observation window within a unit time, whether the observation window contains a jam window according to preset conditions, and to count the jam windows contained therein, wherein the preset conditions include a determination condition preset for the number of empty frames in the audio frame; as well as An evaluation module is used to evaluate a jam rate, wherein the jam rate is a ratio of a cumulative number of jam windows determined within the unit time to a cumulative number of all observation windows within the unit time.

10. A computing device comprising: a memory for storing computer-executable instructions; as well as A processor, configured to execute the computer-executable instructions to perform the method according to any one of claims 1 to 8. 11 . A non-transitory computer-readable storage medium having computer-executable instructions stored thereon, wherein when the instructions are executed by a computer, the computer is caused to perform the method according to claim 1 .

Images