dialog detector

Abstract

The present application relates to a method of extracting audio features in a dialog detector in response to an input audio signal, the method comprising: dividing the input audio signal into a plurality of frames; extracting frame audio features from each frame; determining a set of context windows, each context window comprising a number of frames surrounding a current frame; for each context window, deriving a relevant context audio feature for the current frame based on the frame audio features of the frames in each respective context; and concatenating each context audio feature to form a combined feature vector to represent the current frame. The context windows with different lengths can improve the response speed and improve robustness.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to PCT Patent Application No. PCT / CN2019 / 083173, filed April 18, 2019, U.S. Provisional Patent Application No. 62 / 840,839, filed April 30, 2019, and EP Patent Application No. 19192553.6, filed August 20, 2019, the entire contents of each of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application generally relates to audio signal processing, and in particular to a dialog detector. BACKGROUND

[0004] Dialog detectors are a key component in a number of audio signal processing algorithms, such as dialog enhancement, noise reduction, and loudness metering. Typically, in current dialog detectors, the input audio signal is first converted into a uniform format in a pre-processing component by means of sampling rate conversion or downmixing, etc. For example, as a pre-processing, the input audio signal can be downmixed into a mono audio signal. Next, the processed audio signal is segmented into short time frames, and audio features are extracted from a context window containing a fixed number of frames to describe the characteristics of each frame. Then, a classifier built using machine learning methods is applied to automatically map the audio features to a confidence score representing the probability of the presence of a dialog. Finally, a post-processing, such as a median or mean filter, can be applied to remove or smooth out undesired fluctuations of the obtained confidence score. If the confidence score is high, the signal will be classified as a dialog. The dialog signal can then be sent to an audio improvement device, such as a dialog enhancer. SUMMARY

[0005] A first aspect of the present invention relates to a method of extracting audio features in a dialog detector in response to an input audio signal, the method comprising: dividing the input audio signal into a plurality of frames; extracting frame audio features from each frame; determining a set of context windows, each context window containing a number of frames surrounding a current frame; for each context window, deriving a relevant context audio feature for the current frame based on the frame audio features of the frames in each respective context; and concatenating each context audio feature to form a combined feature vector to represent the current frame.

[0006] Accordingly, the present disclosure proposes to use several context windows, each of which includes different number of frames, to represent frames in different contexts, wherein the context windows with different lengths will play different roles in representing audio properties of a target frame. The context windows with different lengths can improve the response speed and improve robustness. To this end, the present disclosure introduces a new process, i.e., combined period context determination, to determine multiple (e.g., three) context windows with different lengths or ranges, e.g., short-term context, mid-term context, and long-term context; and then extract the audio features in the contexts at the audio feature extraction component.

[0007] In some embodiments, a frame feature extraction component extracts frame audio features (i.e., audio features of a frame) from each frame in a plurality of frames divided from the input audio signal, and a combined period context determination component determines the length or range of each context window. Then, the related context audio features are derived based on the frame audio features in each determined context. Each context audio feature is then concatenated and forms a combined feature vector to represent the current frame.

[0008] In some embodiments, the context windows include a short-term context, a mid-term context, and a long-term context. The short-term context represents local information around the current frame. The mid-term context further contains a plurality of look-back frames. The long-term context further contains a plurality of long-term history frames.

[0009] In some embodiments, the length or range (i.e., the number of frames in the corresponding context window) of one or more contexts can be predetermined. For example, if a look-ahead buffer is available, the short-term context can contain the current frame and a look-ahead frame. The mid-term context can contain the current frame, the look-ahead frame, and the look-back frame. The long-term context can contain the current frame, the look-ahead frame, the look-back frame, and the long-term history frame. In one embodiment, the length or range of the look-ahead frame can be predetermined to be up to 23 frames, and the length or range of the look-back frame can be predetermined to be up to 24 frames, and the length or range of the long-term history frame can be predetermined to be up to 48 to 96 frames. In another example, if the look-ahead buffer is not available, the short-term context can contain the current frame and a first portion of the look-back frame. The mid-term context can contain the current frame, the first portion of the look-back frame, and a second portion of the look-back frame. The long-term context can contain the current frame, the first portion of the look-back frame, the second portion of the look-back frame, and the long-term history frame. Accordingly, the length or range of the first portion of the look-back frame can be predetermined to be up to 23 frames, and the length or range of the second portion of the look-back frame can be predetermined to be up to 24 frames, and the length or range of the long-term history frame can be predetermined to be up to 48 to 96 frames.

[0010] In some embodiments, the length or range of one or more contexts can be adaptively determined by analyzing the stationarity of frame-level features. For example, the adaptive determination is based on information related to the amplitude of the input audio signal. Specifically, one way of adaptively determining the length or range of the short-term context is based on strong onset point or transient detection. In another example, the adaptive determination is based on information related to the spectrum of the input audio signal. Specifically, one way of adaptively determining the length or range of the short-term context is based on identifying the maximum spectral inconsistency by using Bayesian information criterion. In addition, the short-term context can extend to both forward and backward directions, or only one direction in the adaptive determination implementation. In some embodiments, the length or range of the context can be predefined in conjunction with the adaptive determination.

[0011] In addition, the present application proposes a pre-cleaning method to remove irrelevant noise in the signal in order to improve the detection accuracy in low SNR conversations. For this purpose, the present application utilizes downmixing with time-frequency dependent gain, where more emphasis is put on the relevant signal.

[0012] In some embodiments, the input audio signal is first divided into a plurality of frames, and then the frames in the left and right channels are converted into spectral representations of the frames. Irrelevant signals in the left and right channels are removed by applying the frequency-dependent gain to the spectrum in the left and right channels, respectively, to obtain the signals after downmixing. In some embodiments, the frequency-dependent gain can be estimated from a covariance matrix.

[0013] Furthermore, the present application introduces a music content detector, such that the music confidence score and the speech confidence score can be jointly considered to correct the original conversation confidence score and obtain a final corrected conversation confidence score, to significantly reduce false alarms in music.

[0014] In some embodiments, a speech content detector receives the features extracted using the context window, and then the speech content detector determines the speech confidence score. Next, a music content detector receives the features extracted using the context window, and then the music content detector determines the music confidence score. The speech confidence score and the music confidence score are combined to obtain the final conversation confidence score. In some embodiments, the final conversation confidence score can be refined by context-dependent parameters, which can be calculated based on the proportion of frames identified as speech or music in the history context. In some embodiments, the history context can be up to or longer than 10 seconds. BRIEF DESCRIPTION OF DRAWINGS

[0015] The included drawings are for illustrative purposes and are in no way limiting of the scope of the disclosed implementations. These drawings are included to demonstrate various potential embodiments of the disclosed methods, systems, and computer readable media. No single drawing is meant to be viewed as limiting of the scope of the disclosure to forms or details shown in that drawing.

[0016] Figure 1 A block diagram of the conversation detector 100 incorporating the combination period context determination component 104 is shown, in accordance with some embodiments.

[0017] Figure 2 A block diagram of the audio feature extraction component 102 in the conversation detector 100 is shown, in accordance with some embodiments.

[0018] Figure 3 A flowchart of a method 200 for pre-determining the length or range of the combination period context using the combination period context determination component 104 is shown, in accordance with some embodiments.

[0019] Figure 4a An example of the pre-determined length or range of the combination period context in the case where a look-ahead buffer is available, in accordance with some embodiments, is shown.

[0020] Figure 4b An example of the pre-determined length or range of the combination period context in the case where a look-ahead buffer is not available, in accordance with some embodiments, is shown.

[0021] Figure 5a A flowchart of another example of a method 400 for adaptively determining the length or range of the combination period context using the combination period context determination component 104 is shown, in accordance with some embodiments.

[0022] Figure 5b A schematic diagram of the method 300 illustrated in the context boundary search range is shown.

[0023] Figure 6a A flowchart of another example of a method 400 for adaptively determining the length or range of the combination period context using the combination period context determination component 104 is shown, in accordance with some embodiments.

[0024] Figure 6b A schematic diagram of the method 400 illustrated in the Bayesian Information Criterion window is shown.

[0025] Figure 7 An example of the adaptively determined length or range of the combination period context in the case where a look-ahead buffer is available, in accordance with some embodiments, is shown.

[0026] Figure 8A flowchart showing an example of a method 500 of downmixing a dialog detector according to some embodiments.

[0027] Figure 9 A block diagram showing a dialog detector 600 incorporating a further integrated music content detector 606 according to some embodiments. DETAILED DESCRIPTION

[0028] As mentioned above, in conventional current dialog detectors, each frame is represented by a context, i.e., a window comprising several frames (e.g., 32 or 48 frames), and classified according to audio features extracted from the frames in this context window. However, one problem with such conventional dialog detectors is that it can sometimes introduce a large delay in detection, because the detector can only determine whether a dialog is present after identifying several dialog frames, which can negatively impact real-time applications. In addition, it cannot extract more robust rhythmic features, which can help to distinguish speech from singing or rapping, and thus it can negatively impact robustness in dialog detection.

[0029] To address these problems, the present application discloses techniques incorporating a set of context windows of different lengths to represent frames at several scales, where the context windows with different lengths will play different roles in representing the audio properties of the target frame. Some examples of methods, systems, and computer-readable media implementing the techniques for audio feature extraction for a dialog detector in response to an input audio signal are disclosed below.

[0030] Figure 1 A block diagram depicting a dialog detector 100 incorporating a combined per-epoch context window determination component 104 according to some embodiments. In Figure 1In overview, the pre-processing component 102 receives an input audio signal. At the pre-processing component 102, the input audio signal can be down-mixed into a mono audio signal. It is then divided into frames. Next, the combination period context determination component 104 and the audio feature extraction component 106 each receive the frames from the pre-processing component 102. Then, at the audio feature extraction component 106, frame audio features are extracted from each frame. Also, at the combination period context determination component 104, the length or range of each context window is determined. Then, the audio feature extraction component 106 receives the determination results from the combination period determination component 104. Next, at the audio feature extraction component 106, the frame audio features in each context window are used to derive each context feature depending on the determined context window. Each context feature group is then concatenated or combined and a joint feature vector is formed. Next, the classifier 108 receives the extracted feature vector from the audio feature extraction component 106. At the classifier 108, a confidence score representing the probability of the presence of a conversation is obtained. Finally, at the post-processing component 110, the obtained confidence score can be smoothed, e.g., by a median filter or a mean filter, to remove its undesired fluctuations.

[0031] Figure 2 A block diagram depicting the audio feature extraction component 106 in the conversation detector 100 according to some embodiments is depicted. In particular, it describes the combination period context feature extraction and combination. In overview, the pre-processing component 102 receives an input audio signal. At the pre-processing component 102, the input audio signal can be down-mixed into a mono audio signal. It is then divided into frames. Next, the combination period context determination component 104 and the audio feature extraction component 106 each receive the frames from the pre-processing component 102. Then, at the audio feature extraction component 106, frame audio features are extracted from each frame. Also, at the combination period context determination component 104, the length or range of each context window is determined. Then, the audio feature extraction component 106 receives the determination results from the combination period determination component 104. Next, at the audio feature extraction component 106, the frame audio features in each context window are used to derive each context feature depending on the determined context window. Each context feature group is then concatenated or combined and a joint feature vector is formed. Next, the classifier 108 receives the extracted feature vector from the audio feature extraction component 106. At the classifier 108, a confidence score representing the probability of the presence of a conversation is obtained. Finally, at the post-processing component 110, the obtained confidence score can be smoothed, e.g., by a median filter or a mean filter, to remove its undesired fluctuations. Figure 2 In overview, the pre-processing component 102 receives an input audio signal. At the pre-processing component 102, the input audio signal can be down-mixed into a mono audio signal. It is then divided into frames. Next, the combination period context determination component 104 and the audio feature extraction component 106 each receive the frames from the pre-processing component 102. Then, at the audio feature extraction component 106, frame audio features are extracted from each frame. Also, at the combination period context determination component 104, the length or range of each context window is determined. Then, the audio feature extraction component 106 receives the determination results from the combination period determination component 104. Next, at the audio feature extraction component 106, the frame audio features in each context window are used to derive each context feature depending on the determined context window. Each context feature group is then concatenated or combined and a joint feature vector is formed. Next, the classifier 108 receives the extracted feature vector from the audio feature extraction component 106. At the classifier 108, a confidence score representing the probability of the presence of a conversation is obtained. Finally, at the post-processing component 110, the obtained confidence score can be smoothed, e.g., by a median filter or a mean filter, to remove its undesired fluctuations.

[0032] Accordingly, the present application uses multiple context windows, instead of one context window, to represent the current frame. In one embodiment, there are three context windows, i.e., a short-term context window, a mid-term context window, and a long-term context window, with different lengths or ranges, to represent the current frame. In particular, the short-term context represents local information around the target frame, so that the detector can respond faster when a conversation occurs. The mid-term context is the counterpart used in existing detectors, because it can provide a reasonable time span for audio content analysis. The long-term context window represents more global information, where only rhythm features are extracted, because the short-term or mid-term context window is usually not long enough to extract robust rhythm features. That is, the present application adds the short-term context window to improve the response speed, and adds the long-term context window to improve the robustness. Therefore, the lengths of the three context windows should be determined during feature extraction. For this purpose, the present application introduces a combined period determination component to determine the lengths of the short-term context window, the mid-term context window, and the long-term context window.

[0033] In an example, the frame audio features can include at least one of sub-band features or full-band features. Examples of sub-band features include: sub-band spectral energy distribution, sub-band spectral contrast, sub-band partial prominence, Mel-frequency cepstral coefficient (MFCC), MFCC flux, and bass energy. Examples of full-band features include: spectral flux, spectral residual, and short-time energy.

[0034] In an example, the context audio features can be derived from one or more frame audio features. For example, the context audio features can include statistical information of the frame audio features, such as mean, mode, median, variance, or standard deviation.

[0035] Additionally or alternatively, the context audio features can include rhythm-related features, such as 2D modulation features, rhythm strength, rhythm clarity, rhythm regularity, average rhythm, and / or window-level correlation (i.e., context-level correlation).

[0036] The above examples of frame audio features and context audio features are not exhaustive, and various other frame audio features and context audio features can be used instead of or in addition to the listed features.

[0037] Figure 3A flowchart of a method 200 for determining the length or range of the combination period context by the combination period context determination component 104 is shown. In this exemplary embodiment, the length or range of the combination period context can be predetermined. In one example, if a lookahead buffer is available, at 202, a short-term context can be determined to contain only the current frame and a few lookahead frames, where the length or range of the lookahead frames can be predefined as 23 frames, thus, the total length or range of the short-term context is 24 frames, in order to analyze the most recent content. At 204, a mid-term context can be determined to contain the current frame, a few lookahead frames, and a few lookback frames, where the length or range of the lookback frames can be predefined as 24 frames, thus, the total length or range of the mid-term context is 48 frames. Next, at 206, a long-term context can be determined to contain the current frame, a few lookahead frames, a few lookback frames, and more historical frames, where the length or range of the long-term historical frames can be predefined as from 48 to 96 frames, thus, the total length or range of the long-term context is from 96 frames to 144 frames, in order to have a stable rhythm feature analysis. Figure 4a This example shows the predetermined length or range of the combination period context. In another example, if a lookahead buffer is not available, at 208, a short-term context can be determined to contain only the current frame and a portion of the lookback frames, where the length or range of the portion of the lookback frames can be predefined as 23 frames, thus, the total length or range of the short-term context is 24 frames. At 210, a mid-term context can be determined to contain the current frame, the portion of the lookback frames, and further lookback frames, where the length or range of the further lookback frames can be predefined as 24 frames, thus, the total length or range of the mid-term context is 48 frames. Next, at 212, a long-term context can be determined to contain the current frame, the portion of the lookback frames, the further lookback frames, and more historical frames, where the length or range of the long-term historical frames can be predefined as from 48 to 96 frames, thus, the total length or range of the long-term context is from 96 frames to 144 frames. Figure 4b This example shows the predetermined length or range of the combination period context. In the method 200, the length or range of the lookahead buffer, the lookback buffer, and the long-term history can all be predefined. Alternatively, other numbers of frames can be used in addition to the numbers of frames described above, as long as it ensures that the short-term context contains only frames with similar properties as the current frame, and the long-term context contains enough historical frames to extract robust rhythm features.

[0038] Alternatively, the length or range of one or more context windows can be adaptively determined in the combination period context determination component 104 by analyzing the stationarity of the frame-level features and grouping the audio frames accordingly. Figure 5aA flowchart depicting an example of a method 300 for adaptively determining the length or range of a combination period context window by using a combination period context determination component 104 is described. In particular, the method 300 is described with an example of a short-term context.

[0039] The method 300 is based on strong transient detection. First, at 302, the short-time energy S(k) of frame k is calculated using the following equation (1):

[0040]

[0041] where [x k,0 ,…,x k,N-1 ] are the PCM samples of frame k. The samples can also be windowed / weighted before the energy is calculated, and the energy can be derived from the full-band or sub-band signals.

[0042] Then, at 304, the frame energy S(k) is asymmetrically smoothed with a fast tracking coefficient when the energy is increasing and a slow decay when the energy is decreasing, as expressed in equation (2):

[0043]

[0044] where is the smoothed short-term energy in the kth audio frame. The parameter a is the smoothing factor.

[0045] Next, at 306, a difference filter is applied on the smoothed energy envelope, and values that exceed a given threshold d can be considered as starting points E onset (k), as expressed in equation (3):

[0046]

[0047] Then, at 308, E onset (k) can be further normalized using the average of the short-term energies within the search range. Next, the boundaries for the length or range of the short-term context can be determined at 310, 312, or 314. At 310, the position with the maximum E onset (k) is taken as the context boundary. At 312, the peak E onset (k) above a certain threshold (e.g., 0.3), which can be tuned between 0 and 1, is picked as the context boundary. At 314, the distance between E onset (k) and the previously identified strong peak can be considered instead of a threshold. That is, only when it has a certain distance (e.g., 1 second) from the previous strong transient, it will be determined as a strong transient and picked as the context boundary. Additionally, at 314, if no strong transient is found in the search range, the entire backtrace frame and / or lookahead frame will be used.Figure 5b A schematic diagram depicting the method 300 illustrated in the context boundary search range. In particular, it illustrates the original waveform of the audio signal, the short-time energy, the normalized differential short-time energy, and the determined range of the short-term context.

[0048] Instead of using amplitude information to determine the range of the context, the adaptive determination of the range of the context can also be based on spectral information. For example, by using the Bayesian Information Criterion (BIC), the maximum spectral inconsistency can be found to determine the range of the context. Figure 6a A method 400 depicting the BIC-based boundary determination. It also describes the method 400 with the short-term context as an example. First, at 402, a timestamp t in the BIC window is assumed to be a true boundary, and the window is best represented by two separate Gaussian models split at time t. Then, at 404, a timestamp t in the BIC window is assumed to not be a true boundary, and the window is best represented by only one Gaussian model. Next, at 406, ABIC is calculated using the following equation (4):

[0049] ABIC(t) = BIC(H0) - BIC(H1) (4)

[0050] where H0 is the assumption at 402, and H1 is the assumption at 404. Figure 6b An example ABIC(t) curve in the BIC window is shown, which is the log-likelihood difference between the two assumptions. Then, at 408, ABIC can be normalized. Next, at 410, if the peak of ABIC(t) is greater than a threshold (which can be tuned between 0 and 1), the peak can be selected as the most likely location of the context boundary.

[0051] Figure 7 An example of adaptively determining the length or range of the combined mid-term context window in the case that a lookahead buffer is available is shown according to some implementations. In particular, the length or range of the short-term context window is adaptively determined based on the method 300 or the method 400, and the length or range of the mid-term context and the long-term context are predefined based on the method 200. As Figure 7 If a lookahead buffer is available, the short-term context can extend in both the lookahead direction and the backtracking direction, as shown in the middle. Alternatively, for example, if a lookahead buffer is not available (not shown), the short-term context can only extend in one direction. The method 300 or the method 400 according to the present application is described with the short-term context as an example, however, the length or range of the mid-term can also be adaptively determined in a similar manner as the method 300 or 400 described above.

[0052] As mentioned above, current dialogue detectors apply mono downmixing on L / R of stereo signals or L / R / C of 5.1 signals to reduce computational complexity. However, mixing all channels together can reduce the SNR of the dialogue and impair the dialogue detection accuracy. For example, dialogues with large noise (e.g., in sports games) or dialogues in dense action scenes can be missed in detection. To solve this problem, a center channel dominant downmixing is applied to reduce dialogue smearing as most dialogues are in channel C in 5.1 signals, as represented in equation (5).

[0053] M = 0.707C + g(L + R) / 2 (5)

[0054] where C, L, R represent the complex spectrum of each time spectral block (i.e., each frame and each frequency bin / band) in center, left and right channels respectively, and g is a parameter between 0 and 1 to reduce the contribution from L and R. However, the above method is applicable to 5.1 signals but not to stereo signals, because dialogues are usually considered as panning signals, thus correlated in L and R in stereo signals.

[0055] To solve this problem, the present application proposes a new downmixing method to remove the uncorrelated noise in the signal to make the downmixed dialogue more prominent. Figure 8 An example of a method 500 of downmixing an input audio signal of a dialogue detector performed in accordance with some embodiments is depicted. First, at 502, the input audio signal is divided into a plurality of frames. Then, at 504, the frames in the left and right channels are converted into spectral representations of the frames. Next, at 506, the uncorrelated signals are removed by the following equation (6):

[0056] M = g1L + g2R (6)

[0057] where L is the spectral representation of the frame in the left channel and R is the spectral representation of the frame in the right channel, and g1and g2are two frequency dependent gains applied to L and R respectively, instead of a wideband gain. For simplicity, the notations on the frequency bands in the equation are ignored. In one embodiment, g1and g2may be estimated from a covariance matrix computed for each band in a certain duration (where only the real part is considered, and the notations on the frequency bands are also ignored), as represented in equation (7):

[0058]

[0059] Then, following the idea of eigenvector analysis and ambient sound extraction in NGCS, g1and g2may be represented as follows.

[0060]

[0061]

[0062] where a, c and d are the covariance coefficients |L| 2 , re(LR * ) and |R| 2 , respectively, in order to simplify the representation of equations (8) and (9). After 506, the downmixed signal after M will be obtained at 508.

[0063] Although the above method 500 is described and developed based on stereo signals, it can also be applied to 5.1 signals. In one embodiment, a 5.1 signal can first be converted to a stereo signal (L c and R c ) with a center dominant downmix, as represented in equations (10) and (11):

[0064] L c = 0.707C + gL (10)

[0065] R c = 0.707C + gR (11)

[0066] Then, L c and R c will follow the method 500 to remove the uncorrelated noise.

[0067] In addition to or instead of the method 500 of removing uncorrelated signals, some other methods can also be applied. In some embodiments, a method similar to echo cancellation can be applied to reduce the noise in the center channel C by using (L+R) / 2 as the reference noise signal. Alternatively, NMF spectral bases can be built for the speech or both speech and noise, and they can be applied to extract clean speech components.

[0068] Further, in the current detector, music signals, especially singing voice in A-Capella (without much music background) or rap which has many similar properties as speech, can be misclassified as speech, thus the false alarms can be significantly increased. The applicants found that for the same frames of misclassification, the music confidence score is also high. Therefore, the applicants introduced a music classifier in parallel with the speech detector, so that the music confidence score can be used as a reference to refine or correct the original speech confidence score, thus significantly reducing the false alarms in music.

[0069] Figure 9A block diagram showing a dialogue detector 600 further integrated with a music content detector 606 according to some embodiments. First, the input audio signal is divided into multiple frames and converted into a spectral representation by discrete Fourier transform (DFT) 602. Then, at feature extraction component 604, the Figure 2 The illustrated procedure extracts features to represent each frame. Next, the music content detector 606 receives the extracted features to obtain a music confidence score C m (t); At the same time, the speech content detector 608 also receives the extracted features to obtain the speech confidence score C s (t). In addition, the music confidence score C can be further smoothed by a median filter or a mean filter m (t) and speech confidence score C s (t). In addition, at the post-processing component 610, the music confidence score C m (t) and speech confidence score C s (t) are combined to obtain the final dialogue confidence score Specifically, the raw dialogue confidence score will be refined to obtain a refined final dialogue confidence score at the post-processing component 610. Generally speaking, if the music confidence score C for the same frame m (t) is also high, then the original dialogue confidence score can be reduced to some extent. However, it may excessively reduce the speech confidence score C s (t), because if there is a musical background to the conversation, then the real conversation content may also generate both a high conversation confidence score and a high music confidence score. To solve this problem, historical context can be applied to determine whether the music confidence score C can be used confidently. s (t) to refine the dialogue confidence score. If the historical context is dialogue-dominated, the refined dialogue confidence score will be more conservative, i.e., the music confidence score is intentionally ignored. Therefore, in some embodiments, the final dialogue confidence score The refinement is done by the following equation (12):

[0070]

[0071] in is the refined dialogue confidence score at frame t, C s (t) is the speech confidence score, C m(t) is a music confidence score, and β is a context-dependent parameter that controls the degree of influence of the music confidence score on the final speech confidence score. In one implementation, β is computed based on the proportion of frames identified as speech or music in the historical context. For example, β can be set to the ratio of frames identified as music in the historical context using a simple binary approach. In particular, if the context is music-dominant, β can be set to 1, and if the context is speech-dominant, β can be set to 0, as expressed in equation (13):

[0072]

[0073] where N m is the number of music frames, and N is the total number of frames in the historical context; r th is a threshold value, typically set to 0.5, but the threshold value can also be adjusted between 0 and 1 depending on the degree of aggressiveness of music frame enforcement. Alternatively, β can be expressed as a continuous function, for example, a linear function as illustrated in equation (14), or a sigmoid function as illustrated in equation (15):

[0074]

[0075]

[0076] where a is a scaling factor that controls the shape of the sigmoid function, and can be set to 5 in the present application. In addition, the historical context used in the estimation of the context-dependent parameter β can be much longer than the historical frame length used for long-term feature extraction, for example, the length or range of the historical context can be set to 10 seconds or even longer.

[0077] The techniques of the speech detector described herein can be implemented by one or more computing devices. For example, a controller of a special-purpose computing device can be hardwired to perform the disclosed operations, or cause such operations to be performed, and can include digital electronic circuits, such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs), that are persistently programmed to perform operations or cause operations to be performed. In some implementations, custom hardwired logic, ASICs, and / or FPGAs with custom programming are combined to implement the techniques.

[0078] In some other implementations, a general-purpose computing device can include a controller incorporating a central processing unit (CPU) that is programmed to cause one or more disclosed operations to be performed in accordance with program instructions in firmware, memory, other storage, or a combination thereof.

[0079] The term "computer readable storage media" as used herein refers to any media that stores instructions and / or data that cause a computer or machine to operate in a specific fashion. Any of the models, detectors, and operations described herein can be implemented as, or using, software code executable by a processor of a controller. The software code can be stored as a series of instructions on a computer readable medium for storage. Examples of suitable computer readable storage media include random access memory (RAM), read only memory (ROM), magnetic media, optical media, solid state drives, flash memory, and any other memory chip or cartridge. The computer readable storage media can be any combination of such storage devices. Any such computer readable storage media can reside on or within a single computational device or on or within a whole computer system, and can be a portion of another computer readable storage media.

[0080] While the subject matter of the present application has been shown and described with reference to particular embodiments thereof, it will be understood by those skilled in the art that changes in form and detail can be made therein without departing from the spirit or scope of the application. Examples of some of these embodiments have been illustrated in the accompanying drawings and described above in detail. It will be understood that the embodiments can be practiced without many of the specific details which have been set forth above to provide a thorough understanding of the embodiments. Further, it will be understood that certain features can be utilized independently from each other. Finally, it will be understood that the scope of the application is not to be interpreted in the limited sense of the particular benefits described herein. Rather, the scope of the application is to be determined by the appended claims.

[0081] Various aspects of the present application can be appreciated from the following enumerated example embodiments (EEEs):

[0082] 1. A method of extracting audio features in a conversation detector in response to an input audio signal, the method comprising:

[0083] extracting, by a frame feature extraction component, frame features from each frame of a plurality of frames partitioned from the input audio signal;

[0084] determining, by a combination period context determination component, a length or range of each context;

[0085] deriving, based on the frame features in each determined context, a related context feature; and

[0086] concatenating each context feature and forming a combined feature vector to represent a current frame.

[0087] 2. The method of EEE 1, wherein the combination period context includes:

[0088] a short-term context, which represents local information around the current frame;

[0089] a mid-term context, which further contains a plurality of look-back frames; and

[0090] a long-term context, which further contains a plurality of long-term history frames.

[0091] 3. The method of EEE 1 or 2, wherein the length or range of one or more contexts can be predetermined.

[0092] 4. The method of EEE 2 or 3, wherein the short-term context contains the current frame and a look-ahead frame if a look-ahead buffer is available; or the short-term context contains the current frame and a first portion of the look-back frames if the look-ahead buffer is not available.

[0093] 5. The method of EEE 2 or 3, wherein the mid-term context contains the current frame, the look-ahead frame, and the look-back frames if the look-ahead buffer is available; or the mid-term context contains the current frame, the first portion of the look-back frames, and a second portion of the look-back frames if the look-ahead buffer is not available.

[0094] 6. The method of EEE 2 or 3, wherein the long-term context contains the current frame, the look-ahead frame, the look-back frames, and long-term history frames if the look-ahead buffer is available; or the long-term context contains the current frame, the first portion of the look-back frames, the second portion of the look-back frames, and the long-term history frames if the look-ahead buffer is not available.

[0095] 7. The method of EEE 1 or 2, wherein the length or range of one or more contexts can be adaptively determined.

[0096] 8. The method of EEE 7, wherein the length or range of one or more contexts can be adaptively determined by analyzing stationarity of frame-level features.

[0097] 9. The method of EEE 8, wherein the adaptive determination of the length or range of one or more contexts is based on information related to amplitude of the input audio signal.

[0098] 10. The method of EEE 2 or 9, wherein the adaptive determination of the length or range of a short-term context comprises:

[0099] computing a short-time energy of a frame of the plurality of frames of the input audio signal;

[0100] smoothing the computed short-time energy;

[0101] determining a difference between the smoothed short-term energy of the frame and the smoothed short-term energy of a previous frame;

[0102] determining a starting point based on whether the difference satisfies a threshold;

[0103] normalizing the starting point;

[0104] determining a boundary of the length or range for the short-term context by one of:

[0105] taking a position with a maximum starting point as a boundary; or

[0106] taking a peak starting point as a boundary based on whether a peak satisfies a threshold; or

[0107] taking a peak starting point as a boundary based on both the threshold and a distance between the starting point and a strong transient immediately preceding the starting point.

[0108] 11. The method of EEE 8, wherein the adaptive determination of the length or range of one or more contexts is based on information related to a spectrum of the input audio signal.

[0109] 12. The method of EEE 2 or 11, wherein the adaptive determination of the length or range of a short-term context comprises:

[0110] assuming that a time stamp in a Bayesian Information Criterion window is a true boundary of the length or range for the short-term context;

[0111] assuming that the time stamp in the Bayesian Information Criterion window is not the true boundary of the length or range for the short-term context;

[0112] determining a differential Bayesian Information Criterion between the two assumptions;

[0113] normalizing the differential Bayesian Information Criterion; and

[0114] taking a peak of the differential Bayesian Information Criterion as a position of the true boundary of the length or range of the short-term context based on whether the peak satisfies a threshold.

[0115] 13. The method of any of EEEs 2, 7-12, wherein the short-term context can extend in both a forward-looking and a backward-looking direction, or only in one direction.

[0116] 14. The method of any of the preceding EEEs, wherein the length or range of the context can be predefined in conjunction with the adaptive determination.

[0117] 15. A method of downmixing an input audio signal for a dialog detector, the method comprising:

[0118] dividing the input audio signal into a plurality of frames;

[0119] transforming the frames in a left channel and a right channel into spectral representations of the frames;

[0120] removing uncorrelated signals in the left channel and the right channel by applying frequency dependent gains to the spectra in the left channel and the right channel, respectively;

[0121] obtaining a downmix signal, and

[0122] feeding the downmix signal to a dialog detector using the method of any of the preceding EEEs 1-14.

[0123] 16. The method of EEE 15, wherein the frequency dependent gains are estimable from a covariance matrix.

[0124] 17. A method of classifying an input audio signal for a dialog detector, the method comprising:

[0125] receiving features extracted according to the method of any of the preceding EEEs 1-14 by a speech content detector;

[0126] determining a speech confidence score by the speech content detector; and

[0127] receiving features extracted according to the method of any of the preceding EEEs 1-14 by a music content detector;

[0128] determining a music confidence score by the music content detector; and

[0129] combining the speech confidence score and the music confidence score to obtain a final dialog confidence score.

[0130] 18. The method of EEE 17, wherein the final dialog confidence score is refinable by a context dependent parameter.

[0131] 19. The method of EEE 18, wherein the context dependent parameter is calculable based on a proportion of frames identified as speech or music in a historical context.

[0132] 20. The method of EEE 19, wherein the historical context is up to or longer than 10 seconds.

Claims

1. A method for obtaining a confidence score representing the probability of the presence of dialogue in a current frame of an input audio signal, the method comprising: Dividing the input audio signal into a plurality of frames; Extracting frame audio features from each frame; determining a set of context windows, each context window including a different number of frames surrounding the current frame; For each context window, deriving relevant context audio features of the current frame based on the frame audio features of the frame in each corresponding context; Concatenate each context audio feature to form a combined feature vector to represent the current frame; and A speech confidence score representing the probability of the presence of dialogue in the current frame is obtained using the combined feature vector.

2. The method of claim 1, wherein the set of context windows comprises: a short-term context window containing the current frame and a number of frames before and / or after the current frame; a medium-term context window containing the frame of the short-term context window and a plurality of lookback frames; and A long-term context window contains the frame of the medium-term context window and a plurality of long-term history frames. The method according to claim 1 , wherein the number of frames in each context window is predetermined.

4. The method of claim 2, wherein the short-term context window contains the current frame and a set of look-ahead frames if a look-ahead buffer is available. 5 . The method according to claim 1 , wherein the number of frames in one or more context windows is adaptively determined based on the extracted frame audio features. 6 . The method of claim 5 , wherein the number of frames in the one or more context windows is adaptively determined by analyzing stationarity of frame-level audio features. 7 . The method of claim 6 , wherein the adaptive determination of the number of frames in one or more context windows is based on information related to an amplitude of the input audio signal. 8 . The method of claim 6 , wherein the adaptive determination of the number of frames in one or more context windows is based on information related to a frequency spectrum of the input audio signal.

9. The method according to any one of claims 1, 2 and 4, further comprising: Convert the frames in the left channel and the right channel into spectral representations of the frames; removing uncorrelated signals in the left and right channels by applying frequency-dependent gains to the spectral representations in the left and right channels, respectively; Obtaining a downmix signal from the left and right channels; and The downmix signal is used as the input audio signal.

10. The method of claim 9, wherein the frequency-dependent gain is estimated from a covariance matrix.

11. A method for classifying an input audio signal of a conversation detector, the method comprising: Obtaining a speech confidence score according to the method of any one of claims 1 to 10; determining a music confidence score based on the combined feature vector; and The speech confidence score and the music confidence score are combined to obtain a final dialogue confidence score.

12. The method of claim 11, wherein the final dialogue confidence score is refined by context-dependent parameters.

13. The method of claim 12, wherein the context-dependent parameter is calculated based on the proportion of frames identified as speech or music in a historical context window. The method of claim 13 , wherein the historical context window is at least 10 seconds long.

15. A conversation detector comprising means for performing the method according to any one of claims 1 to 14.

16. A computer-readable storage medium having stored thereon instructions which, when executed, cause a computer to perform the method according to any one of claims 1 to 14.

Images