Dialog Detector

Abstract

This application relates to a method for extracting audio features in a dialogue detector in response to an input audio signal, the method including: dividing the input audio signal into a plurality of frames; extracting frame audio features from each frame; determining a set of context windows, each of which includes several frames around a current frame; deriving, for each context window, significant context audio features for the current frame based on the frame audio features of frames in the respective context; and concatenating the context audio features to form a composite feature vector representing the current frame. Context windows of different lengths can improve response speed and robustness.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to PCT Patent Application No. PCT / CN2019 / 083173, filed April 18, 2019, U.S. Provisional Patent Application No. 62 / 840,839, filed April 30, 2019, and EP Patent Application No. 19192553.6, filed August 20, 2019, each of which is incorporated by reference in its entirety.

[0002] Technical Field TECHNICAL FIELD This disclosure relates generally to audio signal processing, and more particularly to dialogue detectors. [Background technology]

[0003] Dialogue detectors are a key component of several audio signal processing algorithms, e.g., dialogue enhancement, noise reduction, and loudness metering. In general, in current dialogue detectors, the input audio signal is first converted to a uniform format in a pre-processing component by means such as sampling rate conversion or downmixing. For example, as pre-processing, the input audio signal may be downmixed to a mono audio signal. Next, the processed audio signal is divided into short temporal frames, and audio features are extracted from a context window containing a fixed number of frames to describe the characteristics of each frame. A classifier built using machine learning methods is then applied to automatically map the audio features to a confidence score representing the probability of the presence of dialogue. Finally, post-processing such as a median or mean filter can be applied to remove or smooth unwanted fluctuations in the resulting confidence score. If the confidence score is high, the signal is classified as dialogue. The dialogue signal may then be sent to an audio improvement device, such as a dialogue enhancer. Summary of the Invention [Means for solving the problem]

[0004] A first aspect of the present invention relates to a method of extracting audio features in a dialogue detector responsive to an input audio signal, the method comprising: splitting the input audio signal into a number of frames, extracting frame audio features from each frame, determining a set of context windows, each context window comprising a number of frames surrounding a current frame, deriving, for each context window, associated context audio features for the current frame based on frame audio features of frames in the respective context, and concatenating the context audio features to form a composite feature vector representing the current frame.

[0005] Thus, the present invention proposes to use several context windows, each containing a different number of frames, to represent a frame in different contexts, where context windows of different lengths play different roles in representing the audio characteristics of a target frame. Context windows of different lengths can improve response speed and improve robustness. For this purpose, the present application introduces a new process, combo-term context determination, to determine multiple context windows of different lengths or ranges, for example three context windows, for example short-term context, medium-term context and long-term context. Then, in the audio feature extraction component, audio features are extracted in those contexts.

[0006] In some implementations, the frame feature extraction component extracts frame audio features (i.e., frame audio features) from each of the multiple frames divided from the input audio signal, and the composite duration context determination component determines the length or range of each context window. Then, based on the frame audio features in each determined context, associated context audio features are derived. Then, each context audio feature is concatenated to form a composite feature vector representing the current frame.

[0007] In some implementations, the context window includes a short-term context, a medium-term context, and a long-term context. The short-term context represents local information around the current frame. The medium-term context further includes multiple retrospective frames. The long-term context further includes multiple long-term history frames.

[0008] In some implementations, the length or range of one or more contexts (i.e., the number of frames in each context window) can be predetermined. For example, if a look-ahead buffer is available, the short-term context can include a current frame and a look-ahead frame. The medium-term context can include a current frame, a look-ahead frame, and a retrospective frame. The long-term context can include a current frame, a look-ahead frame, a retrospective frame, and a long-term history frame. In one implementation, the length or range of the look-ahead frame can be predetermined as a length of 23 frames, the length or range of the retrospective frame can be predetermined as a length of 24 frames, and the length or range of the long-term history frame can be predetermined as a length of 48-96 frames. In another example, if a look-ahead buffer is not available, the short-term context can include a current frame and a first portion of a retrospective frame. The medium-term context can include a current frame, the first portion of a retrospective frame, and a second portion of a retrospective frame. The long-term context can include a current frame, the first portion of a retrospective frame, the second portion of a retrospective frame, and a long-term history frame. Thus, the length or range of the first portion of the retrospective frames can be predetermined as being 23 frames long, the length or range of the second portion of the retrospective frames can be predetermined as being 24 frames long, and the length or range of the long-term history frames can be predetermined as being 48-96 frames long.

[0009] In some implementations, the length or range of one or more contexts can be adaptively determined by analyzing stationarity of frame-level features. For example, the adaptive determination is based on information related to the amplitude of the input audio signal. Specifically, one way of adaptively determining the length or range of the short-term context is based on strong onset or transient detection. In another example, the adaptive determination is based on information related to the spectrum of the input audio signal. Specifically, one way of adaptively determining the length or range of the short-term context is based on identifying maximum spectral inconsistency by using a Bayesian Information Criteria. Furthermore, in an adaptive determination implementation, the short-term context can be extended in both the look-ahead and look-back directions, or extended in only one direction. In some implementations, the length or range of the context can be predefined in combination with the adaptive determination.

[0010] Furthermore, this application proposes a pre-cleaning method to remove uncorrelated noise in the signal in order to improve detection accuracy in low SNR dialogues. To this end, this application utilizes downmixing with time-frequency dependent gain, placing more emphasis on correlated signals.

[0011] In some implementations, the input audio signal is first split into frames, and then the frames in the left and right channels are converted into a spectral representation of the frames. The uncorrelated signals in the left and right channels are removed by applying a frequency-dependent gain to the spectrum in the left and right channels, respectively, to obtain a downmixed signal. In some implementations, the frequency-dependent gain can be estimated from a covariance matrix.

[0012] Furthermore, this application introduces a music content detector, whereby both music confidence score and speech confidence score can be jointly considered to modify the original dialogue confidence score and obtain a final modified dialogue confidence score, which significantly reduces false alarms in music.

[0013] In some implementations, a speech content detector receives the features extracted using the context window, and the speech content detector then determines a speech confidence score. A music content detector then receives the features extracted using the context window, and the music content detector then determines a music confidence score. The speech confidence score and the music confidence score are combined to obtain a final dialog confidence score. In some implementations, the final dialog confidence score can be refined by a context-dependent parameter that can be calculated based on the proportion of frames identified as speech or music in the historical context. In some implementations, the historical context can be 10 seconds or longer in length. [Brief description of the drawings]

[0014] The included figures are for illustration purposes and merely provide examples of possible operations for the disclosed inventive methods, systems and computer readable media. These figures are in no way intended to limit any changes in form and detail that may be made by those skilled in the art without departing from the spirit and scope of the disclosed implementations. [Figure 1] 1 illustrates a block diagram of a dialogue detector 100 incorporating a multi-period context determination component 104 according to some implementations. [Diagram 2] 1 shows a block diagram of an audio feature extraction component 102 in a dialog detector 100 according to some implementations. [Diagram 3] 2 illustrates a flowchart of a method 200 of some implementations for predetermining a length or extent of a composite period context by using the composite period context determination component 104. [Figure 4] 4a shows an example of a predetermined length or range of a composite period context when a look-ahead buffer is available according to some implementations. 4b shows an example of a predetermined length or range of a composite period context when a look-ahead buffer is not available according to some implementations. [Figure 5a] 3 illustrates a flowchart of an example of a method 300 of some implementations for adaptively determining a length or extent of a composite period context by using the composite period context determination component 104. [Figure 5b] 3 shows a schematic diagram of a method 300 shown in a context bounded search range. [Figure 6a] 4 illustrates a flowchart of another example method 400 of some implementations for adaptively determining a length or range of a composite period context by using the composite period context determination component 104. [Figure 6b] 4 shows a schematic diagram of the method 400, shown in a Bayesian information criterion window. [Figure 7] 13 illustrates an example of adaptively determined length or extent of a composite duration context when a look-ahead buffer is available, according to some implementations. [Figure 8] 5 shows a flowchart of an example method 500 for downmixing an input audio signal of a dialogue detector, performed according to some implementations. [Figure 9] 6 shows a block diagram of a dialogue detector 600 that further integrates a music content detector 606 according to some implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] As mentioned above, in typical current dialogue detectors, each frame is represented by a context, i.e. a window containing several frames (e.g. 32 or 48 frames), and classified with audio features extracted from the frames within this context window. However, a problem with such conventional dialogue detectors is that they can introduce a large latency in the detection, meaning that the detector can determine whether a dialogue is present only after identifying several dialogue frames, which can have a negative impact on real-time applications. In addition, they cannot extract more robust rhythmic features that can contribute to discriminating speech from singing or rapping, and thus can have a negative impact on the robustness of the dialogue detection.

[0016] To address these issues, the present application discloses a technique that incorporates a set of context windows of different lengths to represent a frame at several scales. Context windows of different lengths play different roles in representing the audio characteristics of a target frame. Some examples of methods, systems and computer-readable media implementing said techniques for audio feature extraction of a dialogue detector responsive to an input audio signal are disclosed as follows.

[0017] FIG. 1 shows a block diagram of a dialogue detector 100 incorporating a composite duration context window determination component 104 according to some implementations. In FIG. 1, the pre-processing component 102 receives an input audio signal. In the pre-processing component 102, the input audio signal may be downmixed to a mono audio signal and then split into frames. Next, the composite duration context determination component 104 and the audio feature extraction component 106 each receive the frames from the pre-processing component 102. Then, in the audio feature extraction component 106, frame audio features are extracted from each frame. Further, in the composite duration context determination component 104, the length or range of each context window is determined. Then, the audio feature extraction component 106 receives the determination result from the composite duration determination component 104. Then, in the audio feature extraction component 106, the frame audio features in each context window are used to derive each context feature depending on the determined context window. Then, each context feature set is concatenated or combined to form a joint feature vector. Next, a classifier 108 receives the extracted feature vector from the audio feature extraction component 106. In the classifier 108, a confidence score is obtained that represents the probability of the presence of a dialogue. Finally, in the post-processing component 110, the obtained confidence score can be smoothed, for example by a median or mean filter, to remove unwanted fluctuations therein.

[0018] FIG. 2 shows a block diagram of the audio feature extraction component 106 in the dialogue detector 100 according to some implementations. Specifically, multi-period context feature extraction and combination is described. In FIG. 2, in the audio feature extraction component 106, frame audio features are extracted from each received frame by the frame feature extraction component 1060. Then, in the multi-period context determination component 104, the length or range of each context window, in this case the short-term context window, the medium-term context window, and the long-term context window, is determined. Then, short-term context audio features, medium-term context audio features, and long-term context audio features are derived based on the frame audio features in the short-term context window, the medium-term context window, and the long-term context window, respectively. Finally, these three context feature sets are concatenated to form a large-dimensional feature vector. For example, if each context feature is 100-dimensional, the concatenated feature will be 300-dimensional.

[0019] Thus, the present application uses multiple context windows instead of one to represent the current frame. In one embodiment, there are three context windows with different lengths or ranges to represent the current frame, namely, a short-term context window, a medium-term context window, and a long-term context window. In particular, the short-term context represents local information around the target frame so that the detector can respond faster when a dialogue appears. The medium-term context is the counterpart used in existing detectors since it can provide a reasonable time span for audio content analysis. The long-term context window represents more global information and extracts only rhythmic features. This is because the short-term or medium-term context windows are typically not long enough to extract robust rhythmic features. That is, the present application adds a short-term context window to improve response speed and a long-term context to improve robustness. Thus, the lengths of these three context windows should be determined during feature extraction. To this end, the present application introduces a composite duration determination component for determining the lengths of the short-term context window, the medium-term context window, and the long-term context window.

[0020] In one example, the frame audio features may include at least one of subband features or fullband features. Examples of subband features include subband spectral energy distribution, subband spectral contrast, subband partial prominence, Mel-frequency cepstral coefficients (MFCCs), MFCC flux, and bass energy. Examples of fullband features include spectral flux, spectral residual, and short-time energy.

[0021] In one example, the context audio features may be derived from one or more frame audio features, for example, the context audio features may include statistics of the frame audio features, such as the mean, mode, median, variance, or standard deviation.

[0022] Additionally or alternatively, the contextual audio features may include rhythm-related features such as 2D modulation features, rhythmic strength, rhythmic clarity, rhythmic regularity, average tempo and / or window-level correlations (i.e., context-level correlations).

[0023] The above examples of frame audio features and context audio features are not intended to be exhaustive, and various other frame audio features and context audio features may be used instead of or in addition to the listed features.

[0024] FIG. 3 shows a flowchart of a method 200 of some implementations for predetermining the length or range of a composite period context by using the composite period context determination component 104. In this exemplary embodiment, the length or range of the composite period context can be predetermined. In one example, if a look-ahead buffer is available, at 202, a short-term context can be determined to include only the current frame and a few look-ahead frames. Here, the length or range of the look-ahead frames can be predefined as 23 frames, so that the total length or range of the short-term context is 24 frames, so that the latest content to be analyzed is to be upcoming. At 204, a medium-term context can be determined to include the current frame, a few look-ahead frames, and a few look-back frames. Here, the length or range of the look-ahead frames can be predefined as 24 frames, so that the total length or range of the medium-term context is 48 frames. Then, at 206, a long-term context can be determined to include the current frame, a few look-ahead frames, a few look-ahead frames, and more history frames. Here, the length or range of the long-term history frames can be predefined as 48 to 96 frames, so that the total length or range of the long-term context is 96 frames to 144 frames, so as to have a stable rhythm feature analysis. Figure 4a shows this example of the predefined length or range of the composite period context.

[0025] In another example, if a look-ahead buffer is not available, then at 208, a short-term context can be determined to include only the current frame and a portion of the retrospective frames. The length or range of the portion of the retrospective frames can be predefined as 23 frames, and thus the total length or range of the short-term context is 24 frames. At 210, a medium-term context can be determined to include the current frame, a portion of the retrospective frames, and a further retrospective frame, where the length or range of the further retrospective frame can be predefined as 24 frames, and thus the total length or range of the medium-term context is 48 frames. Then, at 212, a long-term context can be determined to include the current frame, the portion of the retrospective frames, the further retrospective frames, and more history frames. Here, the length or range of the long-term history frames can be predefined as 48 to 96 frames, and thus the total length or range of the long-term context is 96 to 144 frames. FIG. 4b shows this example of a predefined length or range of a composite period context.

[0026] In the method 200, the length or range of the look-ahead buffer, the look-back buffer, and the long-term history can all be predefined. Alternatively, other numbers of frames apart from the above may be used, as long as it is ensured that the short-term context contains only frames with similar characteristics to the current frame, and the long-term context contains enough historical frames to extract robust rhythmic features.

[0027] Alternatively, the length or range of one or more context windows can be adaptively determined in the composite period context determination component 104 by analyzing the stationarity of frame-level features and grouping audio frames accordingly. Figure 5a shows a flowchart of an example method 300 of some implementations for adaptively determining the length or range of a composite period context window using the composite period context determination component 104. In particular, short-term context is taken as an example to illustrate the method 300. The method 300 is based on the detection of strong transients.

[0028] First, at 302, the short-term energy of frame k is calculated as follows:

number

[0029] Then, at 304, the frame energy is asymmetrically smoothed with a fast tracking factor when the energy increases and a slow decay when the energy decreases, as expressed in equation (2):

number

number

[0030] Next, at 306, a difference filter is applied to the smoothed energy envelope, and values ​​above a given threshold can be considered as onsets, as expressed in equation (3):

number

[0031] Then, in step 308, E onset (k) may be further normalized using the average value of the short-term energy in the search range. The length or range bounds of the short-term context can then be determined at 310, 312 or 314. At 310, the maximum E onset The positions with (k) are taken as context boundaries. In 312, peaks E exceeding a certain threshold, such as 0.3 (tunable between 0 and 1) are onset (k) can be taken as the context boundary. Instead of the threshold, in 314, E onset The distance between (k) and previously identified strong peaks can be taken into account; that is, only when there is a certain distance, such as 1 second, from the previous strong transient, it is determined as a strong transient and taken as the context boundary. Furthermore, in 314, if no strong transient is found in the search range, the entire look-back and / or look-ahead frames are used. Figure 5b shows a schematic diagram of the method 300 shown in the context boundary search range. In particular, it shows the original waveform of the audio signal, the short-term energy, the normalized differential short-term energy, and the scope of the determined short-term context.

[0032] Instead of using amplitude information to determine the scope of the context, the adaptive determination of the scope of the context can also be based on spectral information. For example, the maximum spectral inconsistency can be found to determine the scope of the context by using the Bayesian Information Criterion (BIC). FIG. 6a shows a method 400 of boundary determination based on BIC. Again, a short-term context is taken as an example to explain the method 400. First, in 402, it is assumed that the timestamp t in the BIC window is a true boundary, and the window is better represented by two separate Gaussian models split at time t. Then, in 404, it is assumed that said timestamp t in the BIC window is not a true boundary, and the window is better represented by only one Gaussian model. Then, in 406, the delta BIC is calculated using the following equation (4): ΔBIC(t)=BIC(H 0 )-BIC(H 1 ) (4) Here, H 0 is a hypothesis in 402, and H 1 is a hypothesis at 404. Figure 6b shows an example ΔBIC(t) curve within the BIC window, which is the log likelihood difference between these two hypotheses. The delta BIC may then be normalized at 408. Then, at 410, if the peak of ΔBIC(t) is greater than a threshold (adjustable between 0 and 1), then that peak can be selected as the most likely location of the context boundary.

[0033] FIG. 7 illustrates an example of adaptively determining the length or range of a composite-term context window when a look-ahead buffer is available according to some implementations. In particular, the length or range of a short-term context window is adaptively determined according to method 300 or method 400, and the lengths or ranges of the medium-term and long-term contexts are predefined according to method 200. As shown in FIG. 7, if a look-ahead buffer is available, the short-term context can extend in both the look-ahead and retrospective directions. Alternatively, the short-term context can extend in only one direction, e.g., if a look-ahead buffer is not available (not shown). Although the method 300 or method 400 according to the present application takes the short-term context as an example to describe the adaptive determination, the length or range of the medium-term can also be adaptively determined in a manner similar to the above method 300 or method 400.

[0034] As mentioned above, current dialogue detectors apply mono downmix to L / R for stereo signals or L / R / C for 5.1 signals to reduce computational complexity. However, mixing all channels together may reduce the SNR of dialogue and impair dialogue detection accuracy. For example, dialogue with loud noise (such as in sports games) or dialogue in intensive action scenes may be missed in detection. To address this issue, since most of the dialogue is in channel C of the 5.1 signal, a center channel dominant downmix is ​​applied to reduce the dialogue blurring, as expressed in equation (5). M = 0.707C + g (L + R) / 2 (5) where C, L, R represent the complex-valued spectrum for every time-spectral tile (i.e., for every frame and every bin / band) in the center, left and right channels, respectively, and g is a parameter between 0 and 1 to reduce the contribution from L and R. However, while the above method works for 5.1 signals, it is not applicable for stereo signals, since dialogue is generally considered as a panned signal in stereo signals and is thus correlated in L and R.

[0035] To address this issue, this application proposes a new downmix method that removes uncorrelated noise in a signal to make the dialogue more prominent after downmixing. Figure 8 shows an example of a method 500 for downmixing an input audio signal of a dialogue detector, performed according to some implementations. First, at 502, the input audio signal is divided into multiple frames. Then, at 504, the frames in the left and right channels are converted into a spectral representation of the frame. Then, at 506, the uncorrelated signals are removed by equation (6) as follows: M=g 1 L+g 2 R (6) where L is the spectral representation of the frame in the left channel, R is the spectral representation of the frame in the right channel, and g 1 and g 2 are two frequency-dependent gains that are not broadband gains, applied to L and R, respectively. For simplicity, the frequency band notation is ignored in the formulas. In one implementation, g 1 and g 2 can be estimated from the covariance matrix calculated for each band over a period of time, as expressed in equation (7) (where only the real part is considered, again ignoring the annotation of the frequency bands):

number

[0036] Next, following the idea of ​​eigenvector analysis and ambience extraction in NGCS, 1 and g 2 can be expressed as follows:

number

[0037] Although the above method 500 is described and developed based on a stereo signal, it is also applicable to a 5.1 signal. In one implementation, the 5.1 signal is first downmixed to a stereo signal (L c and R c ) may be converted to L c =0.707C+gL (10) R c =0.707C+gR (11) Then, L c and R c follows this method 500 to remove uncorrelated noise.

[0038] In addition to or instead of the method 500 for removing uncorrelated signals, several other methods can be applied. In some implementations, a method similar to echo cancellation can be applied to reduce noise in the center channel C by using (L+R) / 2 as a reference noise signal. Alternatively, NMF spectral bases for dialogue or both dialogue and noise can be constructed and applied to extract the clean dialogue components.

[0039] Furthermore, in current detectors, music signals, especially acapella (with little accompanying music) or rap vocals, which share many similar characteristics with dialogue, may be misclassified as dialogue, thus significantly increasing false alarms. Applicant finds that for the same misclassified frames, the music confidence score is high. Thus, Applicant introduces a music classifier in parallel with the dialogue detector, which can be used as a reference to refine or modify the original dialogue confidence score, greatly reducing false alarms in music.

[0040] 9 shows a block diagram of a dialogue detector 600 that further integrates a music content detector 606, according to some implementations. First, the input audio signal is split into frames and converted to a spectral representation by a discrete Fourier transform (DFT) 602. Then, in a feature extraction component 604, features representing each frame are extracted according to the procedure shown in FIG. 2. The music content detector 606 then receives the extracted features and calculates a music confidence score C m (t), while the speech content detector 608 also receives the extracted features and obtains the speech confidence score C s (t) is obtained. In addition, the music confidence score C m (t) and the speech confidence score C s (t) may be further smoothed by a median or mean filter. Furthermore, in the post-processing component 610, the music confidence score C m (t) and the speech confidence score C s (t) are combined to give the final dialog confidence score

number

number

[0041] In general, for the same frame, the music confidence score C m If (t) is also high, the confidence score of the original dialogue can be lowered to some extent. However, this does not reduce the confidence score C since the real dialogue content may generate both high dialogue confidence score and high music confidence score when the dialogue exists with a music background. s It may be possible to reduce (t) too much. To address this issue, historical context can be applied to determine whether the music confidence score Cs(t) can be confidently used to refine the dialogue confidence score. If the historical context is dialogue dominated, it is safe to refine the dialogue confidence score, i.e., to ignore the music confidence score.

[0042] Therefore, in some implementations, the final dialog confidence score is refined by the following equation (12):

number

number

number

number

[0043] The dialogue detector techniques described herein can be implemented by one or more computing devices. For example, a controller of a special purpose computing device may be hard-wired to perform or cause to be performed the disclosed operations, or may include digital electronic circuitry, such as one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs), that are permanently programmed to perform or cause to be performed the operations. In some implementations, custom hard-wired logic, ASICs and / or FPGAs with custom programming are combined to accomplish the techniques.

[0044] In some other implementations, a general-purpose computing device may include a controller incorporating a central processing unit (CPU) programmed to perform one or more of the disclosed operations according to program instructions in firmware, memory, other storage, or combinations thereof.

[0045] The term "computer-readable storage medium" as used herein refers to any medium that stores instructions and / or data that cause a computer or machine type to operate in a particular manner. Any of the models, detectors, and operations described herein may be implemented or made to be implemented as software code executable by the processor of the controller using a suitable computer language. The software code may be stored as a series of instructions in a computer-readable medium for storage. Examples of suitable computer-readable storage media include random access memory (RAM), read-only memory (ROM), magnetic media, optical media, solid-state drives, flash memory, and any other memory chip or cartridge. A computer-readable storage medium may be any combination of such storage devices. Such computer-readable storage media may be present on or within a single computing device or an entire computer system, or among other computer-readable storage media in a system or network.

[0046] Although the subject matter of the present application has been specifically shown and described with reference to implementations thereof, it will be understood by those skilled in the art that changes may be made in the form and details of the disclosed implementations without departing from the spirit or scope of the present disclosure. Some examples of these implementations are illustrated in the accompanying drawings, and specific details are described to provide a thorough understanding thereof. It should be noted that implementations may be practiced without some or all of these specific details. Moreover, well-known features may not be described in detail to promote clarity. Finally, although advantages have been discussed herein with reference to certain implementations, it will be understood that the scope should not be limited by reference to such advantages. Rather, the scope should be determined with reference to the appended claims.

[0047] Various aspects of the present invention can be understood from the following enumerated example embodiments (EEE). [EEE1] A method for extracting audio features in a dialogue detector in response to an input audio signal, the method comprising: extracting, by a frame feature extraction component, frame features from each of the plurality of frames segmented from the input audio signal; determining the length or extent of each context according to a composite duration context determinant; deriving relevant context features based on frame features in each determined context; and concatenating each of the context features to form a composite feature vector that represents the current frame. method. [EEE2] The compound period context is: a short-term context that represents local information around the current frame; a mid-term context that further includes multiple retrospective frames; and a long-term context further comprising a plurality of long-term historical frames; The method described in EEE1. [EEE3] The method according to any one of claims 1 to 2, wherein the length or extent of one or more contexts can be predetermined. [EEE4] The method of any one of claims 2 to 3, wherein if a look-ahead buffer is available, the short-term context includes a current frame and a look-ahead frame, or if the look-ahead buffer is not available, the short-term context includes a current frame and a first portion of the retrospective frame. [EEE5] The method of claim 2 or 3, wherein if a look-ahead buffer is available, the medium-term context includes a current frame, a look-ahead frame and a retrospective frame, or if the look-ahead buffer is not available, the medium-term context includes a current frame, the first portion of the retrospective frame and a second portion of the retrospective frame. [EEE6] The method of any one of claims 2 to 3, wherein if a look-ahead buffer is available, the long-term context includes a current frame, a look-ahead frame, a retrospective frame and a long-term history frame, or if the look-ahead buffer is not available, the long-term context includes a current frame, the first portion of the retrospective frame, a second portion of the retrospective frame and the long-term history frame. [EEE7] The method according to any one of claims 1 to 2, wherein the length or scope of one or more contexts can be adaptively determined. [EEE8] The length or extent of one or more contexts can be adaptively determined by analyzing the stationarity of frame-level features, as described in EEE7. [EEE9] The method according to EEE8, wherein the adaptive determination of the length or extent of the one or more contexts is based on information related to the amplitude of the input audio signal. [EEE10] The adaptive determination of the length or range of the short-term context comprises: Calculating a short-time energy of a frame of a plurality of frames of an input audio signal; Smoothing the calculated short-term energies; determining a difference between a smoothed short-term energy of the frame and a smoothed short-term energy of a previous frame; determining a rise based on whether the difference meets a threshold; Normalizing the rise; The following steps: take the position with the greatest rise as the boundary; or Taking the peak onset as the boundary based on whether the peak meets a threshold; or Taking the peak onset as a boundary based on a threshold and the distance between the onset and the strong transient event immediately preceding the onset; determining a length or range boundary of the short-term context by any one of The method according to claim 8, wherein the first and second steps are [EEE11] The method according to EEE8, wherein the adaptive determination of the length or extent of the one or more contexts is based on information related to the spectrum of the input audio signal. [EEE12] The adaptive determination of the length or range of the short-term context comprises: assuming timestamps within a Bayesian information criterion window as true boundaries for the length or extent of said short-term context; assuming that the timestamps within a Bayesian information criterion window are not true bounds on the length or extent of the short-term context; determining a differential Bayes information criterion between the two hypotheses; A step of regularizing the differential Bayes information criterion; and considering a peak in the Differential Bayes Information Criterion as the location of a true boundary for the length or extent of said short-term context based on whether the peak satisfies a threshold. The method according to claim EEE2 or 11. [EEE13] The method of any one of EEE2, 7 to 12, wherein the short-term context can extend in both look-ahead and look-back directions, or in only one direction. [EEE14] 13. The method of any one of the preceding EEE, wherein the length or range of the contexts can be predefined in combination with adaptive determination. [EEE15] 1. A method for downmixing an input audio signal of a dialogue detector, the method comprising: Segmenting an input audio signal into a plurality of frames; converting the left and right channel frames into a spectral representation; removing uncorrelated signals in the left and right channels by applying a frequency dependent gain to the left and right channel spectra, respectively; obtaining a downmixed signal; and feeding the downmixed signal to a dialogue detector having a method according to any one of the preceding claims EEE1 to 14. method. [EEE16] The method according to EEE15, in which the frequency dependent gain can be estimated from the covariance matrix. [EEE17] 1. A method for classifying an input audio signal of a dialogue detector, the method comprising: receiving, by a speech content detector, features extracted according to the method of any one of the preceding claims EEE1 to 14; determining, by a speech content detector, an utterance confidence score; receiving, by a music content detector, features extracted according to the method of any one of the preceding claims EEE1 to 14; determining a music confidence score by a music content detector; combining the speech confidence score and the music confidence score to obtain a final dialog confidence score. method. [EEE18] The method according to EEE17, wherein the final dialog confidence score can be refined by context-dependent parameters. [EEE19] The method according to EEE18, wherein the context dependent parameters can be calculated based on the percentage of frames identified as speech or music in the historical context. [EEE20] The method according to claim 8, wherein the historical context may be 10 seconds or longer.

Claims

1. 1. A computer-implemented method for obtaining a confidence score representative of a probability of the presence of dialogue in a current frame of an input audio signal, the method comprising: dividing the input audio signal into a plurality of frames; extracting frame audio features from each frame; determining a set of context windows, each context window including a number of frames surrounding a current frame; for each context window, deriving context audio features for a current frame based on frame audio features of frames in that context window; concatenating the context audio features derived for the set of context windows determined for a current frame to form a composite feature vector representing the current frame; and using the composite feature vector to obtain an utterance confidence score representing a probability of the presence of dialogue in the current frame; the number of frames in the one or more context windows is adaptively determined based on the extracted frame audio features; method.

2. The set of context windows is: A short-term context window that includes the current frame and a number of frames preceding and / or following the current frame; a medium-term context window including a frame of the short-term context window and a plurality of retrospective frames; and a long-term context window including a frame of the medium-term context window and a plurality of long-term history frames; The method of claim 1.

3. The method of claim 1 or 2, wherein the number of frames in the one or more context windows is adaptively determined by analyzing stationarity of frame-level audio features.

4. The method of claim 3 , wherein the adaptive determination of the number of frames in one or more context windows is based on information related to the amplitude of the input audio signal.

5. The method of claim 3 , wherein the adaptive determination of the number of frames in one or more context windows is based on information related to the spectrum of the input audio signal.

6. A computer-implemented method for obtaining a confidence score representative of a probability of the presence of dialogue in a current frame of an input audio signal, the method comprising: dividing the input audio signal into a plurality of frames; extracting frame audio features from each frame; determining a set of context windows, each context window including a number of frames surrounding a current frame; for each context window, deriving context audio features for a current frame based on frame audio features of frames in that context window; concatenating the context audio features derived for the set of context windows determined for a current frame to form a composite feature vector representing the current frame; and using the composite feature vector to obtain an utterance confidence score representing a probability of the presence of dialogue in the current frame; The method further comprises: converting the frames in the left and right channels into a spectral representation of the frames; removing uncorrelated signals in the left and right channels by applying frequency dependent gains to the spectral representations in the left and right channels, respectively; obtaining a downmixed signal from a left channel and a right channel; and using the downmixed signal as the input audio signal. method.

7. The method of claim 6 , wherein the frequency-dependent gain is estimated from a covariance matrix.

8. 1. A method for classifying an input audio signal of a dialogue detector, the method comprising: Obtaining an utterance confidence score according to the method of any one of claims 1 to 7; determining a music confidence score based on the composite feature vector; combining the speech confidence score and the music confidence score to obtain a final dialogue confidence score. method.

9. A method for classifying an input audio signal of a dialogue detector, the method comprising: Obtaining an utterance confidence score according to the method of any one of claims 1 to 7; determining a music confidence score based on the composite feature vector; combining the speech confidence score and the music confidence score to obtain a final dialogue confidence score; The method, wherein the final dialogue confidence score is refined by context-dependent parameters.

10. The method of claim 9 , wherein the context-dependent parameter is calculated based on a percentage of frames identified as speech or music in a historical context window at least 10 seconds long.

11. Dialogue detector comprising means for carrying out the method according to any one of claims 1 to 10.

12. A computer readable storage medium storing instructions which, when executed, cause a computer to perform the method of any one of claims 1 to 10.

Images