Dialogue detector

Abstract

The present application relates to a method for extracting audio features from a dialogue detector in response to an input audio signal, the method comprising dividing the input audio signal into a plurality of frames; extracting frame audio features from each frame; determining a set of context windows, wherein each context window comprises a plurality of frames surrounding a current frame; for each context window, deriving relevant context audio features for the current frame based on frame audio features of frames in each individual context; and chaining each context audio feature to form a combined feature vector to represent the current frame. Context windows of different lengths can improve response speed and improve robustness.

Description

Technology Field

[0001] Cross-reference regarding related applications

[0002] This application claims priority to PCT patent application no. PCT / CN2019 / 083173 filed April 18, 2019, U.S. provisional patent application no. 62 / 840,839 filed April 30, 2019, and EP patent application no. 19192553.6 filed August 20, 2019, each of which is incorporated by reference in its entirety.

[0003] Technology field

[0004] The present application disclosure generally relates to audio signal processing, and in particular to dialogue detectors. Background Technology

[0005] Dialogue detectors are a key component in multiple audio signal processing algorithms, such as dialogue enhancement, noise reduction, and loudness meters. Generally, in current dialogue detectors, the input audio signal is first converted into a uniform format in a pre-processing component by means of sampling rate conversion or down-mixing. For example, as a pre-processing step, the input audio signal can be down-mixed into a mono audio signal. Next, the processed audio signal is separated into short time frames, and audio features are extracted from a context window containing a fixed number of frames to describe the characteristics of each frame. Subsequently, a classifier, constructed using machine learning methods, is applied to automatically map the audio features to confidence scores representing the probability of the presence of dialogue. Finally, post-processing, such as median or mean filters, can be applied to remove or smooth out undesirable variations in the acquired confidence scores. A signal will be classified as dialogue if it has a high confidence score. After that, the dialogue signal can be transmitted to an audio enhancement device, such as a dialogue enhancer.

[0006] A first aspect of the present invention relates to a method for extracting audio features in a dialogue detector in response to an input audio signal, the method comprising dividing the input audio signal into a plurality of frames; extracting frame audio features from each frame; determining a set of context windows, wherein each context window comprises a plurality of frames surrounding a current frame; for each context window, deriving a relevant context audio feature for the current frame based on the frame audio features of the frames in each individual context; and chaining each context audio feature to form a combined feature vector to represent the current frame.

[0007] The present invention therefore proposes using multiple context windows, each containing a different number of frames, to represent frames in different contexts, wherein context windows of different lengths will play different roles in representing the audio properties of the target frame. Context windows of different lengths can improve response speed and improve robustness. To this end, the present application introduces a new process, a combo-term context determination, to determine a plurality of, for example, three, context windows of different lengths or ranges, e.g., short-term context, mid-term context, and long-term context; and then audio features are extracted from the contexts in an audio feature extraction component.

[0008] In some embodiments, a frame feature extraction component extracts frame audio features (i.e., frame audio features) from each frame of a plurality of frames partitioned from the input audio signal, and a composite-period context determination component determines the length or range of each context window. Then, relevant context audio features are derived based on the frame audio features in each determined context. Each context audio feature is then chained to form a combined feature vector to represent the current frame.

[0009] In some implementations, context windows include a short-term context, a medium-term context, and a long-term context. The short-term context represents local information around the current frame. The medium-term context additionally includes multiple look-back frames. The long-term context additionally includes multiple long-term history frames.

[0010] In some embodiments, the length or range of one or more contexts (i.e., the number of frames in each context window) may be predetermined. For example, if a look-ahead buffer is available, a short-term context may include a current frame and look-ahead frames. A medium-term context may include a current frame, look-ahead frames, and look-back frames. A long-term context may include a current frame, look-ahead frames, look-back frames, and long-term history frames. In one embodiment, the length or range of look-ahead frames may be predetermined to 23 frames, the length or range of look-back frames may be predetermined to 24 frames, and the length or range of long-term history frames may be predetermined to 48 to 96 frames. In another example, if a look-ahead frame is not available, a short-term context may include a first portion of a current frame and look-back frames. The mid-term context may include a current frame, a first part of look-back frames, and a second part of look-back frames. The long-term context may include a current frame, a first part of look-back frames, a second part of look-back frames, and long-term history frames. Therefore, the length or range of the first part of look-back frames may be predetermined to 23 frames, the length or range of the second part of look-back frames may be predetermined to 24 frames, and the length or range of the long-term history frames may be predetermined to 48 to 96 frames.

[0011] In some embodiments, the length or range of one or more contexts may be adaptively determined by analyzing the stationarity of frame-level features. For example, the adaptive determination is based on information related to the amplitude of the input audio signal. Specifically, one method for adaptively determining the length or range of short contexts is based on strong onset or transient detection. In another example, the adaptive determination is based on information related to the spectrum of the input audio signal. Specifically, one method for adaptively determining the length or range of short contexts is based on identifying the largest spectral mismatch by using Bayesian Information Criteria. Additionally, short contexts may be extended in both look-ahead and look-back directions, or in adaptive determination embodiments, in only one direction. In some embodiments, the length or range of contexts may be predefined in combination with the adaptive determination.

[0012] Furthermore, the present application proposes a pre-cleaning method for removing uncorrelated noise from a signal to improve detection accuracy in low-SNR dialogues. To this end, the present application utilizes downmixing with time-frequency dependent gains by further emphasizing the correlated signal.

[0013] In some embodiments, the input audio signal is first divided into multiple frames, and then the frames in the left and right channels are converted into spectral representations of the frames. Uncorrelated signals in the left and right channels are removed by applying frequency-dependent gains to the spectra in the left and right channels, respectively, so as to obtain a signal after downmixing. In some embodiments, the frequency-dependent gains can be estimated from the covariance matrix.

[0014] Furthermore, the present application introduces a music content detector such that both the music reliability score and the speech reliability score can be considered together to correct the original dialogue reliability score and obtain the final corrected dialogue reliability score, thereby significantly reducing false alarms in music.

[0015] In some embodiments, the speech content detector receives features extracted using context windows, and then the speech content detector determines a speech confidence score. Next, the music content detector receives features extracted using context windows, and then the music content detector determines a music confidence score. The speech confidence score and the music confidence score are combined to obtain a final dialogue confidence score. In some embodiments, the final dialogue confidence score may be refined by a context-dependent parameter that can be calculated based on the ratio of frames identified as speech or music in the history context. In some embodiments, the history context may be 10 seconds or longer. Brief explanation of the drawing

[0016] The included drawings are for exemplary purposes only and serve to provide possible examples and operations of the methods, systems, and computer-readable media of the disclosed invention. These drawings do not in any way limit any variations in form and detail that may be made by a person skilled in the art without departing from the spirit and scope of the disclosed embodiments. FIG. 1 illustrates a block diagram of a dialogue detector (100) incorporating a complex-period context determination component (104) according to some embodiments. FIG. 2 illustrates a block diagram of an audio feature extraction component (102) in a dialogue detector (100) according to some embodiments. FIG. 3 illustrates a flowchart of a method (200) of several embodiments for predetermining the length or range of a complex-period context by using a complex-period context determination component (104). FIG. 4a illustrates an example of a predetermined length or range of a complex-period context where a look-ahead buffer is available, depending on some implementations. FIG. 4b illustrates an example of a predetermined length or range of a composite-period context when a look-ahead buffer is not available, depending on some implementations. FIG. 5a illustrates a flowchart of an example of a method (300) of some embodiments for adaptively determining the length or range of a complex-period context by using a complex-period context determination component (104). FIG. 5b illustrates a schematic diagram of the method (300) exemplified in the context boundary search range. FIG. 6a illustrates a flowchart of another example of a method (400) of some embodiments for adaptively determining the length or range of a complex-period context by using a complex-period context determination component (104). FIG. 6b illustrates a schematic diagram of the method (400) exemplified in the Bayesian information reference window. FIG. 7 illustrates an example of an adaptively determined length or range of a complex-period context where a look-ahead buffer is available, depending on some implementations. FIG. 8 illustrates a flowchart of an example of a method (500) for downmixing an input audio signal of a dialogue detector, performed according to some embodiments. FIG. 9 illustrates a block diagram in which, according to some embodiments, a dialogue detector (600) additionally incorporates a music contact detector (606). Specific details for implementing the invention

[0017] As mentioned above, in conventional current dialogue detectors, each frame is represented by a context, that is, a window containing multiple frames (such as 32 or 48 frames), and audio features detected from the frames in this context window are classified. However, the problem with these conventional dialogue detectors is that they can sometimes introduce significant latency during detection because the detector can determine whether dialogue is present only after identifying multiple dialogue frames, which can negatively impact real-time applications. Additionally, they can extract stronger rhythmic features that contribute to identifying speech from singing voices or rap, which can therefore negatively affect robustness during dialogue detection.

[0018] To address these issues, the present application discloses techniques for integrating sets of different-length context windows to represent frames at various scales, wherein context windows of different lengths will play different roles in representing audio properties of a target frame. Some examples of methods, systems, and computer-readable media implementing the above techniques for audio feature extraction of a dialogue detector responsive to an input audio signal are disclosed as follows.

[0019] FIG. 1 illustrates a block diagram of a dialogue detector (100) incorporating a complex-period context window determination component (104) according to some embodiments. In FIG. 1, a pre-processing component (102) receives an input audio signal. In the pre-processing component (102), the input audio signal may be down-mixed into a mono audio signal. It is then divided into frames. Next, the complex-period context determination component (104) and the audio feature extraction component (106) each receive frames from the pre-processing component (102). Then, in the audio feature extraction component (106), frame audio features are extracted from each frame. Also, in the complex-period context determination component (104), the length or range of each context window is determined. The audio feature extraction component (106) then receives the determination result from the complex-period determination component (104). Next, in the audio feature extraction component (106), frame audio features in each context window are used to derive each context feature based on the determined context window. Each set of context features is then chained or combined to form a joint feature vector. Next, a classifier (108) receives the feature vector extracted from the audio feature extraction component (106). In the classifier (108), a confidence score indicating the probability of the presence of a dialogue is obtained. Finally, in the post-processing component (110), the obtained confidence scores may be smoothed by, for example, a median filter or an average filter to remove undesirable variations.

[0020] FIG. 2 illustrates a block diagram of an audio feature extraction component (106) in a dialogue detector (100) according to some embodiments. Specifically, it describes complex-period context feature extraction and combination. In FIG. 2, in the audio feature extraction component (106), frame audio features are extracted from each received frame by the frame feature extraction component (1060). Then, in the complex-period context determination component (104), the length or range of each context window, in this case, the short-term context window, the medium-term context window, and the long-term context window, is determined. Next, short-term context audio features, medium-term context audio features, and long-term context audio features are derived based on the frame audio features in the short-term context window, the medium-term context window, and the long-term context window, respectively. Finally, these sets of context features are chained to form a multi-dimensional feature vector. For example, assuming that each context feature is 100 dimensions, the subsequent chained features will be 300 dimensions.

[0021] Therefore, rather than a single context window to represent the current frame, the present application uses multiple context windows. In one embodiment, there are three context windows of different lengths or ranges to represent the current frame: a short context window, a medium context window, and a long context window. Specifically, the short context represents local information around the target frame so that the detector can respond more quickly when a dialogue appears. The medium context is the counterpart used in conventional detectors because it can provide an appropriate time span for audio content analysis. The long context window represents more global information where only rhythmic features are extracted, as the short or medium context windows are typically not long enough to extract strong rhythmic features. That is, the present application adds a short context window to improve response speed and a long context to provide robustness. Accordingly, the lengths of these three context windows must be determined during feature extraction. To this end, the present application introduces a composite-period determination component to determine the lengths of a short-term context window, a medium-term context window, and a long-term context window.

[0022] In the example, frame audio features may include at least one of sub-band features or full-band features. Examples of sub-band features include: sub-band spectral energy distribution, sub-band spectral contrast, sub-band partial protrusion, Mel-frequency cepstral coefficients (MFCC), MFCC plus, and base energy. Examples of full-band features include: spectral flux, residual, and short-term energy.

[0023] In the example, context audio features can be derived from one or more frame audio features. For example, context audio features may include statistics of frame audio features, such as mean, mode, median, variance, or standard deviation.

[0024] Additionally or alternatively, context audio features may include rhythm-related features such as 2D modulation features, rhythm intensity, rhythm clarity, rhythm regularity, average tempo and / or window-level correlation (i.e., context-level correlation).

[0025] The aforementioned examples of frame audio features and context audio features are not exhaustive, and various other frame audio features and context audio features may be used instead of or other than those listed.

[0026] FIG. 3 illustrates a flowchart of a method (200) of several embodiments for predetermining the length or range of a complex-period context by using a complex-period context determination component (104). In these representative embodiments, the length or range of complex-period contexts may be predetermined. In one example, if a look-ahead buffer is available, in 202, the short-term context may be determined to include only a few look-ahead frames, the length or range of the current frame and look-ahead frames may be predetermined as 23 frames to analyze the most recently received content, and thus the total length or range of the short-term context is 24 frames. In 204, the medium-term context may be determined to include a current frame, a few look-ahead frames, and a few look-back frames, wherein the length or range of the look-back frames may be predetermined as 24 frames, and thus the total length or range of the medium-term context is 48 frames. Next, in 206, the long-term context may be determined to include a current frame, several look-ahead frames, several look-back frames, and more history frames to have a stable rhythmic feature analysis, wherein the length or range of the long-term history frames may be predefined to be 48 to 96 frames, and thus the total length or range of the long-term context is 96 to 144 frames. Fig. 4a illustrates such an example of the predefined length or range of the complex-period context. In another example, if the look-ahead buffer is not available, in 208, the short-term context may be determined to include only the current frame and a portion of the look-back frames, wherein the length or range of the portion of the look-back frames may be predefined to be 23 frames, and thus the total length or range of the short-term context is 24 frames.In 210, the mid-term context may be determined to include the current frame, a portion of look-back frames, and additional look-back frames, wherein the length or range of the additional look-back frames may be predefined as 24 frames, and thus the total length or range of the mid-term context is 48 frames. Next, in 212, the long-term context may be determined to include the current frame, a portion of look-back frames, additional look-back frames, and more history frames, wherein the length or range of the long-term history frames may be predefined as 48 to 96 frames, and thus the total length or range of the long-term context is 96 to 144 frames. FIG. 4b illustrates such an example of the predefined length or range of the composite-term context. In method 200, the length or range of the look-ahead buffer, the look-back buffer, and the long-term history may all be predefined. Alternatively, excluding the aforementioned number of frames, another number of frames may be used, provided that the short-term context includes only frames with attributes similar to the current frame and the long-term frame includes sufficient historical frames to extract strong rhythmic features.

[0027] Alternatively, the length or range of one or more context windows may be adaptively determined in the complex-period context determination component (104) by analyzing the stationarity of frame-level features and grouping audio frames accordingly. FIG. 5a depicts a flowchart of an example of a method (300) of some embodiments for adaptively determining the length or range of a complex-period context window by using the complex-period context determination component (104). In particular, it takes a short-term context as an example to illustrate the method (300). Method 300 is based on strong temporal detection. First, in 302, the short-term energy (S(k)) of frame (k) is calculated using the following equation (1):

[0028] (1)

[0029] Here [x k,0 , … , x k,N - 1] are PCM samples of frame k. The samples can also be windowed / weighted before calculating the energy, and the energy can be derived from a full-band or sub-band signal.

[0030] Then, in 304, the frame energy (S(k)) is asymmetrically flattened, having a fast tracking factor when the energy increases and slow decay when the energy decreases, as shown in Equation (2):

[0031] (2)

[0032] Here is the short-term energy flattened in the k-th audio frame. The parameter α is the flattening factor.

[0033] Next, at 306, a difference filter is applied to the flattened energy envelope, and values ​​exceeding a given threshold (δ) are onset (E) as shown in Equation (3). onset It can be considered as (k):

[0034] (3)

[0035] After that, at 308, E onset (k) has the average value of short-term energy in the search range and can be further normalized. Next, the boundary for the length or range of the short-term context can be determined at 310, 312, or 314. At 310, maximum E onset The position with (k) will be taken as a context boundary. At 312, peak E exceeding a specific threshold, such as 0.3 (it can be tuned between 0 and 1). onset (k) can be chosen as a context boundary. Instead of a threshold, at 314, E onsetThe distance between (k) and a previously identified strong peak may be considered. That is, only when it has a specific distance from the previous strong transient, such as 1 second, it is determined as a strong transient and will be chosen as a context boundary. Also, in 314, if no strong transient is found in the search range, the entire look-back frames and / or look-ahead frames will be used. FIG. 5b illustrates a schematic diagram of the method (300) exemplified in the context boundary search range. In particular, it exemplifies the original waveform of the audio signals, the short-term energy, the differential short-term energy after normalization, and the range of the determined short-term context.

[0036] Instead of using amplitude information to determine the range of the context, the adaptive determination of the range of the context may also be based on spectral information. For example, the maximum spectral mismatch can be detected to determine the range of the context by using the Bayesian Information Criterion (BIC). FIG. 6a depicts a method (400) for BIC-based boundary determination. It also takes a short-term context as an example to describe the method (400). First, in 402, the time stamp (t) in the BIC window is assumed to be the actual boundary, and it is better to represent the window by two separate Gaussian models separated at time T. Then, in 404, the time stamp (t) in the BIC window is assumed not to be the actual boundary, and it is better to represent the window by only one Gaussian model. Next, in 406, Delta BIC is calculated using the following Equation (4):

[0037] (4)

[0038] Here, H0 is the hypothesis at 402 and H1 is the hypothesis at 404. Figure 6b illustrates an exemplary ΔBIC(t) curve in the BIC window, which is the log-likelihood difference between the two hypotheses. Then, at 408, delta BIC can be normalized. Next, at 410, if the peak of ΔBIC(t) is greater than the threshold (which can be tuned between 0 and 1), the peak can be chosen as the most likely location of the context boundary.

[0039] FIG. 7 illustrates an example of adaptively determining the length or range of a composite-period context window when a look-ahead buffer is available, according to some embodiments. In particular, the length or range of the short-term context window is adaptively determined based on method (300) or method (400), and the length or range of the medium-term and long-term contexts is predefined based on method (200). As illustrated in FIG. 7, the short-term context may be extended in both the look-ahead direction and the look-back direction if a look-ahead buffer is available. Alternatively, the short-term context may be extended in only one direction, for example, if a look-ahead buffer is not available (not illustrated). While the method (300) or method (400) according to the present application takes the short-term context as an example to illustrate the adaptive determination, the length or range of the medium-term may also be adaptively determined in a manner similar to that of the method (300 or 400).

[0040] As mentioned above, current dialogue detectors apply mono downmixing on L / R for stereo signals or on L / R / C for 5.1 signals to reduce computational complexity. However, mixing all channels together can reduce the SNR of the dialogue and impair the accuracy of dialogue detection. For example, dialogue with large noise (such as in sports games) or dialogue in scenes of intense action may be lost during detection. To address this problem, as shown in Equation (5), center channel dominance downmixing is applied to reduce dialogue smearing, as most of the dialogue is on channel C in 5.1 signals.

[0041] M = 0.707C + g(L+R) / 2 (5)

[0042] Here, C, L, and R represent the complex-value spectra for all time-spectral tiles (i.e., for all frames and all bins / bands) in the center, left, and right channels, respectively, and g is a parameter between 0 and 1 to reduce the contribution from L and R. However, the above method operates for 5.1 signals, but the dialogue is generally considered as a panned signal, and therefore is not applicable to stereo since it is correlated in L and R.

[0043] To address these problems, the present application proposes a new downmixing method to remove uncorrelated noise from a signal so as to make the dialogue more prominent after downmixing. FIG. 8 illustrates an example of a method (500) for downmixing an input audio signal of a dialogue detector, performed according to some embodiments. First, in 502, the input audio signal is divided into a plurality of frames. Then, in 504, the frames in the left channel and the right channel are converted into a spectral representation of the frames. Next, in 506, uncorrelated signals are removed by Equation (6) as follows:

[0044] M = g1L + g2R (6)

[0045] Here, L is the spectral representation of the frames in the left channel and R is the spectral representation of the frames in the right channel, and g1 and g2 are two frequency-dependent gains rather than broad-band gains applied to L and R, respectively. For simplicity, comments regarding frequency bands in the equation are ignored. In one embodiment, g1 and g2 can be estimated from a covariance matrix calculated for all bands over a specific duration, as shown in Equation (7) (only the real part is considered and comments regarding frequency bands are also ignored):

[0046] (7)

[0047] Subsequently, following the mapping of eigenvector analysis and atmosphere extraction in NGCS, g1 and g2 can be expressed as follows.

[0048] (8)

[0049] (9)

[0050] Here, a, c, and d are the covariance coefficients |L| respectively, to simplify the expressions of Equations (8) and (9). 2 , re(LR * ) and |R| 2 It is an alternative expression. After 506, the signal will be obtained at 508 after downmixing M.

[0051] Although the above method (500) is described and developed based on a stereo signal, it may also be applicable to a 5.1 signal. In one embodiment, the 5.1 signal is first a stereo signal (L) as shown in Equations (10) and (11), which is a center-dominant downmix. c and R c It can be converted to:

[0052] L C = 0.707C + gL (10)

[0053] R C = 0.707C + gR (11)

[0054] After that, L C and R C The method (500) will be followed to remove uncorrelated signals.

[0055] In addition to or instead of the method (500) for removing uncorrelated signals, several other methods may also be applied. In some embodiments, a method similar to echo cancellation may be applied to reduce noise in the center channel (C) by using (L+R) / 2 as a reference noise signal. Alternatively, an NMF spectrum basis may be constructed for dialogue or both dialogue and noise, and these may be applied to extract clean dialogue components.

[0056] Furthermore, in the current detector, music signals, particularly singing voices in a cappella (without much musical background) or rap that shares many attributes similar to dialogue, may be misclassified as dialogue, and thus false alarms may increase significantly. The applicant finds that the music reliability score is also high for the same misclassified frames. Therefore, the applicant introduces a music classifier simultaneously with the dialogue detector, and thus the music reliability score can be used as a standard to refine or correct the original dialogue reliability score so that false alarms in music are significantly reduced.

[0057] FIG. 9 illustrates a block diagram of a dialogue detector (600) additionally integrating a music content detector (606) according to some embodiments. First, the input audio signal is divided into a plurality of frames and converted into a spectral representation by a Discrete Fourier Transform (DFT) (602). Then, in a feature extraction component (604), features are extracted to represent all frames according to the procedure illustrated in FIG. 2. Next, the music content detector (606) [requires] a music reliability score (C m (t)) receives extracted features to obtain (t); meanwhile, the speech content detector (608) also obtains a speech reliability score (C s Receives extracted features to obtain (t)). Also, music reliability score (C m (t)) and speech reliability score (C s (t)) can be further flattened by a median filter or an average filter. In addition, in the post-processing component (610), the music reliability score (C m (t)) and speech reliability score (C s (t)) is the final dialogue reliability score ( ) is combined to obtain. In particular, the original dialogue reliability score is combined with the final dialogue reliability score ( refined in the post-processing component (610). It will be refined to obtain ). Generally, the original dialogue reliability score is the music reliability score (C m If (t)) is also high for the same frame, it can be reduced to a slight degree. However, if the dialogue is present with a music background, the actual dialogue content can also generate both a high dialogue reliability score and a high music reliability score, so it is a speech reliability score (C s(t)) can be over-reduced. To address this issue, the history content uses the music reliability score (C) to refine the dialogue reliability score. s (t)) can be applied to determine whether it can be used reliably. If the history context is dialogue-dominant, it would be more conservative to intend to refine the dialogue reliability score, that is, to ignore the music reliability score. Therefore, in some implementations, the final dialogue reliability score ( ) is refined by the following equation (12):

[0058] (12)

[0059] Here is the refined dialogue confidence score in frame t, and C s (t) is the speech reliability score, and C m (t) is a music reliability score, and β is a context-dependent parameter that controls how much the music reliability score influences the final dialogue score. In one embodiment, β is calculated based on the ratio of frames identified as speech or music in the history context. For example, β can be set as the ratio of frames identified as music in the history context using a simple binary method. In particular, as shown in Equation (13), β can be set to 1 if the context is music dominant, and β can be set to 0 if the context is dialogue dominant:

[0060] (13)

[0061] Here, N m is the number of music frames and N is the total frames in the history context; r this a threshold, typically set to 0.5, but the threshold can also be tuned between 0 and 1 depending on how actively the music frames are active. Alternatively, β can be expressed as a continuous function, as a linear function as exemplified in Equation (14), or as a sigmoid function as exemplified in Equation (15):

[0062] (14)

[0063] (15)

[0064] A is a scale factor that controls the shape of the sigmoid function and can be set to 5 in this application. Additionally, the historical context used for estimating the context-dependent parameter (β) may be much longer than the historical frames used for long-term feature extraction, for example, the length or range of the historical context may be set to 10 seconds or much longer.

[0065] The technologies of the dialogue detectors described in this application may be implemented by one or more computing devices. For example, a controller of a special-purpose computing device may include digital electronic circuits, such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs), which may be hard-wired to perform the disclosed operations or to cause such operations to be performed, and which are permanently programmed to perform the operations or cause the operations to be performed. In some embodiments, custom hard-wired logic, ASICs, and / or FPGAs with custom programming are combined to achieve the technologies.

[0066] In some other embodiments, a general-purpose computing device may include a controller that incorporates a central processing unit (CPU) programmed to perform one or more of operations initiated by program instructions in firmware, memory, other storage devices, or a combination thereof.

[0067] As used in this application, the term “computer-readable storage medium” refers to any medium that stores instructions and / or data that cause a type of computer or machine to perform in a particular manner. Any of the models, detectors, and operations described in this application may be implemented as software code executable by a controller’s processor using a suitable computer language, or may be made to be implemented by it. Software code may be stored as a series of instructions on a computer-readable medium for storage. Examples of suitable computer-readable storage media include random access memory (RAM), read-only memory (ROM), magnetic media, optical media, solid-state drives, flash memory, and any other memory chips or cartridges. A computer-readable storage medium may be any combination of these storage devices. Any such computer-readable storage medium may exist on or within a single computing device or an entire computer system, and may be among other computer-readable storage media within a system or network.

[0068] Although the subject matter of this application has been illustrated and described with reference to its embodiments, it will be understood by those skilled in the art that variations in the form and details of the disclosed embodiments may be made without departing from the spirit or scope of this disclosure. Some examples of these embodiments are illustrated in the accompanying drawings, and specific details are provided to provide a thorough understanding thereof. It should be noted that embodiments may be practiced without some or all of these specific details. Additionally, well-known features may not be described in detail to facilitate clarity. Finally, it will be understood that although advantages have been discussed with reference to some embodiments in this application, the scope should not be limited by reference to these advantages. Rather, the scope should be determined by reference to the appended claims.

[0069] Various aspects of the present invention can be understood from the following exemplary embodiments (EEEs):

[0070] 1. A method for extracting audio features from a dialogue detector in response to an input audio signal, wherein the method comprises:

[0071] Extracting frame features from each frame of a plurality of frames divided from the input audio signal by means of a frame feature extraction component;

[0072] Determining the length or range of each context by means of a complex-period context determination component;

[0073] Deriving relevant context features based on frame features in each determined context; and

[0074] It includes chaining each context feature to form a combined feature vector to represent the current frame.

[0075] 2. As a method of EEE 1, the above-mentioned complex-period context is:

[0076] A short-term context representing local information around the current frame;

[0077] A mid-term context further comprising multiple look-back frames; and

[0078] It includes a long-term context that additionally includes multiple long-term history frames.

[0079] 3. As a method of EEE 1 or EEE 2, the length or range of one or more contexts may be determined in advance.

[0080] 4. As a method of EEE 2 or EEE 3, the short context comprises the current frame and the look-ahead frames if the look-ahead buffer is available; or the short context comprises a first portion of the current frame and the look-back frames if the look-ahead buffer is not available.

[0081] 5. As a method of EEE 2 or EEE 3, the intermediate context comprises the current frame, the look-ahead frames, and the look-back frames if the look-ahead buffer is available; or the intermediate context comprises the current frame, a first portion of the look-back frames, and a second portion of the look-back frames if the look-ahead buffer is not available.

[0082] 6. As a method of EEE 2 or EEE 3, the long-term context comprises the current frame, the look-ahead frames, the look-back frames, and the long-term history frames if the look-ahead buffer is available; or the long-term context comprises the current frame, a first portion of the look-back frames, a second portion of the look-back frames, and the long-term history frames if the look-ahead buffer is not available.

[0083] 7. As a method of EEE 1 or EEE 2, the length or range of one or more contexts may be adaptively determined.

[0084] 8. As a method of EEE 7, the length or range of one or more contexts can be adaptively determined by analyzing the stationarity of frame-level features.

[0085] 9. As a method of EEE 8, the adaptive determination of the length or range of one or more contexts is based on information related to the amplitude of the input audio signal.

[0086] 10. As a method of EEE 2 or EEE 9, the adaptive determination of the length or range of the short-term context is:

[0087] Calculating the short-time energy of a frame of a plurality of frames of the above-mentioned input audio signal;

[0088] Flattening the short-time energy calculated above;

[0089] Determining the difference between the flattened short-time energy of the above frame and the flattened short-time energy of the previous frame;

[0090] Determining onset based on whether the above difference satisfies a threshold;

[0091] Normalizing the above onset;

[0092] As determining the boundaries for the length or range of the above short-term context:

[0093] Taking a position with the maximum onset as the above boundary; or

[0094] Taking the peak onset as the boundary based on whether the peak satisfies a threshold; or

[0095] It includes setting a boundary for the length or range of the short-term context by taking the peak onset as the boundary based on both the threshold and the distance between the onset and the strong transient immediately preceding the onset.

[0096] 11. As a method of EEE 8, the adaptive determination of the length or range of one or more contexts is based on information related to the spectrum of the input audio signal.

[0097] 12. As a method of EEE 2 or EEE 11, the adaptive determination of the length or range of the short-term context is:

[0098] Estimating a timestamp in a Bayesian information reference window as an actual boundary for the length or range of the above short-term context;

[0099] Estimating the timestamp in a Bayesian information reference window as not being the actual boundary for the length or range of the aforementioned short-term context;

[0100] Determining the Bayesian information criterion between the two hypotheses above;

[0101] Normalizing the above-mentioned Bayesian information standard; and

[0102] It includes considering the peak of the second-order Bayesian information criterion as the location of the actual boundary for the length or range of the short-term context based on whether the peak satisfies a threshold.

[0103] 13. As a method of any one of EEE 2, EEE 7 to EEE 12, the short context may be extended in both look-ahead and look-back directions, or in only one direction.

[0104] 14. As a method of any one of EEE 1 to EEE 13, the length or range of the contexts may be predefined in combination with the adaptive decision.

[0105] 15. A method for downmixing an input audio signal of a dialogue detector, wherein the method comprises:

[0106] Dividing an input audio signal into multiple frames;

[0107] Converting frames in the left and right channels into a spectral representation of the frames;

[0108] Removing uncorrelated signals in the left channel and the right channel by applying frequency-dependent gains to the spectra in the left channel and the right channel, respectively;

[0109] Acquiring a downmixed signal; and

[0110] It includes supplying the downmixed signal to a dialogue detector using any one of EEE 1 to EEE 14 methods.

[0111] 16. As a method of EEE 15, the frequency-dependent gains can be estimated from the covariance matrix.

[0112] 17. A method for classifying an input audio signal of a dialogue detector, wherein the method comprises:

[0113] Receiving features extracted by a speech content detector according to any one of the methods EEE 1 to EEE 14;

[0114] Determining a speech reliability score by the above-mentioned speech content detector; and

[0115] Receiving features extracted by a music content detector according to any one of the methods EEE 1 to EEE 14;

[0116] Determining a music reliability score by the above music content detector; and

[0117] It includes combining the speech reliability score and the music reliability score to obtain a final dialogue reliability score.

[0118] 18. As a method of EEE 17, the final dialogue reliability score can be refined by context-dependent parameters.

[0119] 19. As a method of EEE 18, the context-dependent parameter may be calculated based on the ratio of frames identified as speech or music in a historical context.

[0120] 20. As a method of EEE 19, the history context may be 10 seconds or longer.

Claims

Claim 1 A method for obtaining a confidence score representing the probability of the presence of a dialogue in a current frame of an input audio signal, comprising: dividing the input audio signal into a plurality of frames by one or more computing devices; extracting frame audio features from each frame (I) by one or more computing devices; determining a set of context windows by one or more computing devices, wherein each context window comprises a plurality of frames surrounding the current frame; deriving relevant context audio features for the current frame based on the frame audio features of frames in each individual context by one or more computing devices for each context window; and chaining each context audio feature to form a combined feature vector to represent the current frame by one or more computing devices. A method for obtaining a speech confidence score representing the probability of the presence of a dialogue in the current frame using the combined feature vector by the one or more computing devices, wherein the number of frames in one or more context windows is adaptively determined based on extracted frame audio features. Claim 2 A method for obtaining a confidence score representing the probability of the existence of a dialogue, wherein the set of context windows comprises: a short-term context window comprising the current frame and a plurality of frames preceding and / or following the current frame; a mid-term context window comprising the frames of the short-term context window and a plurality of look-back frames; and a long-term context window comprising the frames of the mid-term context window and a plurality of long-term history frames. Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 A method for obtaining a confidence score representing the probability of the existence of a dialogue, wherein, in claim 1 or 2, the number of frames in one or more context windows is adaptively determined by analyzing the stationarity of frame-level audio features. Claim 7 In claim 6, the adaptive determination of the number of frames in one or more context windows is a method for obtaining a confidence score representing the probability of the existence of a dialogue based on information related to the amplitude of the input audio signal. Claim 8 In claim 6, the adaptive determination of the number of frames in one or more context windows is a method for obtaining a confidence score representing the probability of the existence of a dialogue based on information related to the spectrum of the input audio signal. Claim 9 A method for obtaining a confidence score representing the probability of the existence of a dialogue, wherein, in claim 1 or 2, the one or more computing devices convert frames in a left channel and a right channel into a spectral representation of the frames; the one or more computing devices remove uncorrelated signals in the left channel and the right channel by applying frequency-dependent gains to the spectral representations in the left channel and the right channel, respectively; the one or more computing devices obtain a downmixed signal from the left and right channels; and the one or more computing devices use the downmixed signal as the input audio signal. Claim 10 In claim 9, the frequency-dependent gains are a method for obtaining a confidence score representing the probability of the existence of a dialogue, which is estimated from a covariance matrix. Claim 11 A method for classifying an input audio signal of a dialogue detector, comprising: a step of obtaining a speech reliability score according to the method of claim 1 or 2; a step of determining a music reliability score based on a combined feature vector; and a step of combining the speech reliability score and the music reliability score to obtain a final dialogue reliability score. Claim 12 A method for classifying an input audio signal of a dialogue detector, wherein the final dialogue reliability score is refined by a context-dependent parameter in claim 11. Claim 13 A method for classifying an input audio signal of a dialogue detector, wherein the context-dependent parameter is calculated based on, for example, the ratio of frames classified as speech or music in a history context window of at least 10 seconds in length. Claim 14 A dialogue detector comprising means for performing a method according to claim 1 or 2. Claim 15 A computer-readable storage medium, wherein the computer-readable storage medium stores instructions that cause a computer to perform a method according to claim 1 or 2 when executed.

Images