Method and system for robust processing of speech classifiers

Abstract

The present disclosure relates to a method and system for performing speech classification on an audio signal. The method includes obtaining the audio signal including a sequence of audio frames and determining, for each audio frame, a first speech confidence metric using a first speech classifier. For each given audio frame in at least a subset of the sequence of audio frames, the method includes classifying each respective audio frame in a first context window associated with the given audio frame as a speech frame or a non-speech frame by comparing the first speech confidence metric of the given audio frame to a first predetermined threshold, determining an adaptive threshold based on a number of speech frames in the first context window, and determining a first binary speech classification indicator for the given audio frame based on the first speech confidence metric and the adaptive threshold.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 483,584, filed February 7, 2023, and U.S. Provisional Patent Application No. 63 / 592,686, filed October 24, 2023, each of which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates to methods and systems for performing speech classification of audio signals. [Background technology]

[0003] In many types of audio processing, a dialogue or speech classifier is used to identify the presence or absence of dialogue or speech in an audio signal. For example, in dialogue enhancement (DE), it is often very useful to extract a dialogue classifier that indicates at what points in the audio signal dialogue is active, e.g., to control when to apply different types of dialogue enhancement processing.

[0004] Many different types of speech classifiers exist. Recently, many speech classifiers based on neural networks have been proposed. Neural network-based and other types of speech classifiers typically predict a speech confidence (typically a value between zero and one) that indicates the likelihood that speech is present in each frame of an audio signal. By comparing the speech confidence with a predetermined threshold (typically 0.5), a classification can be obtained in the form of a predicted binary speech label, "true" or "false," indicating whether speech is present or not in each frame.

[0005] During training of the speech classifier, a large set of training frames is fed to the speech classifier, which predicts a speech confidence for each frame. A loss (e.g., binary cross-entropy loss) is then calculated by comparing the speech confidence with ground truth speech labels. The learnable parameters of the speech classifier are updated to minimize the average loss until training converges.

[0006] The trained classifier can then be used in a variety of applications where it is useful to know when speech is present in an audio signal. For example, a simple speech separator can be implemented by applying a gating function to the audio signal based on the predicted speech label. The gating function can be configured to mute or attenuate the audio signal when the speech label indicates that no speech is present, leaving a processed audio signal with isolated speech. Summary of the Invention [Problem to be solved by the invention]

[0007] One drawback of existing speech classifiers is that misclassification of frames is highly dependent on the type of audio content. For example, music content presents a particularly challenging challenge, as speech classifiers tend to classify certain types of music as speech, resulting in many false positives. Therefore, it is an object of the present disclosure to provide an improved speech classification method that is accurate and works effectively for a variety of audio content types. [Means for solving the problem]

[0008] According to a first aspect of the present disclosure, there is provided a method for performing speech classification on an audio signal, the method including: obtaining the audio signal including a sequence of audio frames; and determining, for each audio frame, a first speech confidence metric indicative of a likelihood of speech being present in the audio frame using a first speech classifier. The method further includes, for each given audio frame in at least a subset of the sequence of audio frames, classifying each respective audio frame in a first context window associated with the given audio frame as a speech frame or a non-speech frame by comparing the first speech confidence metric of the given audio frame against a first predetermined threshold; determining an adaptive threshold based on a number of speech frames in the first context window; and determining a first binary speech classification indicator for the given audio frame based on the first speech confidence metric and the adaptive threshold.

[0009] The term "speech" refers to any type of conversational audio communication, including single-voice speech (monologue), two-voice speech, or multiple-voice speech (conversation). However, the term "speech" does not include singing, which is typically classified as musical content rather than speech content. However, because both singing and speech are human vocalizations, it is often difficult for speech classifiers to distinguish between the two. Often, speech classifiers produce many false positives for musical content accompanied by singing, by incorrectly classifying singing as speech.

[0010] An adaptive threshold based on the number of speech frames in a first context window may be used to adjust the speech threshold on a frame-by-frame basis based on the context of each frame, improving classification accuracy for many types of audio content, including improving false positives for musical content with vocals.

[0011] On the other hand, using a fixed threshold of 0.5 for binarizing speech confidence can achieve a similar false-negative error rate in speech label prediction. That is, the false-negative rate (the rate at which speech is incorrectly classified as non-speech) and the false-positive rate (the rate at which non-speech is incorrectly classified as speech) are roughly equal. A similar false-negative error rate provides a balanced, general-purpose speech classifier that can be used for many types of audio content. However, in some applications, it may be more important to not miss speech content than to correctly classify non-speech content (i.e., as non-speech content). This can be achieved by using a lower threshold, such as 0.2 or 0.1, to label more frames as speech content. A lower threshold reduces false negatives but increases false positives. While a low threshold may work well for movie or sports audio content, applying the same low threshold to music content may result in level pumping or signal instability after processing due to the increased false positives.

[0012] The adaptive threshold of the present disclosure adjusts the threshold based on the number of speech frames detected in the first context window, thereby achieving low error rates for a variety of audio content types, including sports audio content, movie audio content, and music.

[0013] In some embodiments, determining a first binary speech classifier comprises binarizing the first speech confidence metric using the adaptive threshold.

[0014] The term "binarization" refers to converting a range of numeric values ​​(e.g., a continuous range, e.g., 0 to 1) into a binary Boolean value having two states. Binarization is performed by comparing the numeric value with a threshold such that if the numeric value exceeds the threshold, the Boolean value assumes a first state (true, T state), and if the numeric value is below the threshold, the Boolean value assumes a second state (false, F state). If the numeric value is equal to the threshold, the Boolean value can assume either the first state or the second state. It is therefore understood that the binarization process can be configured such that if the numeric value is equal to or greater than the threshold, the Boolean value assumes the first state, or if the numeric value is equal to or less than the threshold, the Boolean value assumes the second state. Thus, the first binary speech classification index is obtained by binarization using an adaptive threshold that is updated and changed on a frame-by-frame basis using the adaptive threshold.

[0015] In some embodiments, the method further includes, for each audio frame, determining a second speech confidence metric indicative of the likelihood of speech being present in the frame using a second speech classifier different from the first speech classifier. The method further includes, for each given audio frame in the subset of the sequence of audio frames: classifying each respective audio frame in a second context window associated with the given audio frame as a speech frame or a non-speech frame by comparing the second speech confidence metric of the respective frame against a second predetermined threshold, and (a) determining a second adaptive threshold based on a number of speech frames in the second context window and / or the number of frames in the first context window, and determining an enhanced binary speech classification index for the given audio frame based on the adaptive threshold and the second adaptive threshold, or (b) the adaptive threshold is based on the number of speech frames in the first context window and the number of speech frames in the second context window.

[0016] That is, combining the capabilities of two different speech classifiers can produce a more accurate binary speech classifier. Either the number of speech frames in each context window can be combined to determine a single, more accurate adaptive threshold, or the number of speech frames in each context window can be used to form the respective (first and second) adaptive thresholds, thereby using both thresholds to determine an enhanced binary classifier. Both approaches improve classification accuracy and can be used alone or in combination.

[0017] According to a second aspect of the present disclosure, there is provided a computer program product comprising instructions that, when executed by a computer, direct the computer to perform the method of the first aspect of the present disclosure.

[0018] According to a third aspect of the present disclosure, there is provided a system including one or more processors configured to perform the method of the first aspect of the present disclosure.

[0019] The present disclosure in the second and third aspects has the same or equivalent advantages as the present disclosure in the first aspect. The functions described with respect to the method may have corresponding features in the system or computer program product. [Brief explanation of the drawings]

[0020] DESCRIPTION OF THE DRAWINGS Aspects of the present disclosure will be described more fully with reference to the accompanying drawings, in which example embodiments are shown.

[0021] [Figure 1] 1 illustrates a block diagram of a speech isolation system using a speech classification system according to some aspects.

[0022] [Figure 2] 1 illustrates a block diagram of an utterance classification system implementing adaptive thresholding according to some aspects.

[0023] [Figure 3]1 is a flow diagram outlining a method for determining binary utterance classification metrics using adaptive thresholds, according to some aspects.

[0024] [Figure 4] FIG. 1 is a block diagram illustrating a variable threshold applicator according to some aspects.

[0025] [Figure 5] 10 is a graph illustrating an example of a threshold function for determining an adaptive threshold based on speech rate.

[0026] [Figure 6] FIG. 1 is a block diagram of an utterance classification system 10 with two speech classifiers according to some embodiments.

[0027] [Figure 7] FIG. 10 is a block diagram illustrating a second variable threshold applier that optionally obtains information from the first variable threshold applier, according to some aspects.

[0028] [Figure 8] 10 is a graph illustrating a scaling function for determining a modified speech rate according to some aspects.

[0029] [Figure 9] 10 is a graph illustrating an example function for setting a parameter u based on a relationship between a first speech rate and a second speech rate. DETAILED DESCRIPTION OF THE INVENTION

[0030] Detailed Description The systems and methods disclosed herein can be implemented as software, firmware, hardware, or a combination thereof. In hardware implementations, the division of tasks does not necessarily correspond to the division of physical units. Conversely, one physical component may have multiple functions, and one task may be performed by several physical components working together.

[0031] The computer hardware may be, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smartphone, an AR / VR wearable, an automotive infotainment system, a web appliance, a network router, a switch or bridge, or any device capable of executing instructions (sequential or otherwise) that specify actions that the computer hardware is to perform. Additionally, the present disclosure is intended to relate to any collection of computer hardware that individually or collectively executes instructions to perform one or more of the concepts described herein.

[0032] Certain or all components may be implemented by one or more processors that accept computer-readable (also called machine-readable) code including a set of instructions that, when executed by one or more of the processors, perform at least one of the methods described herein. Any processor capable of executing a set of instructions (sequential or otherwise) that specify actions to be performed is included. For example, a typical processing system (e.g., computer hardware) may include one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem including a hard drive, an SSD, RAM, and / or ROM. A bus subsystem may also be included for communication between components. Software may reside in the memory subsystem and / or the processor when executed by the computer system.

[0033] One or more processors may operate as standalone devices or may be connected (e.g., networked) to other processor(s). Such networks may be built based on various network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof.

[0034] Software may be distributed on computer-readable media, including computer storage media (or non-transitory media) and communication media (or transient media). As is well known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, EEPROM, flash memory, other memory technology, CD-ROMs, Digital Versatile Disks (DVDs) and other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage and other magnetic storage devices, or any other medium that can be used to store the desired information and that is accessible by a computer. Additionally, those skilled in the art are well aware that communication media (transitory) typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transmission mechanism, and includes information delivery media.

[0035] FIG. 1 illustrates an exemplary audio processing system 1 that uses a speech classification system 10 to separate speech content from an audio signal. The audio signal is provided to a speech separator 20, which processes the audio signal to obtain speech content. The speech separator 20 can be a neural network-based speech separator or other type of speech separator. In a basic non-neural network implementation, the speech separator 20 is a bandpass filter configured to pass typical speech frequencies while attenuating or completely blocking other frequencies. Examples of typical speech frequencies are between 150 Hz and 8 kHz, with female speech typically ranging from 350 Hz to 8 kHz and male speech typically ranging from 150 Hz to 6 kHz. Narrower frequency bands can also be used to capture the most important speech frequencies for maintaining intelligibility, such as a frequency band from 300 Hz to 3.4 kHz. The output of the speech separator 20 is provided to a soft-gating unit 30, which is controlled by the speech classification system 10.

[0036] The speech classification system 10 is configured to determine whether an audio signal contains speech content. Typically, the speech classification system 10 determines a binary speech classification index for each frame of the audio signal. This binary speech classification index has a high value (true state, T) and a low value (false state, F) indicating whether speech is present in the frame. That is, the speech classification system 10 determines for each frame whether to classify it as a speech frame or a non-speech frame.

[0037] The soft gating unit 30 is configured to pass the output of the speech separator 20 (i.e., the isolated speech) through the soft gating unit 30 for speech frames and to suppress the output of the speech separator 20 from passing through the soft gating unit 30 for non-speech frames. For example, the speech separator 20 is not perfect and will pass some residual non-speech audio content when no speech is present. Thus, the soft gating unit 30 acts as a further filter to suppress this residual non-speech content by preventing the content of the non-speech frames from passing through the soft gating unit 30.

[0038] In some aspects, the soft gating unit 30 is configured to apply a gain of 0 dB when speech is present in a frame and a lower gain (e.g., −25 dB) to frames where no speech is present. Optionally, the gating unit 30 completely mutes the output audio signal from the speech separator 20 when no speech is present in a frame. To prevent abrupt transitions between speech and non-speech portions, the soft gating unit 30 may implement a finite transition time for the applied gains, such that the gains applied to non-speech frames smoothly transition to the gains applied to speech frames, and vice versa. The soft gating unit 30 is controlled by a binary speech classification index, but may also be configured to apply a continuously varying gate, for example, by smoothing the binary speech classification index.

[0039] The estimated speech signal output from the soft gating unit 30 is a cleaner speech signal with enhanced intelligibility compared to the output of the speech separator 20 .

[0040] However, the functionality of the soft-gating unit 30 (and thus the resulting speech isolation level in the output estimated speech audio signal) is limited by the accuracy of the speech classifier 10, and a need exists for a speech classification system 10 that is accurate for many types of audio content (e.g., accurate for sports and movie content as well as music content). It will be appreciated that the speech classification system 10 has many uses and is currently being implemented in many applications. For example, in addition to the speech isolation audio processing system shown in FIG. 1 , the speech classification system 10 can be used to label audio data for neural network training, to modify audio rendering or audio processing techniques depending on the presence or absence of speech, or to control noise suppression algorithms in applications such as teleconferencing or videoconferencing. Thus, the new and improved speech classification system 10 can be used in many embodiments.

[0041] 2 is a block diagram illustrating details of an example embodiment of the speech classification system 10 for determining the binary speech classifier SB1. With additional reference to the flowchart of FIG. 3, the operation of the speech classification system 10 will now be described in detail.

[0042] In step S1, the speech classification system 10 receives an input audio signal. The speech classification system 10 includes a first speech classifier 1a configured to determine, for each frame of the audio signal, a first speech confidence metric SM1 indicative of the likelihood that speech is present in that audio frame in step S2. The first speech classifier 1a may be any type of speech classifier configured to determine the first speech confidence metric SM1 for each frame. For example, the first speech classifier 1a may be based on a trained neural network.

[0043] The first speech confidence metric SM1 determined in step S2 is typically a value defined in the range [0,1], with 0 indicating a 0% probability that speech is present in the audio frame, 1 indicating a 100% probability that speech is present, and values ​​between 0 and 1 indicating a probability between 0% and 100%. Of course, the first speech confidence metric SM1 may be defined differently, for example, in the range [−1,1] or [0,100]. In the following, we will assume that the first speech confidence metric SM1 is defined in the range [0,1], but those skilled in the art will understand that this is merely an example and that the speech classification system 10 can be similarly implemented even if the first speech confidence metric SM1 is defined differently.

[0044] The first speech confidence metric SM1 for each frame is provided to an adaptive threshold applicator 2a, which first determines a first adaptive threshold to which the first speech confidence metric SM1 should be compared, and then applies the adaptive threshold by binarizing the first speech confidence metric SM1 at (i.e., using) the first adaptive threshold.

[0045] Adaptive Threshold Application 2a determines a first adaptive threshold on a frame-by-frame basis by analyzing the frames of a context window associated with each frame.

[0046] With further reference to FIG. 4, the operation of the adaptive thresholder 2a will now be described in more detail.

[0047] The first speech confidence metric SM1 of each of the frames 41 to 46 is supplied to the threshold applicator 2a. As shown in FIG. 4, the first speech confidence metric SM1 of the earliest frame 41 is 0.1, the first speech confidence metric SM1 of the next frame 42 is 0.4, the first speech confidence metric SM1 of the next frame 43 is 0.7, the first speech confidence metric SM1 of the next frame 44 is 0.4, the first speech confidence metric SM1 of the next frame 45 is 0.2, and the first speech confidence metric SM1 of the next frame 46 is 0.9. The first speech confidence metric SM1 of each frame is supplied to the threshold comparator 21. In step S3, the threshold comparator 21 calculates each first speech confidence metric SM1 as follows: 、 Binarization by comparing with a predetermined first threshold T1 。 In the illustrated example, the predetermined first threshold T1 is equal to 0.5, which is a typical value (although other values ​​are possible). For example, the first predetermined threshold T1 may be between 0.1 and 0.6, such as about 0.5 or about 0.3. If the first speech confidence metric SM1 exceeds the first predetermined threshold T1, 、 The frame is classified as a speech frame (represented by true, i.e., binary value 1). 。 Otherwise, the frame is classified as a non-speech frame (false, represented by F, e.g., binary value 0). In the example shown, the first speech confidence metric SM1 of frame 46 is 0.9, which exceeds a first predetermined threshold, and therefore frame 46 is considered a speech frame (true state, indicated by T).

[0048] The results of the binary speech / non-speech classification are provided to a speech ratio calculator 22, which determines the number of frames classified as speech frames within a first context window 40 associated with each given frame, NoF1. The first context window 40 for each given frame may include one or more of the following: (A) one or more frames preceding the given frame, (B) one or more frames following the given frame, and (C) the given frame itself. The first context window 40 comprises a number of frames that collectively capture at least 5 seconds, at least 10 seconds, at least 20 seconds, or at least 30 seconds of temporal context. That is, each (given) frame can be said to be associated with a corresponding first context window that includes at least one of the following: one or more earlier frames, one or more later frames, and optionally the frame itself. The same type of first context window (in terms of window length and number of earlier / later frames) may be used for each frame, but the set of frames included in the first context window of the first frame will be different from the set of frames included in the first context window of the second frame.

[0049] Illustratively, a first context window 40 associated with a given frame 45 includes the given frame 45, two preceding frames 43 and 44, and one succeeding frame 46. This illustrative context window 40 comprises a total of four frames, including the given frame 45, for a total context window length W1 of four frames. The context window 40 is then moved so that the next frame 46 takes over the role of the given frame, and so on.

[0050] It will be appreciated that the exemplary context window described above is merely an example, and that many different types of context windows can be used. In some embodiments, context windows with lengths of 10, 100, or greater than 1000 frames may be used. For example, context windows of 235 frames (representing approximately 5 seconds of content) or 1407 frames (representing approximately 30 seconds of content) may be used.

[0051] Various types of context window sizes are available, including one or more look-back frames and / or one or more look-ahead frames. While a given frame is often included in the exemplary first context window 40 shown in Figure 4, it is also possible to define a context window that does not include the given frame but consists of several (e.g., three) look-back frames preceding the given frame.

[0052] 4, an exemplary context window consisting of a given frame, two look-back frames, and one look-ahead frame is used. Speech ratio calculator 22 counts the number of speech frames NoF1 within this context window and finds 0, 1, 1, 1, 2, and 2 speech frames when the given frame is frames 41, 42, 43, 44, 45, and 46, respectively (assuming that the frame preceding frame 41 is a non-speech frame F and the frame following frame 46 is a speech frame T). Each frame is thereby associated with a number of speech frames NoF1 value indicating the number of speech frames within first context window 40 for that particular frame.

[0053] Optionally, the speech ratio calculator 22 determines the speech ratio r1 for each frame as the ratio of NoF1 to the total number of frames (i.e., frame length W1) in the context window 40. In the current example, there are four frames in the context window 40, resulting in speech ratios of 0.00, 0.25, 0.25, 0.25, 0.50, and 0.50 for frames 41, 42, 43, 44, 45, and 46, respectively.

[0054] In the illustrated example, where the context window is only four frames long, it will be appreciated that speech ratios r1 and NoF1 can take on any of five possible values, namely, speech ratio r1 = 0, 0.25, 0.5, 0.75, or 1. In many embodiments, the context window length is longer, at least 10 frames, at least 100 frames, or at least 1000 frames, which allows for many more possible values ​​of r1 / NoF1 and allows r1 / NoF1 to vary more smoothly from frame to frame.

[0055] The NoF1 value or speech ratio r1 is supplied to the adaptive threshold calculator 24, which calculates an adaptive threshold T for each frame 41 to 46 in step S4. a1 Calculate the adaptive threshold T a1 is updated frame by frame and can be determined, for example, using a threshold function f(NoF) or f(r). The adaptive threshold T a1 are fed to the binary metrics extractor 26, which extracts the first speech confidence metric SM1 for each frame by the adaptive threshold T a1 to obtain a binary speech classification index SB1 in step S5. Comparing the exemplary binary speech classification index SB1 with the speech / non-speech classification by the threshold comparator 21 confirms that the classification results are different, demonstrating that adaptive threshold classification can achieve a low false positive level for many types of audio content. The first binary speech classification index SB1 can be used as the output of the speech classification system 10.

[0056] In some embodiments, the adaptive threshold calculator 25 calculates the number of speech frames NoF1 as an adaptive threshold T a or by using a threshold function f(NoF1) that maps the speech rate r1 to an adaptive threshold T a1 Determine the adaptive threshold based on NoF1 or speech rate r1 using a threshold function f(r1) that maps to 1) There are many examples of the threshold function f(NoF1) or f(r1), and in some embodiments, the threshold function f(NoF1) or f(r1) is a monotonically decreasing function with respect to the number of speech frames NoF1 (increasing speech rate r1) in the first context window 40. The term "monotonically decreasing" encompasses not only a monotonically non-increasing function, but also a strictly monotonically decreasing function, which is a stronger requirement. That is, the threshold function f(.) is monotonically non-increasing if f(x) ≥ f(y) when y ≥ x, and is strictly monotonically decreasing if f(x) > f(y) when y > x.

[0057] Some examples of threshold functions f(r1) are given below, but it is understood that these are merely illustrative and that other functions can also be used. It is further understood that these functions can be converted to functions of the form f(NoF1) by replacing r1 with NoF1 / W1, where W1 is the total number of frames in the first context window 40.

[0058] As a first exemplary threshold function (Example A), a sigmoid function is used:

number

number

number

number

[0059] The example function f A (r1), f B (r1), f C Any one of (r1) can be implemented in the adaptive threshold calculator 24 to determine an adaptive threshold for each frame in step S4, which reduces false negatives in music and / or speech content.

[0060] Table I compares the error rates (expressed as percentages) obtained for music content and speech / movie content for a traditional speech classifier that compares the first speech confidence metric SM1 to a static threshold T (T=0.1 or T=0.5) versus the improved speech classifier system 10 that uses adaptive thresholds derived from threshold functions A-D for different context window lengths (here expressed as durations measured in seconds rather than frames). Table I illustrates the drawback of using a predetermined static threshold T for the final binarization of the first speech confidence metric SM1. Using a higher static threshold (e.g., T=0.5) results in a relatively low false positive (FP) rate for music (1.62%) but a high false negative (FN) rate for speech / movie content (3.31%). This means that speech content is often misclassified as non-speech for speech / movie content. On the other hand, using a lower static threshold (e.g., 0.1) improves the FN rate for speech / movie content (0.83%), resulting in significantly less missed speech content. However, lowering the static threshold results in an increased FP rate for music (4.89%), meaning more music content is misclassified as speech.

[0061] The adaptive threshold-based speech classification system 10 achieves speech classification results with a lower FP rate for music content compared to a low static threshold T and a lower FN rate for speech / movie content compared to a high static threshold. For example, using function C and a 30-second context window, the FP rate for music is 1.71%, while the FN and FP rates for speech / movie content are 1.30% and 6.44%, respectively. Thus, adaptive thresholding provides speech classification that achieves good performance in terms of both a low FP rate for music and a low FN rate for speech / movie content, i.e., good performance for both music and speech. In contrast, static thresholding performs exceptionally well for only one type of audio content but poorly for the other type. While the FP rate for speech / movie content is still relatively high, it is an improvement over processing using a low static threshold (T=0.1); furthermore, false positives for speech / movie content are generally much less noticeable and distracting to listeners than false negatives for speech / movie content or false positives for music content. [Table 1]

[0062] 6, a speech classification system 10 according to some embodiments may incorporate a second speech classifier 1b that supplements the first speech classifier 1a. The second speech classifier 1b is different from the first speech classifier 1a. As will be described later, by combining the two different speech classifiers 1a and 1b, it is possible to utilize the features of each classifier 1a and 1b and achieve a more accurate binary speech classifier SB1 or an enhanced binary speech classifier SBE.

[0063] For example, the two speech classifiers 1a, 1b may be neural network-based speech classifiers 1a, 1b that have different network architectures, that have been trained in different ways, and / or that have been trained with different training data.

[0064] In some embodiments, the second speech classifier 1b comprises a speech separator configured to generate an output speech signal having isolated speech content separated from the input speech signal. The speech separator may be based on, for example, a trained neural network. The speech signal is fed to the speech separator, which predicts an output speech signal including isolated speech with background speech content suppressed or completely removed. To determine the second speech confidence metric SM2 using the output speech signal, a ratio between the spectral energy of the output speech signal and the spectral energy of the input speech signal is determined and used as the second speech confidence metric SM2. This ratio is determined for the entire signal. That is, the same second speech confidence metric SM2 is determined for all frames. Alternatively, the ratio is determined based on a fraction of the average spectral energy of a second context window or a fraction of the average spectral energy of each frame of the speech signal and the output speech signal.

[0065] This spectral energy ratio can be used directly as the second speech activity metric SM2.

[0066] The speech separator of the second speech classifier 1b may be the speech separator 20 used in the example embodiment described in connection with FIG. 1 or may be a separate speech separator. Thus, in some embodiments, the output audio signal (including isolated speech) from the speech separator of the second speech classifier 1b is provided to a gating unit 10 controlled by the first binary speech classifier SB1 or the enhanced binary speech classifier SBE. In some embodiments, the second speech classifier 1b is not used, or a second speech classifier 1b without a speech separator is utilized, whereby the speech separator 20 may be an external speech separator 20 that predicts a speech-separated audio signal having isolated speech. In such embodiments, the speech-separated audio signal is provided to a gating unit 10 controlled by the first binary speech classifier SB1 or the enhanced binary speech classifier SBE. Furthermore, an external speech separator may be used even if the second speech classifier 1b comprises a speech classifier which generates an output audio signal, the speech separated audio signal being fed to a gating unit, and the output audio signal being used to determine the second speech confidence metric SM2.

[0067] The incorporation of the second speech classifier 1b together with the first speech classifier 1a will be described in detail with reference to the flowchart of FIG. 3. The input speech signal acquired in step S1 is also provided to the second speech classifier 1b, which determines a second speech confidence metric SM2 for each frame of the input speech signal in step S7. Like the first speech confidence metric SM1, the second speech confidence metric SM2 may be in the range [0, 1] or any other suitable range. Because the second speech classifier 1b is different from the first speech classifier 1a, a frame-by-frame comparison of the first and second speech confidence metrics SM1 and SM2 will generally indicate that the first and second speech classification metrics SM1 and SM2 are different from each other. Of course, the first and second speech classification metrics SM1 and SM2 may be the same for some frames.

[0068] 7, the second speech confidence metric SM2 is supplied to the second adaptive threshold applicator 2b, which, in step S8, binarizes the second speech confidence metric SM2 for each frame using a second threshold comparator 21b to classify each frame as a speech frame or a non-speech frame. The second threshold comparator 21b uses a second predetermined threshold T2 to determine whether each frame is a speech frame or a non-speech frame by binarizing the second speech confidence metric SM2 with the second predetermined threshold T2. The second predetermined threshold T2 is 、 The first predetermined threshold T1 may be equal to or different from the first predetermined threshold T2. For example, T1 and T2 may both be equal to 0.5, or T1 may be equal to 0.5 and T2 may be lower, such as 0.1 or 0.2. In some embodiments, as described above, the second speech classifier 1b comprises a speech separator and determines the second speech confidence metric SM2 based on the energy ratio. In such embodiments, it may be beneficial to set T2 lower than T1, for example, T1 greater than 0.4 (e.g., equal to 0.5) and T2 less than 0.3 (e.g., equal to 0.2 or 0.1).

[0069] Once each frame has been assigned a speech or non-speech label based on the second speech confidence metric SM2, this information can be utilized in a variety of ways to improve the final speech classification accuracy of the speech classification system 10.

[0070] In one embodiment, a second number of speech frames NoF2, determined based on the second speech confidence metric SM2, is counted within a second context window 40b by a speech ratio calculator 22b of the second adaptive threshold applicator 2b. The second context window 40b may be of the same type (i.e., in terms of the number of look-ahead and / or look-back frames) or may be of a different type. In the example shown in FIG. 6, the total context window length of the second context window 40b is 3. That is, if frame 45 is a given frame, the second context window 40b includes look-back frame 44, the given frame 45 itself, and one look-ahead frame 46. This illustrative example assumes that any frames preceding frame 41 are non-speech (F) frames, and that all frames following frame 46 are speech (T) frames. Optionally, the speech ratio calculator 22b may calculate a second speech ratio r2. 、 It is determined as the ratio of the number of speech frames NoF2 to the length W2 of the second context window 40b.

[0071] Next, in step S9, for each frame, a second adaptive threshold T a2 is determined by the second adaptive threshold calculator 24b based on the number of speech frames NoF2 in the second context window or the second speech ratio r2. 。 A second adaptive threshold level T a2 is supplied to the second binary metrics extractor 26b, which in step S10 binarizes the second speech confidence metric SM2 to determine a second binary speech classification index SB2. The second adaptive threshold calculator 24b can determine an adaptive threshold using a threshold function. The threshold function implemented by the second adaptive threshold calculator 24b is monotonically decreasing and is, for example, similar to or identical to any of the threshold functions A to D in Equations 1 to 4 above.

[0072] Thus, in this embodiment, the second adaptive thresholder 2b performs the corresponding function of the first adaptive thresholder 2a described in detail in relation to Figure 3 above, except that the second adaptive thresholder 2b operates on the second speech confidence metric SM2 from the second classifier 1b.

[0073] The two speech classifiers 1a and 1b output two potentially different binary speech classification indices SB1 and SB2 for one or more frames as a result of post-processing performed by the first and second adaptive threshold applicators 2a. For example, for a given frame, the first binary speech classification indices SB1 are true (T, indicating speech) and the second binary speech classification indices SB2 are false (F, indicating non-speech). The two binary speech classification indices SB1 and SB2 may be provided to an enhanced binary speech extractor 3 to combine them to form an enhanced binary speech classification indices SBE. In step S11, the enhanced binary speech extractor 3 determines the enhanced binary speech classification indices SBE as an either-or combination of SB1 and SB2. That is, SBE is true if at least one of SB1 and SB2 is true, and false otherwise. Alternatively, the enhanced binary speech extractor 3 may implement a strict "AND" combination of SB1 and SB2, i.e., SBE is true only if both SB1 and SB2 are true, and false otherwise. The enhanced binary speech classification index SBE may be the final output of the speech classification system.

[0074] In the embodiments described above, the two speech classifiers 1 a, 1 b and the adaptive thresholders 2 a, 2 b operate independently of each other. However, in some embodiments, it has been found that the benefits of using two classifiers can be more effectively exploited by allowing the adaptive thresholders 2 a, 2 b to communicate with each other.

[0075] In one embodiment, the first number of speech frames NoF1 or the first speech ratio r1 determined by the first adaptive threshold applicator 2a is sent to the second adaptive threshold applicator 2b and used to control the second adaptive threshold applicator 2b. This means that the second adaptive threshold applicator 2b does not need the threshold comparator 21b or the speech ratio calculator 22b, because the first number of speech frames NoF1 or the first speech ratio r1 is provided from the first speech applicator to replace the second number of speech frames NoF2 or the second speech ratio r2. The adaptive threshold calculator 24b then determines the adaptive threshold using a threshold function. For example, a threshold function f defined as follows: E (r)

number

[0076] The opposite configuration is also envisaged, i.e. the second adaptive threshold applicator determines the second number of frames NoF2 or the second speech rate r2 and supplies this information to the first adaptive threshold applicator 2a to control the first adaptive threshold calculator 2a.

[0077] In general, the example described above works well for both music content and movie / sports content. Two classifiers 1a and 1b operate in parallel, and their respective binary speech classification metrics SB1 and SB2 are combined by a final binary speech extractor 3. Optionally, NoF1 or r1 of the first classifier can be fed to the second classifier, or vice versa.

[0078] For some complex content types, the agreement between the first classifier 1a and the second classifier 1b may reveal additional useful information for determining a more accurate adaptive threshold. For example, the parameters of the adaptive threshold function implemented by each adaptive threshold applier 2a, 2b may be controlled based on the relationship between NoF1 / r1 and NoF2 / r2. Additionally or alternatively, the initially determined NoF1 / r1 and NoF2 / r2 (and / or the initially determined first and second adaptive thresholds) may be modified (e.g., increased or decreased) based on the degree of agreement between NoF1 / r1 and NoF2 / r2. The modified NoF1 / r1 and NoF2 / r2 (and / or the modified first and second adaptive thresholds) may be used as a substitute for the initially determined NoF1 / r1 and NoF2 / r2 (and / or the first and second adaptive thresholds). The actual method of controlling the adaptive threshold function is determined based on the type of classifier 1a, 1b used. For example, if one classifier is more accurate at speech detection, its speech confidence metric should have a greater influence on the final binary speech classification than the speech confidence metric of the other classifier when NoF1 / r1 or NoF2 / r2 are low.

[0079] In some embodiments, there are four main cases to consider regarding the different relationships between NoF1 / r1 and NoF2 / r2: 。 These cases are illustrated for a setting where the first classifier 1a has a high false negative rate (i.e., makes conservative classifications that tend to miss some speech) and the second classifier 1b has a high false positive rate (i.e., makes exaggerated classifications that tend to classify some non-speech as speech). Other classifiers with other characteristics may benefit from similar or different interactions between NoF1 / r1 and NoF2 / r2.

[0080] In Case 1, NoF1 / r1 is low and NoF2 / r2 is high. In this case, the first classifier 1a detects less speech than the second classifier 1b. In this example, this is typical for music content in the second speech classifier 1b. In this case, the adaptive threshold of the second adaptive threshold applicator 2b is configured to increase as NoF1 / r1 decreases. As shown in FIG. 7, this can be achieved by providing NoF1 / r1 to an optional speech ratio adjuster 23b, which adjusts NoF2 / r2 such that decreasing NoF2 / r2 results in a subsequent increase in the adaptive threshold in adaptive threshold calculator 24b. Additionally or alternatively, NoF1 / r1 is provided to optional adaptive threshold adjuster 25b, which increases the adaptive threshold output from adaptive threshold calculator 24b.

[0081] This adjustment can have the effect of avoiding false positives caused by the second classifier 1b. a2 increases, resulting in fewer frames being labeled as speech frames. The terms "high" and "low" refer to NoF1 or r1 being above or below a first case threshold, and the same applies to NoF2 or r2 being high or low above or below a second case threshold.

[0082] In case 2, NoF1 / r1 is high and NoF2 / r2 is high. This means that both classifiers 1a and 1b identify many utterances, and the first classifier 1a probably does not have many false negatives. In this case, the second classifier 1b is only considered if the second classifier has very high confidence. Therefore, at higher NoF1 / r1, the second adaptive threshold is increased (using speech rate adjuster 23b or adaptive threshold adjuster 24b). 、 The second classifier 1b avoids introducing false positives.

[0083] In case 3, NoF1 / r1 is high and NoF2 / r2 is low. This case rarely occurs in the illustrated setting. However, since the first speech classifier 1a identifies a lot of speech data, the false negative rate is likely low. In case 3, a similar approach as in case 2 is applied. However, at higher NoF1 / r1, the second adaptive threshold is increased. 、 This avoids the second classifier 1b introducing false positives.

[0084] In case 4, NoF1 / r1 is low and NoF2 / r2 is low. In this case, both classifiers 1a and 1b have a low, but non-zero, number of speech frames or speech rate. This is likely due to sparse speech content rather than music content or content that simultaneously burdens both classifiers. In this case, the adaptive threshold of the second adaptive thresholder 2b is decreased as NoF1 / r1 decreases. This allows the second adaptive thresholder 2b to identify more speech if it matches the first classifier 1a and incorporate more speech frames identified by the second classifier 1b. However, some testing has confirmed that case 4 can also occur with some types of music. That is, in some embodiments, case 4 is not used, and only cases 1 through 3 are used.

[0085] The above four cases are applied to four types of audio content that meet the above requirements (i.e., whether NoF1 / r1 and NoF2 / r2 are above or below the corresponding case thresholds). In some embodiments, it is determined which of the four cases is valid for each frame, and the adaptive threshold function is modified accordingly, for example, by an additive or multiplicative adjustment term or coefficient introduced by speech rate adjuster 23b or adaptive threshold adjuster 24b. It is also contemplated that the parameters of the adaptive threshold functions implemented in each adaptive threshold applicator 2a, 2b may be dynamically tuned or the functions may be swapped depending on which of the four cases applies in the current frame.

[0086] Furthermore, while the above cases 1 to 4 relate to modifying the second NoF2 / r2 of the second adaptive thresholder 2b based on the NoF1 / r1 of the first adaptive thresholder 2a, the opposite case is also possible, i.e., modifying the first NoF1 / r1 of the first adaptive thresholder 2a based on the NoF2 / r2 of the second adaptive thresholder 2b. For example, in case 1 where NoF1 / r1 is low and NoF2 / r2 is high (e.g., indicating music content including vocals), NoF1 / r1 is decreased based on the difference between NoF1 / r1 and NoF2 / r2. As the difference between NoF1 / r1 and NoF2 / r2 increases, NoF1 / r1 is decreased. Generally, as NoF1 / r1 decreases, the first adaptive threshold increases. This, in turn, helps achieve a higher first adaptive threshold to further avoid false positives. Or directly based on the difference between NoF1 / r1 and NoF2 / r2 、 Increase the first adaptive threshold, i.e., as the difference between NoF1 / r1 and NoF2 / r2 increases 、 Increase the first adaptive threshold. For example, in case 1, NoF1 / r1 is decreased and / or the first adaptive threshold is increased only if NoF1 / r1 and NoF2 / r2 are sufficiently inconsistent (i.e., there is a sufficiently large difference).

[0087] In Case 2, classifiers 1a and 1b detect high NoF1 / r1 and high NoF2 / r2, respectively, which may indicate movie content or other speech-heavy content. In this case, NoF1 / r1 is increased or the first adaptive threshold is decreased based on the difference between the detected NoF1 / r1 and NoF2 / r2. As the difference between NoF1 / r1 and NoF2 / r2 decreases, NoF1 / r1 is increased and / or the first adaptive threshold is decreased. For example, in Case 2, NoF1 / r1 is increased and / or the first adaptive threshold is decreased only if NoF1 / r1 and NoF2 / r2 are sufficiently consistent (i.e., the difference is sufficiently small).

[0088] In case 3, the first adaptive threshold can be maintained without the influence of NoF2 / r2. 、 The number of speech frames or speech rate for both classifiers 1a, 1b is low. This may be the result of audio content with sparse speech or difficult audio content that burdens both classifiers 1a, 1b (rather than triggering the second classifier 1b to detect speech, such as music). In Case 4, the first adaptive threshold is decreased and / or NoF1 / r1 is increased to capture more speech frames identified by the first classifier 1a.

[0089] It is also possible to implement both types of correction simultaneously, ie, correcting NoF1 / r1 based on NoF2 / r2 and correcting NoF2 / r2 based on NoF1 / r1.

[0090] In some aspects, the first adaptive threshold applicator uses the first speech rate r1 and the second speech rate r2 to form a modified speech rate r' that replaces the first speech rate r1. The modified speech rate r' may be determined, for example, using the following formula:

number

[0091] The parameter u determines the steepness of the exponential scaling factor that modifies the scaling of r1 or NoF1 to form r' or MNoF. In FIG. 8, the value of the exponential scaling factor is shown as the ratio r1 / r2 for different values ​​of the parameter u. The scaling factor u is adjusted based on the ratio r1 / r2 and r1. In some embodiments, the value of the scaling factor u is set according to the function shown in FIG. 9, where a higher value of u is assigned when r2 is greater than 0.4 and r1 is less than 0.2. With this type of function, the u parameter becomes larger when r1 is smaller than r2, thereby reducing the modified speech ratio and achieving more conservative speech classification. In particular, when the second classifier 1b is implemented using a speech separator, this configuration of u has been shown to reduce the false positive rate in music by correctly labeling singing voices as non-speech.

[0092] Since the number of speech frames NoF1, NoF2 or the speech ratio r1, r2 of either adaptive threshold applicator 2a, 2b can be sent to the other adaptive threshold applicator 2a, 2b, it is assumed that only one of the first and second binary speech classification indicators SB1, SB2 is determined as the final binary classification of speech classification system 10. For this purpose, in some embodiments, final binary speech extractor 3 may be omitted, and one of adaptive threshold calculators 1a, 1b outputs the number of speech frames or the speech ratio, which output is supplied to the other adaptive threshold calculator 1a, 1b, which determines the adaptive threshold for final binarization based on both NoF1 and NoF2 or both r1 and r2. For example, the first adaptive threshold applicator 2a determines the modified MNoF or r' value using Equation 6, or implements extended threshold functions f(NoF1, NoF2), f(r1, r2) that map each combination of NoF1 and NoF2, or r1 and r2, to an adaptive threshold.

[0093] Unless otherwise indicated, and as will be apparent from the following description, the terms "processing," "computing," "calculating," "determining," "analyzing," and the like in the description of this disclosure are understood to refer to acts and / or processes by computer hardware or computing systems or similar electronic computing devices that manipulate and / or transform data represented as physical (e.g., electronic) quantities into other data represented as similar physical quantities.

[0094] In the above description of exemplary embodiments of the present disclosure, it should be noted that various features may be grouped together in a single embodiment, drawing, or description thereof for the purpose of simplifying the disclosure and facilitating understanding of one or more inventive aspects. However, this method of disclosure should not be interpreted as an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects reside in less than all features of a single embodiment disclosed above. Accordingly, the claims following the detailed description are expressly incorporated into this detailed description, with each claim standing on its own as an independent embodiment of the present disclosure. Furthermore, while some embodiments described herein may include some features included in other embodiments but not others, combinations of features from different embodiments will be understood by those skilled in the art to be within the scope of the present disclosure and constitute different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

[0095] Furthermore, some of the embodiments described herein are described as methods or combinations of elements of methods that can be implemented by a processor or other function-performing means of a computer system. Thus, a processor having instructions for carrying out such a method or element of a method constitutes a means for carrying out the method or element of a method. Where a method includes several elements (e.g., several steps), no ordering of such elements is implied unless otherwise specified. Furthermore, elements described herein for apparatus embodiments are examples of means for carrying out the functions performed by such elements for implementing embodiments of the present disclosure. Many specific details are described in the description. However, it will be understood that embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure an understanding of this description. Thus, although specific embodiments of the present disclosure have been described, those skilled in the art will understand that other or further modifications can be made without departing from the spirit of the present disclosure. All such changes and modifications are intended to be within the scope of the present disclosure.

[0096] Various aspects of the present disclosure can be understood from the following enumerated example embodiments (EEE):

[0097] EEE1. 1. A method for performing speech classification on an audio signal, comprising: obtaining the audio signal comprising a sequence of audio frames; for each speech frame, determining, using a first speech classifier, a first speech confidence metric indicative of the likelihood that speech is present in the speech frame; For each given audio frame in at least a subset of said sequence of audio frames: classifying each respective speech frame in a first context window associated with the given speech frame as a speech frame or a non-speech frame by comparing the first speech confidence metric of each speech frame against a first predetermined threshold; determining an adaptive threshold based on a number of speech frames in the first context window; determining a first binary speech classification index for the given frame of speech based on the first speech confidence metric and the adaptive threshold; A method comprising:

[0098] EEE2. The method of EEE1, wherein the adaptive threshold is determined using a function that maps the number of frames of speech in the first context window to an adaptive threshold.

[0099] EEE3. The method of EEE2, wherein the function monotonically decreases as the number of speech frames in the first context window increases.

[0100] EEE4. 10. The method of claim 9, wherein determining the first binary utterance classification index comprises binarizing the first utterance confidence metric using the adaptive threshold.

[0101] EEE5. determining, for each audio frame, a second speech confidence metric indicative of the likelihood of speech being present in the frame using a second speech classifier different from the first speech classifier; For each given audio frame in the subset of the sequence of audio frames: classifying each respective speech frame in a second context window associated with the given speech frame as a speech frame or a non-speech frame by comparing the second speech confidence metric of each frame against a second predetermined threshold; determining a second adaptive threshold based on the number of speech frames in the second context window and / or the number of speech frames in the first context window, and determining an enhanced binary speech classification index for the given audio frame based on the adaptive threshold and the second adaptive threshold; and / or the adaptive threshold is based on the number of speech frames in the first context window and the number of speech frames in the second context window. And, The method according to any one of EEE1 to 3, further comprising:

[0102] EEE6. the second speech classifier comprises a speech separator configured to generate an output audio signal having isolated speech content separated from the input audio signal; The method further comprises: processing the audio signal with the speech separator to obtain the output audio signal having isolated speech content; determining the second speech confidence metric based on a spectral energy ratio between a spectral energy metric of the output speech signal and a spectral energy metric of the speech signal; The method according to EEE5, comprising:

[0103] EEE7. The spectral energy ratio is an average spectral energy metric of the output speech signal over all frames of the speech signal, over all frames of the second context window, or for each frame individually; the average spectral energy metric of the speech signal over all frames of the speech signal, over all frames of the second context window, or for each frame individually; The method of claim 6, wherein the ratio is determined as

[0104] EEE8. The adaptive threshold is based on the number of speech frames in the first context window and the number of speech frames in the second context window, and the method further comprises: determining the first binary speech classification index by binarizing the first speech confidence metric using the adaptive threshold; The method according to any one of EEE5 to 7, comprising:

[0105] EEE9. Determining the adaptive threshold based on the number of speech frames in the first context window and the number of speech frames in the second context window includes: obtaining an extension function that links the number of speech frames in the first context window and the number of speech frames in the second context window to a variable threshold; evaluating the expansion function using the number of speech frames in the first context window and the number of speech frames in the second context window; The method according to EEE8, comprising:

[0106] EEE10. when the number of speech frames in the first context window is greater than a first predetermined number and the number of speech frames in the second context window is less than a second predetermined number, the expansion function is based only on the number of speech frames in the first context window; when the number of speech frames in the first context window is less than or equal to the first predetermined number and the number of speech frames in the second context window is greater than or equal to the second predetermined number, the extended threshold function is based on both the number of speech frames in the first context window and the number of speech frames in the second context window; Method as described in EEE9.

[0107] EEE11. For each given audio frame in the subset of the sequence of audio frames: determining a second adaptive threshold based on the number of speech frames in the second context window and / or the number of speech frames in the first context window; determining the first binary speech classification index by binarizing the first speech confidence metric for the given frame of speech using a first variable threshold; determining a second binary speech classification index by binarizing the second speech confidence metric for the given frame of speech using a second variable threshold; determining the enhanced binary speech classification index based on the first binary speech classification index and the second binary speech classification index; The method according to any one of EEE5 to 10, further comprising:

[0108] EEE12. The method of claim 8, wherein the enhanced binary speech classification indicator indicates that an utterance is valid only if at least one of the first binary speech classification indicator and the second binary speech classification indicator indicates that the utterance is valid.

[0109] EEE13. providing the first binary speech classification index and the audio signal to a gating unit; applying a gating gain to the speech signal based on the first binary speech classification index by the gating unit to form a gated speech signal; The method according to any one of EEE1 to 4, further comprising:

[0110] EEE14. providing the enhanced binary speech classification index and the audio signal to a gating unit; applying a gating gain to the speech signal based on the enhanced binary speech classification index by the gating unit to form a gated speech signal; The method according to any one of EEE5 to 12, further comprising:

[0111] EEE15. processing the audio signal with a speech separator to obtain a speech-separated audio signal having isolated speech content; applying the gating gain to the speech-segregated audio signal by the gating unit; The method of EEE13 or 14, further comprising:

[0112] EEE16. The method of any of the preceding EEEs, wherein the first context window associated with the given audio frame includes the given audio frame.

[0113] EEE17. The method of claim 8, wherein the first context window includes at least one look-ahead frame that temporally follows the given audio frame and at least one look-back frame that temporally precedes the given audio frame.

[0114] EEE18. The method of any preceding EEE, wherein the duration of the first or second context window is at least 5 seconds, at least 10 seconds, at least 20 seconds, or at least 30 seconds.

[0115] EEE19. The method of any of the preceding EEE methods, wherein the first predetermined threshold is between 0.1 and 0.6, such as about 0.5 or about 0.3.

[0116] EEE20. A computer program product comprising instructions that, when executed by a computer, cause the computer to carry out a method according to any one of EEE1 to EEE19.

[0117] EEE21. A computer-readable storage medium storing a computer program according to EEE20.

[0118] EEE22. A system comprising one or more processors configured to perform the method described in any of EEE1-19.

Claims

1. 1. A method for performing speech classification on an audio signal, comprising: obtaining the audio signal comprising a sequence of audio frames; for each speech frame, determining, using a first speech classifier, a first speech confidence metric indicative of the likelihood that speech is present in the speech frame; For each given audio frame in at least a subset of the sequence of audio frames: classifying each respective speech frame in a first context window associated with the given speech frame as a speech frame or a non-speech frame by comparing the first speech confidence metric of each speech frame against a first predetermined threshold; determining an adaptive threshold based on a number of speech frames in the first context window; determining a first binary speech classification index for the given frame of speech based on the first speech confidence metric and the adaptive threshold; A method comprising:

2. The method of claim 1 , wherein the adaptive threshold is determined using a function that maps the number of frames of speech in the first context window to an adaptive threshold.

3. The method of claim 2 , wherein the function monotonically decreases as the number of speech frames in the first context window increases.

4. The method of claim 1 , wherein determining the first binary speech classification index comprises binarizing the first speech confidence metric using the adaptive threshold.

5. determining, for each audio frame, a second speech confidence metric indicative of the likelihood that speech is present in the frame using a second speech classifier different from the first speech classifier; For each given audio frame in the subset of the sequence of audio frames: classifying each respective speech frame in a second context window associated with the given speech frame as a speech frame or a non-speech frame by comparing the second speech confidence metric of each frame against a second predetermined threshold; determining a second adaptive threshold based on the number of speech frames in the second context window and / or the number of speech frames in the first context window; and determining an enhanced binary speech classification index for the given audio frame based on the adaptive threshold and the second adaptive threshold; and / or the adaptive threshold is based on the number of speech frames in the first context window and the number of speech frames in the second context window; And, The method of any one of claims 1 to 3, further comprising:

6. the second speech classifier comprises a speech separator configured to generate an output audio signal having isolated speech content separated from the input audio signal; The method further comprises: processing the audio signal with the speech separator to obtain the output audio signal having isolated speech content; determining the second speech confidence metric based on a spectral energy ratio between a spectral energy metric of the output speech signal and a spectral energy metric of the speech signal; The method of claim 5 , comprising:

7. The spectral energy ratio is an average spectral energy metric of the output speech signal over all frames of the speech signal, over all frames of the second context window, or for each frame individually; the average spectral energy metric of the audio signal over all frames of the audio signal, over all frames of the second context window, or for each frame individually; The method of claim 6, wherein the ratio is determined as a ratio between

8. The adaptive threshold is based on the number of speech frames in the first context window and the number of speech frames in the second context window, and the method further comprises: determining the first binary utterance classification index by binarizing the first utterance confidence metric using the adaptive threshold; The method of any one of claims 5 to 7, comprising:

9. Determining the adaptive threshold based on the number of speech frames in the first context window and the number of speech frames in the second context window includes: obtaining an extension function that links the number of speech frames in the first context window and the number of speech frames in the second context window to a variable threshold; evaluating the expansion function using the number of speech frames in the first context window and the number of speech frames in the second context window; The method of claim 8, comprising:

10. when the number of speech frames in the first context window is greater than a first predetermined number and the number of speech frames in the second context window is less than a second predetermined number, the expansion function is based only on the number of speech frames in the first context window; when the number of speech frames in the first context window is less than or equal to the first predetermined number and the number of speech frames in the second context window is greater than or equal to the second predetermined number, the extended threshold function is based on both the number of speech frames in the first context window and the number of speech frames in the second context window; 10. The method of claim 9.

11. For each given audio frame in the subset of the sequence of audio frames: determining a second adaptive threshold based on the number of speech frames in the second context window and / or the number of speech frames in the first context window; determining the first binary speech classification index by binarizing the first speech confidence metric for the given frame of speech using a first variable threshold; determining a second binary speech classification index by binarizing the second speech confidence metric for the given frame of speech using a second variable threshold; determining the enhanced binary speech classification index based on the first binary speech classification index and the second binary speech classification index; The method of any of claims 5 to 10, further comprising:

12. 12. The method of claim 11, wherein the enhanced binary speech classifier indicates that an utterance is valid only if at least one of the first binary speech classifier and the second binary speech classifier indicates that the utterance is valid.

13. providing the first binary speech classification index and the audio signal to a gating unit; applying a gating gain to the speech signal based on the first binary speech classification index by the gating unit to form a gated speech signal; The method of any one of claims 1 to 4, further comprising:

14. providing the enhanced binary speech classification index and the audio signal to a gating unit; applying a gating gain to the speech signal based on the enhanced binary speech classification index by the gating unit to form a gated speech signal; 13. The method of any of claims 5 to 12, further comprising:

15. processing the audio signal with a speech separator to obtain a speech-separated audio signal having isolated speech content; applying the gating gain to the speech-segregated audio signal by the gating unit; 15. The method of claim 13 or 14, further comprising:

16. The method of claim 1 , wherein the first context window associated with the given audio frame includes the given audio frame.

17. 17. The method of claim 16, wherein the first context window includes at least one look-ahead frame that temporally follows the given audio frame and at least one look-back frame that temporally precedes the given audio frame.

18. 18. The method of any preceding claim, wherein the duration of the first or second context window is at least 5 seconds, at least 10 seconds, at least 20 seconds or at least 30 seconds.

19. 19. The method of any of claims 1 to 18, wherein the first predetermined threshold is between 0.1 and 0.6, such as about 0.5 or about 0.

3.

20. A computer program product comprising instructions that, when said program is executed by a computer, cause said computer to carry out the method according to any of claims 1 to 19.

21. A computer-readable storage medium storing the computer program of claim 20.

22. A system comprising one or more processors configured to perform the method of any of claims 1 to 19.

Images