Method and system for robust processing of speech classifiers

Abstract

The present disclosure relates to a method and system for performing speech classification on an audio signal. The method includes obtaining an audio signal comprising a sequence of audio frames, and determining, for each audio frame, a first speech confidence metric using a first speech classifier. For each given audio frame of at least a subset of the sequence of audio frames, the method comprises classifying each respective audio frame of a first context window associated with the given audio frame as a speech frame or a non-speech frame by comparing a first speech confidence metric for the respective audio frame with a first predetermined threshold, an adaptive threshold is determined based on a number of speech frames of the first context window, and a first binary speech classification indicator is determined for the given audio frame based on the first speech confidence metric and the adaptive threshold.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 483,584 filed on February 7, 2023 and U.S. Provisional Patent Application No. 63 / 592,686 filed on October 24, 2023, the entire contents of both applications are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to methods and systems for performing speech classification of audio signals. Background Art

[0004] For many types of audio processing, dialogue or speech classifiers are used to identify the presence of dialogue or speech in an audio signal. For example, in dialogue enhancement (DE), it is often very useful to extract a dialogue classifier that indicates at which point in time in the audio signal dialogue is active, for example, in order to control when different types of dialogue enhancement processing should be applied.

[0005] There are many different types of dialogue classifiers. In recent years, many neural network-based dialogue classifiers have been proposed. Neural network-based dialogue classifiers, as well as other types of dialogue classifiers, typically predict a dialogue confidence score (typically a value between 0 and 1) that indicates the likelihood of dialogue being present in each frame of the audio signal. By comparing the dialogue confidence score to a predetermined threshold (typically 0.5), a classification in the form of a predicted binary dialogue label is obtained, indicating for each frame whether dialogue is "true" or "false."

[0006] During the training of a dialogue classifier, a large set of training frames is fed to the dialogue classifier, which predicts a dialogue confidence for each frame. A loss (e.g., binary cross-entropy loss) is then calculated by comparing the dialogue confidence with the ground-truth dialogue labels. The learnable parameters of the dialogue classifier are updated to minimize the average loss until training converges.

[0007] The trained classifier can then be used in a variety of applications where it is useful to know when dialogue is present in an audio signal. For example, a simple dialogue separator can be implemented by applying a gating function to the audio signal based on the predicted dialogue marker. The gating function can, for example, be configured to mute or attenuate the audio signal when the dialogue marker indicates the absence of dialogue, leaving a processed audio signal with isolated dialogue. Summary of the Invention

[0008] A drawback of existing speech classifiers is that the misclassification of frames is strongly dependent on the type of audio content. For example, music content proves to be particularly challenging, as speech classifiers tend to classify certain types of music as speech, resulting in many false-positive alarms. Therefore, an object of the present disclosure is to provide an improved method for classifying speech that is accurate and applicable to a wide range of different audio content types.

[0009] According to a first aspect of the present disclosure, there is provided a method for speech classification of an audio signal, comprising: obtaining an audio signal comprising a sequence of audio frames; determining a first speech confidence metric for each audio frame using a first speech classifier, wherein the first speech confidence metric indicates a likelihood of speech being present in the audio frame. The method further comprises, for each given audio frame in at least one subset of the sequence of audio frames, classifying the corresponding audio frame as a speech frame or a non-speech frame by comparing the first speech confidence metric for each corresponding audio frame in a first context window associated with the given audio frame with a first predetermined threshold; determining an adaptive threshold based on the number of speech frames in the first context window; and determining a first binary speech classification indicator for the given audio frame based on the first speech confidence metric and the adaptive threshold.

[0010] The term "speech" refers to any type of conversational voice communication, such as the speech of a single voice (monologue), two voices (dialogue), or multiple voices. However, the term "speech" does not include singing voices, as singing voices are typically classified as music content rather than speech content. However, since both singing voices and speech are human utterances, speech classifiers often have difficulty distinguishing the two. Typically, speech classifiers generate many false positives for music content with singing voices by incorrectly classifying singing voices as speech.

[0011] Regarding the adaptive threshold based on the number of speech frames in the first context window, the threshold for determining whether speech is present can be adjusted on a frame-by-frame basis based on the context of each frame. The adaptive threshold improves classification accuracy for many types of audio content, including reducing false positives for music content with singing voices.

[0012] In contrast, using a fixed threshold of 0.5 to binarize speech confidence can achieve an equal false negative error rate for speech tag prediction. In other words, the false negative rate (i.e., speech is mistakenly classified as non-speech) is approximately equal to the false positive rate (i.e., non-speech is mistakenly classified as speech). Equal false negative error rates provide a well-balanced and comprehensive speech classifier that can be used for many types of audio content. However, for some applications, it may be more important that speech content is not missed than that non-speech content is correctly classified (i.e., classified as non-speech content). This can be achieved by using a lower threshold of, for example, 0.2 or even 0.1, so that more frames are marked as speech content. Although a lower threshold reduces false negatives, false positives increase. It has been found that low thresholds work well for movie content or sports audio content, but if the same low threshold is used for music content, the increased false positives may introduce level fluctuations and / or signal instability after processing.

[0013] With respect to the adaptive threshold of the present disclosure, the threshold is adjusted based on the number of speech frames detected in the first context window, which enables low error rates to be achieved for a wider variety of audio content types including sports audio content, movie audio content, and music.

[0014] According to some implementations, determining the first binary speech classification indicator includes binarizing the first speech confidence measure with an adaptive threshold.

[0015] The term "binarization" refers to converting a numerical value in a range (e.g., a continuous range, such as from 0 to 1) into a binary Boolean value with two states. Binarization is performed by comparing a numerical value with a threshold, wherein if the numerical value is higher than the threshold, the Boolean value is in a first state (true, T state), and if the numerical value is lower than the threshold, the Boolean value is in a second state (false, F state). When the numerical value is equal to the threshold, the Boolean value can be in the first state or the second state. Therefore, it should be understood that the binarization process can be configured so that if the numerical value is higher than or equal to the threshold, the Boolean value is in the first state, or configured so that if the numerical value is lower than or equal to the threshold, the Boolean value is in the second state. Therefore, a first binary speech classification indicator is obtained by binarization using an adaptive threshold that is updated and changed frame by frame.

[0016] According to some implementations, the method further includes determining, for each audio frame, a second speech confidence metric using a second speech classifier different from the first speech classifier, the second speech confidence metric indicating a likelihood of speech being present in the frame. The method further includes, for each given audio frame in the subset of the sequence of audio frames: classifying the corresponding audio frame as a speech frame or a non-speech frame by comparing the second speech confidence metric for each corresponding audio frame of a second context window associated with the given audio frame to a second predetermined threshold, and (a) determining a second adaptive threshold based on the number of speech frames in the second context window and / or the number of speech frames in the first context window, and determining an enhanced binary speech classification indicator for the given audio frame based on the adaptive threshold and the second adaptive threshold, or, (b) wherein the adaptive threshold is based on the number of speech frames in the first context window and the number of speech frames in the second context window.

[0017] In other words, the capabilities of two different speech classifiers can be combined to form a more accurate binary speech classification indicator. The number of speech frames in each context window is combined to determine a more accurate single adaptive threshold, or the number of speech frames in each context window is used to form corresponding (first and second) adaptive thresholds, whereby both thresholds are used to determine an enhanced binary classification indicator. Both approaches can improve classification accuracy and can be used alone or in combination.

[0018] According to a second aspect of the present disclosure, there is provided a computer program product comprising instructions, which, when a computer executes the program, causes the computer to perform the method of the first aspect of the present disclosure.

[0019] According to a third aspect of the present disclosure, a system is provided comprising one or more processors, wherein the one or more processors are configured to execute the method of the first aspect of the present disclosure.

[0020] The disclosures according to the second and third aspects have the same or equivalent advantages as the disclosure according to the first aspect.Any function described with respect to the method may have corresponding features in the system or computer program product. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Various aspects of the present disclosure will be described in more detail with reference to the accompanying drawings, in which exemplary embodiments are shown.

[0022] Figure 1 is a block diagram illustrating a speech separation system using a speech classification system according to some implementations.

[0023] Figure 2 is a block diagram illustrating a speech classification system implementing an adaptive threshold according to some implementations.

[0024] Figure 3 is a flow chart outlining a method for determining a binary speech classification metric via an adaptive threshold according to some embodiments.

[0025] Figure 4 is a block diagram illustrating a variable threshold applicator according to some embodiments.

[0026] Figure 5 is a graph illustrating an exemplary threshold function for determining an adaptive threshold based on a speech ratio.

[0027] Figure 6 is a block diagram illustrating a speech classification system 10 including two speech classifiers according to some embodiments.

[0028] Figure 7 is a block diagram illustrating a second variable threshold applicator that optionally obtains information from a first variable threshold applicator according to some embodiments.

[0029] Figure 8 is a graph illustrating a scaling function for determining a modified speech ratio according to some embodiments.

[0030] Figure 9 is a graph showing an exemplary function for setting the parameter u based on the relationship between the first voice ratio and the second voice ratio. DETAILED DESCRIPTION

[0031] The systems and methods disclosed in this application can be implemented as software, firmware, hardware, or a combination thereof. In hardware implementations, the division of tasks does not necessarily correspond to the division of physical units; on the contrary, a physical component can have multiple functions, and a task can be performed by several physical components in cooperation.

[0032] The computer hardware may be, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a smartphone, an AR / VR wearable device, an automotive infotainment system, a network device, a network router, a switch or a bridge, or any machine capable of executing (sequentially or otherwise) instructions that specify actions to be taken by the computer hardware. Furthermore, the present disclosure relates to any collection of computer hardware that, alone or in combination, executes instructions to perform any one or more of the concepts discussed herein.

[0033] Some or all of the components may be implemented by one or more processors that accept computer-readable (also referred to as machine-readable) code comprising an instruction set that, when executed by the one or more processors, results in the implementation of at least one of the methods described herein. Any processor that is capable of executing (sequentially or otherwise) an instruction set that specifies an action to be taken is included. Thus, an example is a typical processing system (e.g., computer hardware) that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may also include a storage subsystem that includes a hard drive, an SSD, RAM, and / or ROM. A bus subsystem for communication between components may be included. During execution of the software by the computer system, the software may reside in the storage subsystem and / or within the processor.

[0034] The one or more processors may operate as standalone devices or may be connected to, for example, a network to other processors. Such a network may be based on a variety of network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0035] The software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transient media). As known to those skilled in the art, the term "computer storage media" includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, various forms of physical (non-transitory) storage media, such as EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store the required information and can be accessed by a computer. In addition, as known to those skilled in the art, (transitory) communication media typically contain computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information transfer medium.

[0036] Figure 1An exemplary audio processing system 1 is shown that uses a speech classification system 10 to separate speech content from an audio signal. The audio signal is provided to a speech separator 20, which processes the audio signal to obtain the speech content. The speech separator 20 can be a neural network-based speech separator or any other type of speech separator 20. In a basic non-neural network implementation, the speech separator 20 is a bandpass filter configured to allow typical speech frequencies to pass while attenuating or completely blocking other frequencies. An example of typical speech frequencies is between 150 Hz and 8 kHz, with female speech typically extending from 350 Hz to 8 kHz and male speech typically extending from 150 Hz to 6 kHz. A narrower frequency band can also be used to capture the most important speech frequencies for maintaining intelligibility, such as a band from 300 Hz to 3.4 kHz. The output of the speech separator 20 is provided to a soft gating unit 30 controlled by the speech classification system 10.

[0037] The speech classification system 10 is configured to determine whether an audio signal includes speech content. Typically, the speech classification system 10 determines a binary speech classification indicator for each frame of the audio signal, where the binary speech classification indicator has a high value (true state, T) and a low value (false state, F) indicating whether speech is present in the frame. In other words, the speech classification system 10 will determine for each frame whether it is classified as a speech frame or a non-speech frame.

[0038] The soft gating unit 30 is configured to allow the output of the speech separator 20 (i.e., isolated speech) to pass through the soft gating unit 30 for speech frames, and to prevent the output of the speech separator 20 from passing through the soft gating unit 30 for non-speech frames. For example, because the speech separator 20 is not perfect, it will allow some residual non-speech audio content to pass through when speech is not present, and the soft gating unit 30 will act as a further filter to suppress this residual non-speech content by not allowing the content of non-speech frames to pass through the soft gating unit 30.

[0039] In some implementations, the soft gating unit 30 is configured to apply a 0 dB gain when speech is present in the frame and a lower gain (e.g., -25 dB) when speech is not present in the frame. Optionally, the gating unit 30 completely suppresses the output audio signal from the speech separator 20 when speech is not present in the frame. In order to make the transition between speech and non-speech portions less abrupt, the soft gating unit 30 can implement a finite transition time for the applied gain so that the gain applied to the non-speech frame smoothly transitions to the gain applied to the speech frame, and vice versa. Although the soft gating unit 30 is controlled by a binary speech classification indicator, it can be configured to apply a continuously varying gating, for example, by smoothing the binary speech classification indicator.

[0040] Therefore, the estimated speech audio signal output by the soft gating unit 30 may be a cleaner speech signal with enhanced intelligibility compared to the output of the speech separator 20 .

[0041] However, the functionality of the soft gating unit 30 (and the resulting speech isolation in the output estimated speech audio signal) is limited by the accuracy of the speech classifier 10, and a speech classification system 10 that is accurate for many types of audio content is desirable (e.g., accurate for music content as well as sports or movie content). Furthermore, it should be understood that the speech classification system 10 has many uses and can be, and is currently being, implemented in many applications. For example, in addition to Figure 1 In addition to the illustrated speech separation audio processing system, the speech classification system 10 can be used to label audio data for neural network training, to change audio rendering or audio processing techniques based on the presence of speech, to control noise suppression algorithms in, for example, teleconferencing or videoconferencing applications, etc. Thus, the new and improved speech classification system 10 can be used in many implementations.

[0042] Figure 2 is a block diagram detailing an example implementation of a speech classification system 10 for determining a binary speech classification indicator SB1. Figure 3 Referring to the flowchart in FIG. 1 , the operation of the speech classification system 10 will now be described in detail.

[0043] At step S1, the speech classification system 10 obtains an input audio signal. The speech classification system 10 includes a first speech classifier 1a, which is configured to determine a first speech confidence measure SM1 for each frame of the audio signal at step S2, the first speech confidence measure SM1 indicating the likelihood of speech being present in the audio frame. The first speech classifier 1a can be any type of speech classifier configured to determine the first speech confidence measure SM1 for each frame. For example, the first speech classifier 1a can be based on a trained neural network.

[0044] The first speech confidence measure SM1 determined in step S2 is typically a value defined in the range [0, 1], where 0 represents a 0% probability of speech being present in the audio frame, 1 represents a 100% probability of speech being present, and values ​​between 0 and 1 represent probabilities between 0% and 100%. Of course, the first speech confidence measure SM1 can be defined differently, for example, in the range [-1, 1] or [0, 100]. In the following, it will be assumed that the first speech confidence measure SM1 is defined in the range [0, 1], but those skilled in the art will understand that this is merely exemplary and that the speech classification system 10 can be implemented in a similar manner, where the first speech confidence measure SM1 is defined differently.

[0045] The first speech confidence measure SM1 of each frame is provided to an adaptive threshold applicator 2a, which first determines a first adaptive threshold with which the first speech confidence measure SM1 should be compared, and secondly applies the adaptive threshold by binarizing the first speech confidence measure SM1 with (i.e. using) the first adaptive threshold.

[0046] The adaptive threshold application 2a determines a first adaptive threshold on a frame-by-frame basis by analyzing the frames in a context window associated with each frame.

[0047] Further references Figure 4 , the operation of the adaptive threshold applicator 2a will now be described in more detail.

[0048] The first speech confidence measure SM1 of each frame 41-46 is provided to the threshold applicator 2a. Figure 4 As shown, the first speech confidence measure SM1 of the earliest frame 41 in time is 0.1, the first speech confidence measure SM1 of the subsequent frame 42 is 0.4, the first speech confidence measure SM1 of the subsequent frame 43 is 0.7, the first speech confidence measure SM1 of the subsequent frame 44 is 0.4, the first speech confidence measure SM1 of the subsequent frame 45 is 0.2, and the first speech confidence measure SM1 of the subsequent frame 46 is 0.9. The first speech confidence measure SM1 of each frame is provided to the threshold comparator 21, and the threshold comparator 21 binarizes each first speech confidence measure SM1 by comparing it with a predetermined first threshold T1 in step S3. In the example shown, the predetermined first threshold T1 is equal to 0.5, which is a typical value (of course, other values ​​are also possible). For example, the first predetermined threshold T1 can be between 0.1 and 0.6, such as approximately 0.5 or approximately 0.3. If the first speech confidence measure SM1 exceeds a first predetermined threshold value T1, the frame is classified as a speech frame (represented by a true T, e.g., a binary value of 1), otherwise the frame is classified as a non-speech frame (represented by a false F, e.g., a binary value of 0). In the example shown, frame 46 has a first speech confidence measure SM1 of 0.9, which exceeds the first predetermined threshold value, and thus frame 46 is designated as a speech frame (represented by a true state T).

[0049] The binary speech / non-speech classification result is provided to a speech ratio calculator 22, which determines the number of frames NoF1 classified as speech frames in the first context window 40 associated with each given frame. The first context window 40 of each given frame may include one or more of the following: (A) one or more frames before the given frame, (B) one or more frames after the given frame, and (C) the given frame itself. The first context window 40 may include multiple frames that collectively capture a temporal context of at least 5 seconds, at least 10 seconds, at least 20 seconds, or at least 30 seconds. That is, each (given) frame can be considered to be associated with its own corresponding first context window, wherein the corresponding first context window includes at least one of the following: one or more earlier frames and one or more later frames, and optionally the frame itself. Although the same type of first context window can be used for each frame (in terms of window length and the number of earlier / later frames), the set of frames included in the first context window of the first frame will be different from the set of frames included in the first context window of the second frame.

[0050] As an illustrative example, a first context window 40 associated with a given frame 45 includes the given frame 45, two preceding frames 43, 44, and a subsequent frame 46. This exemplary context window 40 thus includes a total of four frames, including the given frame 45, with a total context window length of W1 for the four frames. The context window 40 is then shifted so that the subsequent frame 46 plays the role of the given frame, and so on.

[0051] The exemplary context window mentioned above is merely an example, and it should be understood that many different types of context windows may be used. In some embodiments, context windows having a length exceeding ten frames, one hundred frames, or even one thousand frames may be used. For example, a context window having 235 frames (representing approximately 5 seconds of content) or 1407 frames (representing approximately 30 seconds of content) may be used.

[0052] Many types of context window sizes can be used with the inclusion of one or more look-back frames and / or one or more look-ahead frames. Figure 4 In the exemplary first context window 40 shown, it is also possible to define a context window that includes, for example, several (eg, three) review frames before a given frame, but does not include the given frame itself.

[0053] exist Figure 4, an exemplary context window comprising a given frame, two look-back frames, and one look-ahead frame is used. Speech ratio calculator 22 counts the number of speech frames NoF1 in the context window and finds zero, one, one, one, two, and two speech frames, since the given frames are frames 41, 42, 43, 44, 45, and 46, respectively (assuming that any frame before frame 41 is a non-speech F frame and any frame after frame 46 is a speech T frame). Thus, each frame is associated with a number of speech frames NoF1 value indicating the number of speech frames in the first context window 40 for that particular frame.

[0054] Optionally, the speech ratio calculator 22 determines a speech ratio r1 for each frame, which is the ratio between NoF1 and the total number of frames (i.e., the frame length W1) in the context window 40. In the present example, there are four frames in the context window 40, and speech ratios of 0.00, 0.25, 0.25, 0.25, 0.50, and 0.50 are given for frames 41, 42, 43, 44, 45, and 46, respectively.

[0055] It should be understood that for the illustrative example where the context window is only four frames long, the speech ratio r1 and NoF1 can take one of five possible values ​​of the speech ratio r1, namely 0, 0.25, 0.5, 0.75, and 1. In many implementations, the context window length is much longer, having at least ten, at least one hundred, or at least one thousand frames, whereby more r1 / NoF1 values ​​are possible and r1 / NoF1 can vary more smoothly from one frame to the next.

[0056] The NoF1 value or speech ratio r1 is provided to the adaptive threshold calculator 24 which calculates the adaptive threshold T for each frame 41-46 in step S4. a 1. The adaptive threshold Ta1 is updated frame by frame and can be determined, for example, using a threshold function f(NoF) or f(r). Then, in step S5, the adaptive threshold Ta1 for each frame is provided to the binary metric extractor 26, which binarizes the first speech confidence metric SM1 for each frame using the adaptive threshold Ta1 for each frame to obtain a binary speech classification indicator SB1. By comparing the exemplary binary speech classification indicator SB1 with the speech / non-speech classification performed by the threshold comparator 21, it can be seen that the classification results are different, and it has been found that using adaptive threshold classification, a lower false positive level can be achieved for many types of audio content. The first binary speech classification indicator SB1 can be used as an output of the speech classification system 10.

[0057] In some embodiments, the adaptive threshold calculator 25 uses the method of mapping the speech frame number NoF1 to the adaptive threshold T a The threshold function f(NoF1) or the speech ratio r1 is mapped to the adaptive threshold Ta1 A threshold function f(r1) is used to determine the adaptive threshold based on NoF1 or the speech ratio r1. There are many exemplary functions f(NoF1) or f(r1) that can be used, and in some implementations, the threshold function f(NoF1) or f(r1) is a monotonically decreasing function as the number of speech frames NoF1 in the first context window 40 increases (the speech ratio r1 increases). The term "monotonically decreasing" is intended to cover stronger requirements of monotonically non-increasing functions and monotonically strictly decreasing functions. That is, if it satisfies f(x)≥f(y) when y≥x, then the threshold function f(.) is monotonically non-increasing; if it satisfies f(x)>f(y) when y>x, then it is strictly monotonically decreasing.

[0058] Some examples of threshold functions f(r1) will now be given, but it will be appreciated that these are merely exemplary and that other functions may be used. It will also be appreciated that these functions may be converted to functions of the form f(NoF1) if r1 is replaced by NoF1 / W1, where W1 is the total number of frames in the first context window 40.

[0059] As a first exemplary threshold function (Example A), the sigmoid function is used:

[0060]

[0061] As a second exemplary threshold function (Example B), a linear approximation of the sigmoid function in Equation 1 is used:

[0062] g B (r1) = 0.5 + 2 (0.18 - r1) (Formula 2)

[0063] f B (r1)=max(g B (r1),0.1).

[0064] As a third exemplary threshold function (Example C), a linear approximation of Equation 1 with an upper limit is used:

[0065] g C (r1) = 0.5 + 2 (0.2 - r1) (Formula 3)

[0066] f C (r1)=min(max(g C (r1),0.1),0.6).

[0067] As a fourth exemplary threshold function (Example D), the linear approximation of Equation 3 is shifted left to obtain the following threshold function, which generates a smaller threshold to reclaim more speech:

[0068] gD (r1) = 0.2 + 2 (0.2 - r1) (Formula 4)

[0069] f D (r1)=min(max(g D (r1),0.1),0.6).

[0070] Figure 5 The function f is shown in A (r1),f B (r1) and f C (r1), the speech ratio r1 ranges from 0 (no speech frames in the context window 40) to 1 (all frames in the context window 40 are speech frames).

[0071] The above exemplary function f A (r1),f B (r1) and f C Any of (r1) can be implemented by the adaptive threshold calculator 24 to determine an adaptive threshold for each frame in step S4, and the adaptive threshold will result in lower false negatives for music and / or speech content.

[0072] In Table I, the error rates (expressed in percentages) for music content and speech / movie content obtained by a conventional speech classifier by comparing the first speech confidence measure SM1 with a static threshold T of T = 0.1 or T = 0.5 are shown and compared with an improved speech classification system 10 using an adaptive threshold, which is extracted using a threshold function AD for different context window lengths (here expressed in seconds instead of frames). Table I highlights the disadvantages of using a predetermined static threshold T for the final binarization of the first speech confidence measure SM1, namely that if a higher static threshold is used (e.g., T = 0.5), the false positive (FP) rate for music is quite low (1.62%), but the false negative (FN) rate for speech / movie content is very high (3.31%), which means that for speech / movie content, a lot of speech content is misclassified as non-speech. On the other hand, if a lower static threshold is used (e.g., 0.1), the FN rate for speech / movie content is improved (0.83%), resulting in much less speech content being lost. However, with a lower static threshold, the FP rate for music increases (to 4.89%), which means that a lot of music content is misclassified as speech.

[0073] Using the adaptive threshold-based speech classification system 10, the speech classification results obtained provide a lower FP rate for music content compared to a low static threshold T, and a lower FN rate for speech / movie content compared to a high static threshold. For example, using function C and a 30-second context window, the FP rate for music is 1.71%, while the FN and FP rates for speech / movie content are 1.30% and 6.44%, respectively. Thus, adaptive threshold processing provides speech classification that achieves good performance in both low FP rates for music and low FN rates for speech / movie content, meaning that it performs well for both music and speech, whereas static thresholding performs particularly well for only one type of audio content and poorly for the other. While the FP rate for speech / movie content remains relatively high, it is still an improvement over processing using a low static threshold (where T = 0.1). Furthermore, false positives for speech / movie content are generally less likely to be noticeable and disruptive to the listener than false negatives for speech / movie content or false positives for music content.

[0074] Table I

[0075]

[0076] Go to Figure 6 , shows that the speech classification system 10 according to some embodiments can be combined with a second speech classifier 1b, which assists the first speech classifier 1a, wherein the second speech classifier 1b is different from the first speech classifier 1a. By combining two different speech classifiers 1a, 1b, the strength of each classifier 1a, 1b can be utilized to achieve a more accurate binary speech classification indicator SB1 or an enhanced binary speech classification indicator SBE, which will be described below.

[0077] For example, the two speech classifiers 1a, 1b may be neural network based speech classifiers 1a, 1b which have different network architectures, have been trained differently and / or have been trained with different training data.

[0078] According to some embodiments, the second speech classifier 1b includes a speech separator that is configured to generate an output audio signal having isolated speech content separated from the input audio signal. The speech separator can, for example, be based on a trained neural network. The audio signal is provided to the speech separator, which predicts an output audio signal that includes isolated speech with any background audio content completely suppressed or removed. In order to determine the second speech confidence measure SM2 using the output audio signal, the ratio of the spectral energy of the output audio signal to the spectral energy of the input audio signal is determined and used as the second speech confidence measure SM2. The ratio can be determined for the entire signal, which means that the same second speech confidence measure SM2 is determined for all frames. Alternatively, the ratio is determined based on a portion of the average spectral energy of the second context window or a portion of the average spectral energy of each frame of the audio signal and the output audio signal.

[0079] The spectral energy ratio can then be used directly as the second speech activity measure SM2.

[0080] The second speech classifier 1b may be a speech separator in combination with Figure 1 The speech separator 20 used in the described exemplary implementation may be a separate speech separator. Thus, in some embodiments, the output audio signal (containing the isolated speech) from the speech separator in the second speech classifier 1b is provided to a gating unit 10, which is controlled by a first binary speech classification indicator SB1 or an enhanced binary speech classification indicator SBE. In some embodiments, the second speech classifier 1b is not used, or a second speech classifier 1b is utilized without a speech separator, whereby the speech separator 20 may be an external speech separator 20 that predicts a speech-separated audio signal containing the isolated speech. In such an implementation, the speech-separated audio signal is provided to a gating unit 10, which is controlled by a first binary speech classification indicator SB1 or an enhanced binary speech classification indicator SBE. Furthermore, even if the second speech classifier 1b includes a speech classifier that generates an output audio signal, an external speech separator may still be used, wherein the speech-separated audio signal is provided to the gating unit, and the output audio signal is used to determine the second speech confidence measure SM2.

[0081] Further references Figure 3, the combination of the second speech classifier 1b and the first speech classifier 1a will now be described in detail. The input audio signal obtained in step S1 is also provided to the second speech classifier 1b, which determines a second speech confidence measure SM2 for each frame of the input audio signal in step S7. Similar to the first speech confidence measure SM1, the second speech confidence measure SM2 can be a value in the range [0, 1] or any other suitable range. Since the second speech classifier 1b is different from the first speech classifier 1a, a frame-by-frame comparison of the first and second speech confidence measures SM1, SM2 will usually reveal that the first and second speech classification measures SM1, SM2 are different from each other. Of course, for some frames, it may appear that the first and second speech classification measures SM1, SM2 are the same.

[0082] Further references Figure 7 The second speech confidence measure SM2 is provided to the second adaptive threshold applicator 2b. In step S8, for each frame, the second adaptive threshold applicator 2b uses the second threshold comparator 21b to binarize the second speech confidence measure SM2 to classify each frame as a speech frame or a non-speech frame. The second threshold comparator 21b uses a second predetermined threshold T2 to determine whether each frame is a speech frame or a non-speech frame by binarizing the second speech confidence measure SM2 using the second predetermined threshold T2. The second predetermined threshold T2 can be equal to or different from the first predetermined threshold T1, for example, T1 and T2 are both equal to 0.5, or T1 is equal to 0.5 and T2 is less than or equal to 0.1 or 0.2. In some implementations, as described above, the second speech classifier 1b includes a speech separator and determines the second speech confidence measure SM2 based on the energy ratio. In such embodiments, it may be beneficial to set T2 lower than T1, for example, such that T1 is greater than 0.4 (e.g., equal to 0.5) and T2 is less than 0.3 (e.g., equal to 0.2 or 0.1).

[0083] Where a speech or non-speech label is assigned to each frame based on the second speech confidence measure SM2, this information can be exploited in different ways to enhance the final speech classification accuracy of the speech classification system 10.

[0084] In one implementation, the second number of speech frames NoF2 determined based on the second speech confidence measure SM2 is counted in the second context window 40b by the speech ratio calculator 22b of the second adaptive threshold applicator 2b. The second context window 40b may be the same type of context window (i.e., in terms of the number of look-ahead frames and / or look-back frames), or may be different. Figure 6In the illustrated example, the total context window length of second context window 40b is three. That is, when frame 45 is a given frame, second context window 40b includes review frame 44, given frame 45 itself, and a look-ahead frame 46. Similarly, in this illustrative example, it is assumed that any frames before frame 41 are non-speech F frames, and any frames after frame 46 are speech T frames. Optionally, speech ratio calculator 22b determines a second speech ratio r2 as the ratio of the number of speech frames NoF2 to the length W2 of second context window 40b.

[0085] Then, in step S9, the second adaptive threshold calculator 24b determines the second adaptive threshold T for each frame based on the number of speech frames NoF2 or the second speech ratio r2 in the second context window. a2 The second adaptive threshold level T a2 is provided to a second binary metric extractor 26b, which determines a second binary speech classification indicator SB2 by binarizing the second speech confidence metric SM2 in step S10. The second adaptive threshold calculator 24b can use a threshold function to determine the adaptive threshold. The threshold function implemented by the second adaptive threshold calculator 24b can be monotonically decreasing and, for example, similar to or equal to one of the threshold functions AD of equations 1-4 above.

[0086] Therefore, in this implementation, as above combined with Figure 3 As with the first adaptive threshold applicator 2a described in detail, the second adaptive threshold applicator 2b performs a corresponding function, except that the second adaptive threshold applicator 2b operates on the second speech confidence measure SM2 from the second classifier 1b.

[0087] The two speech classifiers 1a, 1b output two binary speech classification indicators SB1, SB2 through post-processing implemented by the first and second adaptive threshold applicators 2a, 2b, which may be different for one or more frames. For example, for a given frame, the first binary speech classification indicator SB1 is true (T, indicating speech), while the second binary speech classification indicator SB2 is false (F, indicating non-speech). In order to combine the two binary speech classification indicators SB1, SB2 to form an enhanced binary speech classification indicator SBE, the two binary speech classification indicators SB1, SB2 can be provided to the enhanced binary speech extractor 3, which determines the enhanced binary speech classification indicator SBE as an "either-or" combination of SB1 and SB2 in step S11, which means that SBE will be true if at least one of SB1 and SB2 is true, and false otherwise. Alternatively, the enhanced binary speech extractor 3 implements a strict AND combination of SB1 and SB2, which means that SBE is true only when both SB1 and SB2 are true, otherwise it is false. The enhanced binary speech classification indicator SBE may be the final output of the speech classification system.

[0088] In the implementation described above, the two speech classifiers 1a, 1b and the adaptive threshold applicators 2a, 2b operate independently of each other. However, for some implementations it has been recognized that the benefits of using two classifiers can be better exploited by allowing the adaptive threshold applicators 2a, 2b to exchange information.

[0089] In one implementation, the first number of speech frames NoF1 or the first speech ratio r1 determined by the first adaptive threshold applicator 2a is sent to the second adaptive threshold applicator 2b and used to control the second adaptive threshold applicator 2b. This may result in the second adaptive threshold applicator 2b not needing a threshold comparator 21b or a speech ratio calculator 22b, because the first number of speech frames NoF1 or the first speech ratio r1 is provided by the first speech applicator to replace the second number of speech frames NoF2 or the second speech ratio r2. The adaptive threshold calculator 24b can then use a threshold function to determine the adaptive threshold. For example, using the threshold function f defined as follows: E (r)

[0090] g E (r) = 0.5 + 6 (0.2 - r1) (Formula 5)

[0091] f E (r)=max(g E (r1),0.1)

[0092] But many other types of threshold functions can be used.

[0093] The opposite arrangement is also envisaged, ie the second adaptive threshold applicator determines the second number of frames NoF2 or the second speech ratio r2 and provides this information to the first adaptive threshold applicator 2a to control the first adaptive threshold calculator 2a.

[0094] In general, the above example works well for both music content and movie / sports content. The two classifiers 1a, 1b operate in parallel, and their respective binary speech classification metrics SB1, SB2 are combined by a final binary speech extractor 3. Optionally, NoF1 or r1 of the first classifier is provided to the second classifier, and vice versa.

[0095] For some complex content types, the coordination between the first classifier 1a and the second classifier 1b can reveal more useful information that can be used to determine more accurate adaptive thresholds. For example, the parameters of the adaptive threshold function implemented by each adaptive threshold applier 2a, 2b can be controlled based on the relationship between NoF1 / r1 and NoF2 / r2. Additionally or alternatively, the initially determined NoF1 / r1 and NoF2 / r2 (and / or the initially determined first and second adaptive thresholds) can be modified, for example, increased or decreased, based on the consistency between NoF1 / r1 and NoF2 / r2. The modified NoF1 / r1 and NoF2 / r2 (and / or the modified first and second adaptive thresholds) can then be used as a replacement for the initially determined NoF1 / r1 and NoF2 / r2 (and / or the first and second adaptive thresholds). The exact manner in which the adaptive threshold function is controlled depends on the type of classifier 1a, 1b used. For example, if one classifier is more accurate at detecting speech, then its speech confidence metric has a greater impact on the final binary speech classification than the speech confidence metric of another classifier when NoF1 / r1 or NoF2 / r2 is low.

[0096] In some implementations, there are four main cases to consider for different relationships between NoF1 / r1 and NoF2 / r2. These cases will now be illustrated for a setting where the first classifier 1a exhibits a high false negative rate (i.e., performs conservative classification and tends to miss some speech) and where the second classifier 1b exhibits a high false positive rate (i.e., performs exaggerated classification and tends to classify some non-speech as speech). Other classifiers with other characteristics may benefit from similar or different interactions between NoF1 / r1 and NoF2 / r2.

[0097] In the case of NoF1 / r1 being low and NoF2 / r2 being high, the first classifier 1a does not detect as much speech as the second classifier 1b. This often happens for music content using the second speech classifier 1b of this example. In this case, the adaptive threshold of the second adaptive threshold applier 2b is configured to increase as NoF1 / r1 decreases. Figure 7 As shown, this can be achieved by providing NoF1 / r1 to an optional speech ratio adjuster 23b, wherein the speech ratio adjuster 23b adjusts NoF2 / r2 so that NoF2 / r2 decreases, thereby resulting in an increase in the adaptive threshold in a subsequent adaptive threshold calculator 24b. Additionally or alternatively, NoF1 / r1 is provided to an optional adaptive threshold adjuster 25b, which increases the adaptive threshold output by the adaptive threshold calculator 24b.

[0098] This adjustment may have the effect that due to the second adaptive threshold T a2 Increase, resulting in fewer frames being marked as speech frames, thus avoiding false positives caused by the second classifier 1b. The terms "high" and "low" refer to NoF1 or r1 being above or below the first case threshold, and the same applies to NoF2 or r2. When NoF2 or r2 is above or below the second case threshold, No F2 or r2 is high or low.

[0099] In case two, NoF1 / r1 is high and NoF2 / r2 is high. This means that both classifiers 1a and 1b recognize a lot of speech, and the first classifier 1a may not have many false negatives. In this case, the second classifier 1b is only considered if the second classifier has a very high confidence score. Therefore, for higher NoF1 / r1, the second adaptive threshold is increased (using speech ratio adjuster 23b or adaptive threshold adjuster 24b) to avoid false positives introduced by the second classifier 1b.

[0100] In case three, NoF1 / r1 is high and NoF2 / r2 is low. This situation rarely occurs in the example setting. However, since the first speech classifier 1a has already recognized many speech sounds, it is likely to have a low false negative rate. In case three, we apply the same approach as in case two, increasing the second adaptive threshold for higher NoF1 / r1 to avoid false positives introduced by the second classifier 1b.

[0101] In case four, NoF1 / r1 is low and NoF2 / r2 is low. In this case, both classifiers 1a, 1b have a low, but non-zero, number of speech frames or speech ratio. This may be caused by audio content with sparse speech, rather than music content or content that stresses both classifiers at the same time. In this case, the adaptive threshold of the second adaptive threshold applicator 2b decreases as NoF1 / r1 decreases. This allows the second adaptive threshold applicator 2b to recognize more speech in conjunction with the first classifier 1a, and therefore incorporate more speech frames recognized by the second classifier 1b. However, some tests have found that case four may also occur for certain types of music, which means that in some implementations, case four is not used and only cases one to three are used.

[0102] The above four cases apply to the four types of audio content that meet the above requirements (i.e., NoF1 / r1 and NoF2 / r2 are above or below the corresponding case thresholds). In some embodiments, it is determined for each frame which of the four cases is valid, and the adaptive threshold function is modified accordingly, for example by an additive or multiplicative adjustment term or factor introduced by the speech ratio adjuster 23b or the adaptive threshold adjuster 24b. It is also contemplated that the parameters of the adaptive threshold function implemented in each adaptive threshold applicator 2a, 2b are dynamically tuned, or the function is changed depending on which of the four cases applies to the current frame.

[0103] Furthermore, while scenarios 1 through 4 above involve modifying the second NoF2 / r2 of the second adaptive threshold applicator 2b based on NoF1 / r1 of the first adaptive threshold applicator 2a, the opposite scenario is also possible, namely, modifying the first NoF1 / r1 of the first adaptive threshold applicator 2a based on NoF2 / r2 of the second adaptive threshold applicator 2b. For example, in scenario 1, where NoF1 / r1 is low and NoF2 / r2 is high, indicating, for example, music content with singing voice, NoF1 / r1 decreases based on the difference between NoF1 / r1 and NoF2 / r2. As the difference between NoF1 / r1 and NoF2 / r2 increases, NoF1 / r1 decreases. Typically, increasing the first adaptive threshold to reduce NoF1 / r1 means that this helps achieve a higher first adaptive threshold, further avoiding false positives. Alternatively, the first adaptive threshold can be increased directly based on the difference between NoF1 / r1 and NoF2 / r2, such that the first adaptive threshold increases as the difference between NoF1 / r1 and NoF2 / r2 increases. For example, in case one, only when NoF1 / r1 and NoF2 / r2 are sufficiently inconsistent (ie, have a sufficiently large difference), NoF1 / r1 is decreased and / or the first adaptive threshold is increased.

[0104] In Case 2, Classifiers 1a and 1b respectively detect high NoF1 / r1 and high NoF2 / r2, which may indicate movie content or some other speech-intensive content. In this case, based on the difference between the detected NoF1 / r1 and the detected NoF2 / r2, increase NoF1 / r1, and / or decrease the first adaptive threshold. As the difference between NoF1 / r1 and NoF2 / r2 decreases, increase NoF1 / r1 and / or decrease the first adaptive threshold. For example, in Case 2, increase NoF1 / r1, and / or decrease the first adaptive threshold only when NoF1 / r1 and NoF2 / r2 are sufficiently consistent (i.e., have a sufficiently small difference).

[0105] In Case 3, the first adaptive threshold can remain as it is, unaffected by NoF2 / r2. In Case 4, both Classifiers 1a and 1b have low speech frame numbers or speech ratios. This may be the result of audio content with sparse speech or challenging audio content that loads both Classifiers 1a and 1b (rather than music that would trigger the second classifier 1b to detect speech). In Case 4, lower the first adaptive threshold and / or increase NoF1 / r1 to incorporate more speech frames recognized by the first classifier 1a.

[0106] It is also possible to implement both types of corrections simultaneously, i.e., correcting NoF1 / r1 based on NoF2 / r2 and correcting NoF2 / r2 based on NoF1 / r1.

[0107] In some embodiments, the first adaptive threshold applicator uses the first speech ratio r1 and the second speech ratio r2 to form a corrected speech ratio r' that replaces the first speech ratio r1. The corrected speech ratio r' can be determined, for example, using the following formula:

[0108] In the case where r1 < r2 / 2 (Equation 6)

[0109] where u is a parameter that controls the aggressiveness of the exponential scaling factor. In a similar manner, the corrected frame number MNoF can also be determined, for example, by using Equation 6 and replacing r' with MNoF, r1 with NoF1, and r2 with NoF2*(W2 / W1), where W1 and W2 are window lengths in terms of frame numbers.

[0110] The parameter u determines the steepness of the exponential scaling factor, scaling the correction of r1 or NoF1 to form r' or MNoF. In Figure 8 for different values of the parameter u, the values of the exponential scaling factor are shown as a function of the ratio r1 / r2. The scaling factor u is also adjusted based on the ratios r1 / r2 and r1. In some embodiments, according to Figure 9The function shown sets the scaling factor u, assigning a higher u value when r2 exceeds 0.4 and r1 is less than 0.2. With this type of function, when r1 is less than r2, the u parameter becomes larger, which reduces the corrected speech ratio and thus leads to more conservative speech classification. In particular, when the second classifier 1b contains a speech separator, this configuration of u has been shown to reduce the false positive rate for music by correctly labeling signed speech as non-speech.

[0111] Since the number of speech frames NoF1, NoF2 or the speech ratio r1, r2 of any adaptive threshold applicator 2a, 2b can be sent to the other adaptive threshold applicator 2a, 2b, it is conceivable that only one of the first and second binary speech classification indicators SB1, SB2 is determined as the final binary classification of the speech classification system 10. To this end, in some implementations, the final binary speech extractor 3 can be omitted, wherein one of the adaptive threshold calculators 1a, 1b outputs the number of speech frames or the speech ratio, which is provided to the other adaptive threshold calculator 1a, 1b, which determines the adaptive threshold for the final binarization based on both NoF1 and NoF2 or both r1 and r2. For example, the first adaptive threshold applicator 2a determines the modified MNoF or r' value using Equation 6 above, or implements an extended threshold function f(NoF1, NoF2), f(r1, r2) that maps each combination of NoF1 and NoF2 or r1 and r2 to an adaptive threshold.

[0112] Unless otherwise stated, it will be apparent from the following discussion that it should be understood that throughout the public discussion, terms such as "process," "calculate," "calculate," "determine," "analyze," etc., are used to refer to the actions and / or processes of computer hardware or computing systems or similar electronic computing devices to manipulate data represented as physical quantities (e.g., electronic quantities) and / or convert them into other data similarly represented as physical quantities.

[0113] It should be understood that in the above description of exemplary embodiments of the present disclosure, various features are sometimes combined in a single embodiment, figure, or description thereof for the purpose of simplifying the disclosure and aiding in understanding one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed disclosure requires more features than are expressly recited in each claim. On the contrary, as reflected in the following claims, the inventive aspects lie in less than all the features of a single preceding disclosed embodiment. Therefore, the claims following the "Detailed Description" section are hereby expressly incorporated into this "Detailed Description" section, with each claim standing on its own as a separate embodiment of the present disclosure. Furthermore, as will be understood by those skilled in the art, although some embodiments described herein include some features included in other embodiments but not other features, combinations of features of different embodiments are also intended to be included within the scope of the present disclosure and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0114] In addition, some embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other means for performing the function. Therefore, a processor having instructions for performing such a method or method element forms a means for performing such a method or method element. It should be noted that when the method includes several elements, such as several steps, this does not imply the order of these elements unless otherwise specified. In addition, the elements of the device embodiments described herein are examples of means for performing the functions performed by the elements for the purpose of implementing the embodiments of the present disclosure. In the description provided herein, many specific details are set forth. However, it should be understood that the embodiments of the present disclosure can be implemented without these specific details. In other cases, well-known methods, structures, and techniques are not shown in detail to avoid obscuring the understanding of this specification. Therefore, although specific embodiments of the present disclosure have been described, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the present disclosure, and it is intended that all such changes and modifications fall within the scope of the present disclosure.

[0115] Various aspects of the present disclosure may be understood from the following Enumerated Example Embodiments (EEE):

[0116] EEE1. A method for performing speech classification on an audio signal, comprising:

[0117] obtaining an audio signal comprising a sequence of audio frames;

[0118] determining, for each audio frame, a first speech confidence metric using a first speech classifier, the first speech confidence metric indicating a likelihood that speech is present in the audio frame;

[0119] For each given audio frame in at least one subset of the sequence of audio frames:

[0120] classifying each corresponding audio frame of a first context window associated with the given audio frame as a speech frame or a non-speech frame by comparing a first speech confidence measure of the corresponding audio frame to a first predetermined threshold;

[0121] determining an adaptive threshold based on the number of speech frames of the first context window; and

[0122] A first binary speech classification indicator is determined for the given audio frame based on the first speech confidence metric and the adaptive threshold.

[0123] EEE2. The method according to EEE1, wherein the adaptive threshold is determined using a function that maps the number of speech frames in the first context window to an adaptive threshold.

[0124] EEE3. The method according to EEE2, wherein the function monotonically decreases with increasing number of speech frames in the first context window.

[0125] EEE4. A method according to any one of the preceding EEEs, wherein determining the first binary speech classification indicator comprises binarizing the first speech confidence metric using the adaptive threshold.

[0126] EEE5. The method according to any one of EEE1-EEE3, further comprising:

[0127] determining, for each audio frame, a second speech confidence measure using a second speech classifier different from the first speech classifier, the second speech confidence measure indicating a likelihood that speech is present in the frame;

[0128] For each given audio frame in a subset of the audio frame sequence:

[0129] - classifying each corresponding audio frame of a second context window associated with the given audio frame as a speech frame or a non-speech frame by comparing the second speech confidence measure of the corresponding audio frame with a second predetermined threshold;

[0130] - determining a second adaptive threshold based on the number of speech frames in the second context window and / or the number of speech frames in the first context window, and determining an enhanced binary speech classification indicator for the given audio frame based on the adaptive threshold and the second adaptive threshold, and / or

[0131] - wherein the adaptive threshold is based on the number of speech frames in the first context window and the number of speech frames in the second context window.

[0132] EEE6. The method according to EEE5, wherein the second speech classifier comprises a speech separator configured to generate an output audio signal having isolated speech content separated from the input audio signal, and the method further comprises:

[0133] processing the audio signal with the speech separator to obtain the output audio signal having isolated speech content; and

[0134] The second speech confidence measure is determined based on a spectral energy ratio of a spectral energy measure of the output audio signal and a spectral energy measure of the audio signal.

[0135] EEE7. The method according to EEE6, wherein the spectral energy ratio is determined as the ratio between:

[0136] an average spectral energy measure of the output audio signal across all frames of the audio signal, across all frames of the second context window, or for each frame individually, and

[0137] An average spectral energy measure of the audio signal across all frames of the audio signal, across all frames of the second context window, or for each frame individually.

[0138] EEE8. The method according to any one of EEE5-EEE7, wherein the adaptive threshold is based on the number of speech frames in the first context window and the number of speech frames in the second context window, and the method further comprises:

[0139] The first binary speech classification indicator is determined by binarizing the first speech confidence measure using the adaptive threshold.

[0140] EEE9. The method according to EEE8, wherein determining the adaptive threshold based on the number of speech frames in the first context window and the number of speech frames in the second context window comprises:

[0141] Obtaining an expansion function linking the number of speech frames in the first context window and the number of speech frames in the second context window to a variable threshold; and

[0142] The expansion function is evaluated by the number of speech frames in the first context window and the number of speech frames in the second context window.

[0143] EEE10. The method according to EEE9,

[0144] wherein, in the case where the number of speech frames in the first context window is higher than a first predetermined number and the number of speech frames in the second context window is lower than a second predetermined number, the expansion function is based only on the number of speech frames in the first context window, and

[0145] Wherein, when the number of speech frames in the first context window is lower than or equal to a first predetermined number and the number of speech frames in the second context window is higher than or equal to a second predetermined number, the expansion threshold function is based on both the number of speech frames in the first context window and the number of speech frames in the second context window.

[0146] EEE11. The method according to any one of EEE5-EEE10, further comprising:

[0147] For each given audio frame in the subset of the sequence of audio frames:

[0148] determining a second adaptive threshold based on the number of speech frames in the second context window and / or the number of speech frames in the first context window;

[0149] determining the first binary speech classification indicator by binarizing the first speech confidence measure for the given audio frame using a first variable threshold;

[0150] determining a second binary speech classification indicator by binarizing the second speech confidence measure for the given audio frame using a second variable threshold; and

[0151] The enhanced binary speech classification indicator is determined based on the first binary speech classification indicator and the second binary speech classification indicator.

[0152] EEE12. The method according to EEE11, wherein the enhanced binary speech classification indicator indicates that the speech is in an active state only when at least one of the first binary speech classification indicator and the second binary speech classification indicator indicates that the speech is in an active state. EEE13.

[0153] EEE13. The method according to any one of EEE1-EEE4, further comprising:

[0154] providing the first binary speech classification indicator and the audio signal to a gating unit; and

[0155] A gating gain is applied by the gating unit to the audio signal based on the first binary speech classification indicator to form a gated audio signal.

[0156] EEE14. The method according to any one of EEE5-EEE12, further comprising:

[0157] providing the enhanced binary speech classification indicator and the audio signal to a gating unit; and

[0158] A gating gain is applied by the gating unit to the audio signal based on the enhanced binary speech classification indicator to form a gated audio signal.

[0159] EEE15. The method according to EEE13 or EEE14, further comprising:

[0160] processing the audio signal with a speech separator to obtain a speech-separated audio signal having isolated speech content; and

[0161] The gated gain is applied to the speech separated audio signal by the gating unit.

[0162] EEE16. The method according to any one of the preceding EEEs, wherein the first context window associated with the given audio frame includes the given audio frame. EEE16.

[0163] EEE17. The method according to EEE16, wherein the first context window includes at least one look-ahead frame that is temporally subsequent to the given audio frame and at least one look-back frame that is temporally preceding the given audio frame. EEE17. The method according to EEE16, wherein the first context window includes at least one look-ahead frame that is temporally subsequent to the given audio frame and at least one look-backward frame that is temporally preceding the given audio frame.

[0164] EEE18. A method according to any one of the preceding EEEs, wherein a duration of the first context window or the second context window is at least 5 seconds, at least 10 seconds, at least 20 seconds or at least 30 seconds.

[0165] EEE19. A method according to any of the preceding EEEs, wherein the first predetermined threshold is between 0.1 and 0.6, for example about 0.5 or about 0.3.

[0166] EEE 20. A computer program product comprising instructions, which, when the program is executed by a computer, cause the computer to perform the method according to any one of EEE1-EEE19.

[0167] EEE21. A computer-readable storage medium storing the computer program according to EEE20.

[0168] EEE22. A system comprising one or more processors, wherein the one or more processors are configured to perform the method according to any one of EEE1-EEE 19.

Claims

1. A method for performing speech classification on an audio signal, comprising: obtaining an audio signal comprising a sequence of audio frames; determining, for each audio frame, a first speech confidence metric using a first speech classifier, the first speech confidence metric indicating a likelihood that speech is present in the audio frame; For each given audio frame in at least one subset of the sequence of audio frames: classifying each corresponding audio frame of a first context window associated with the given audio frame as a speech frame or a non-speech frame by comparing a first speech confidence measure of the corresponding audio frame to a first predetermined threshold; determining an adaptive threshold based on the number of speech frames in the first context window; as well as A first binary speech classification indicator is determined for the given audio frame based on the first speech confidence metric and the adaptive threshold. 2 . The method of claim 1 , wherein the adaptive threshold is determined using a function that maps the number of speech frames in the first context window to an adaptive threshold.

3. The method according to claim 2, wherein: The function decreases monotonically with increasing number of speech frames in the first context window.

4. The method according to any one of the preceding claims, wherein determining the first binary speech classification indicator comprises binarizing the first speech confidence measure using the adaptive threshold.

5. The method according to any one of claims 1 to 3, further comprising: determining, for each audio frame, a second speech confidence measure using a second speech classifier different from the first speech classifier, the second speech confidence measure indicating a likelihood that speech is present in the frame; For each given audio frame in a subset of the audio frame sequence: - classifying each corresponding audio frame of a second context window associated with the given audio frame as a speech frame or a non-speech frame by comparing the second speech confidence measure of the corresponding audio frame with a second predetermined threshold; - determining a second adaptive threshold based on the number of speech frames in the second context window and / or the number of speech frames in the first context window, and determining an enhanced binary speech classification indicator for the given audio frame based on the adaptive threshold and the second adaptive threshold, and / or - wherein the adaptive threshold is based on the number of speech frames in the first context window and the number of speech frames in the second context window.

6. The method of claim 5 , wherein the second speech classifier comprises a speech separator configured to generate an output audio signal having isolated speech content separated from the input audio signal, and the method further comprising: processing the audio signal with the speech separator to obtain the output audio signal having isolated speech content; as well as The second speech confidence measure is determined based on a spectral energy ratio of a spectral energy measure of the output audio signal and a spectral energy measure of the audio signal.

7. The method of claim 6, wherein the spectral energy ratio is determined as the ratio between: an average spectral energy measure of the output audio signal across all frames of the audio signal, across all frames of the second context window, or for each frame individually, and An average spectral energy measure of the audio signal across all frames of the audio signal, across all frames of the second context window, or for each frame individually.

8. The method according to any one of claims 5 to 7, wherein The adaptive threshold is based on the number of speech frames in the first context window and the number of speech frames in the second context window, the method further comprising: The first binary speech classification indicator is determined by binarizing the first speech confidence measure using the adaptive threshold.

9. The method according to claim 8, wherein Determining the adaptive threshold based on the number of speech frames in the first context window and the number of speech frames in the second context window includes: Obtaining an expansion function linking the number of speech frames in the first context window and the number of speech frames in the second context window to a variable threshold; and The expansion function is evaluated by the number of speech frames in the first context window and the number of speech frames in the second context window.

10. The method according to claim 9, in, In the case that the number of speech frames in the first context window is higher than a first predetermined number and the number of speech frames in the second context window is lower than a second predetermined number, the expansion function is based only on the number of speech frames in the first context window, and Wherein, when the number of speech frames in the first context window is lower than or equal to a first predetermined number and the number of speech frames in the second context window is higher than or equal to a second predetermined number, the expansion threshold function is based on both the number of speech frames in the first context window and the number of speech frames in the second context window.

11. The method according to any one of claims 5 to 10, further comprising: For each given audio frame in the subset of the sequence of audio frames: determining a second adaptive threshold based on the number of speech frames in the second context window and / or the number of speech frames in the first context window; determining the first binary speech classification indicator by binarizing the first speech confidence measure for the given audio frame using a first variable threshold; determining a second binary speech classification indicator by binarizing the second speech confidence measure for the given audio frame using a second variable threshold; as well as The enhanced binary speech classification indicator is determined based on the first binary speech classification indicator and the second binary speech classification indicator.

12. The method of claim 11, wherein the enhanced binary speech classification indicator indicates that speech is in an active state only when at least one of the first binary speech classification indicator and the second binary speech classification indicator indicates that speech is in an active state.

13. The method according to any one of claims 1 to 4, further comprising: providing the first binary speech classification indicator and the audio signal to a gating unit; as well as A gating gain is applied by the gating unit to the audio signal based on the first binary speech classification indicator to form a gated audio signal.

14. The method according to any one of claims 5 to 12, further comprising: providing the enhanced binary speech classification indicator and the audio signal to a gating unit; as well as A gating gain is applied by the gating unit to the audio signal based on the enhanced binary speech classification indicator to form a gated audio signal.

15. The method according to claim 13 or 14, further comprising: processing the audio signal with a speech separator to obtain a speech-separated audio signal having isolated speech content; as well as The gated gain is applied to the speech separated audio signal by the gating unit.

16. The method of any preceding claim, wherein the first context window associated with the given audio frame comprises the given audio frame. 17 . The method of claim 16 , wherein the first context window comprises at least one look-ahead frame temporally following the given audio frame and at least one look-back frame temporally preceding the given audio frame.

18. A method according to any one of the preceding claims, wherein The duration of the first context window or the second context window is at least 5 seconds, at least 10 seconds, at least 20 seconds or at least 30 seconds.

19. The method according to any of the preceding claims, wherein the first predetermined threshold value is between 0.1 and 0.6, such as about 0.5 or about 0.

3.

20. A computer program product comprising instructions which, when said program is executed by a computer, cause said computer to perform the method according to any one of claims 1 to 19.

21. A computer-readable storage medium storing the computer program according to claim 20.

22. A system comprising one or more processors configured to perform the method according to any one of claims 1-19.

Images