Detection and enhancement of speech in binaural recordings

Abstract

To provide a method, a system, and a non-transitory computer-readable medium for improving binaural recording of speech.SOLUTION: A method comprises the steps of: dividing received binaural signals into frames; applying a time-frequency transformation to each frame; calculating features of the frame based on the time-frequency representation; classifying each frame as self-speech or external speech by a classifier based on a subset of features; calculating dissimilarity based on the subset of features; segmenting the signal at peaks of a dissimilarity function; determining an overall class of self-speech or external speech for each segment by aggregating classifier data of the frame to which the segment belongs; and processing each segment in a speech enhancement chain based on the determined overall class.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Applications Nos. 63 / 162,289 and 63 / 245,548, filed March 17, 2021 and September 17, 2021, respectively; and Spanish Patent Application No. P202130013, filed January 12, 2021, each of which is incorporated by reference in its entirety.

[0002] [Technical field] The present disclosure relates to a method for speech enhancement in binaural recordings, a system for performing the method, and a non-transitory computer-readable medium storing instructions for performing the method. [Background technology]

[0003] Earbuds, or earphones, are wireless in-ear headphones that pair with smart devices such as phones and tablets, and they are becoming a popular choice for smartphone users to listen to audio and, with the addition of a built-in microphone, capture audio for real-time communication and recording voice messages. Earbuds are a convenient alternative for recording speech without a dedicated microphone, for people who conduct interviews, create vlog or podcast content, or simply want to record voice memos. Summary of the Invention

[0004] In this disclosure, the expression "self-speech" is used to refer to the speech of the person wearing the earphones, and the expression "external speech" is used to refer to speech from someone other than the person wearing the earphones.

[0005] Because the microphones are located in the ears of the person wearing the earphones, when recording self-speech, the propagation of sound from the mouth to the earphones, combined with the directivity of the mouth, causes significant alterations to the audio spectrum, i.e., a loss of high-frequency energy compared to what a conventional microphone positioned in front of the mouth would pick up. When recording external speech, the distance of each external speaker results in a loss of level compared to the volume of the self-speech. Both of these factors (loss of level and loss of high frequencies) lead to significant differences in volume and tonality between the self-speech and the external speech. Compensation for these effects benefits from distinguishing between self-speech and external speech, segmenting the recording, and processing each portion using optimal settings.

[0006] Speaker segmentation and dialing has been an active area of ​​research for many years, using well-established statistical methods such as the Bayes Information Criterion (BIC) and more recent AI-based techniques. While these techniques are effective in detecting changes in the speaker or acoustic conditions, they do not provide additional information, such as whether the speech is self or external. In particular, they operate on monophonic signals (single-channel recordings); therefore, they do not consider spatial aspects of sound, such as those embedded in binaural recordings. While spatial aspects such as the similarity between the signals at the left and right binaural microphones and the direction of arrival contain important information for the task of distinguishing self-speech from external speech, such cues are typically ignored for segmentation purposes.

[0007] Although automatic solutions for speech enhancement exist, they do not detect or use speaker segmentation information and therefore do not allow optimally coordinated processing of self-speech and external speech to achieve balanced timbre and volume.

[0008] This disclosure describes a method for improving binaural recordings of speech by identifying portions corresponding to self-speech and external speech, segmenting the recording accordingly, and then applying independent emphasis to each segment with optimal settings according to self-speech or external speech conditions.

[0009] Taking a binaural signal as input, a time-frequency transform is applied to split the signal into frequency bands. In parallel, the signal is sent to a voice activity detector or voice segment detector to identify which parts of the signal contain speech and avoid processing non-speech parts.

[0010] Spectral features are extracted from the time-frequency representation of the signal and used to classify self-speech and external speech on a frame-by-frame basis. In parallel, some of these features are sent to a dissimilarity segmentation unit, which uses statistical methods to identify speakers or find frames where changes in acoustic conditions occur. The segmentation unit receives information from the classification and dissimilarity segmentation units and combines them by majority voting to make a decision (self or external) for each segment. Segmentation is used to process the recording as multiple independent recordings, each with the appropriate settings derived from the classification into self and external speech. [Brief explanation of the drawings]

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings. [Figure 1] FIG. 1 provides a schematic block diagram of a speech enhancement system according to one embodiment of the present invention. [Figure 2] FIG. 2 is a flowchart of an exemplary process for segmenting a signal according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] [Time-frequency transformation and feature extraction] 1 and 2, a binaural signal s(t) is obtained, which has left and right signals l(t) and r(t). The binaural signal may be obtained in a variety of ways, including recording with earphones worn by a user. The binaural signal is then received by a device on which a speech enhancement system is operating. This device may be part of the device worn by the user, or it may be a separate device. In the latter case, the binaural signal is transmitted to this separate device.

[0013] The system of FIG. 1 includes a frame divider 1 connected to receive a binaural signal and divide it into frames. A time-frequency transform unit 2 is connected to receive the frames, followed by a feature extraction unit 3. A voice activity detector (VAD) 4 is connected in parallel to units 2 and 3 and is also connected to receive the frames of the binaural signal. The outputs of feature extraction unit 3 and VAD 4 are both connected to two blocks: a self-classification block 5 and a dissimilarity segmentation block 6. The outputs from blocks 5 and 6 are both fed to a segmentation unit 7. Furthermore, a speech enhancement chain 8 is connected to receive the frames of the binaural signal from frame divider 1 and the output from segmentation unit 7. The speech enhancement chain 8 outputs a modified binaural signal. The operation of the system and its various components will be described in more detail below, also with reference to the flowchart of FIG. 2.

[0014] In step 1, the binaural signal s(t) is divided into frames n. Then, in step S2, a time-frequency transform unit 2 receives the frames and generates signals L(i,f), R(i,f) with frame indices i=1:N and frequencies f=1:M. The time-frequency transform can be, for example, a discrete Fourier transform, a QMF filter bank, or another transform.

[0015] In step S3, the frame division signals L(i,f) and R(i,f) are grouped for each frequency band, and the following feature quantities are calculated for each frequency band b by the feature extraction unit 3: - Energy per band E(i,b)=Σ f∈b (L 2 (i,f)+R 2 (i,f)); - Inter-channel coherence IC(i,b); - Mel-frequency cepstral coefficients MFCC(i,b)

[0016] Since the present analysis focuses on speech, typically only the frequency range of speech is retained, for example the band between 80 Hz and 4 kHz.

[0017] Additionally, the spectral slope SS(i) is calculated as the slope of a linear approximation of E(i,b) in the frequency range of interest.

[0018] Spectral slope is a measure of how much high frequencies are attenuated, and therefore it is well suited to the task of distinguishing between self and external speech.

[0019] Inter-channel coherence is a measure of the similarity between L and R; given the symmetry of propagation from the mouth to the L and R microphones, L and R can be expected to be nearly identical for self-speech, while dissimilarity is expected for external speech in typical situations.

[0020] MFCCs are commonly used features for speech-related analysis and classification.

[0021] In parallel to step S3 (not shown in Figure 2), the frames of s(t) are sent to the VAD4, which outputs the probability V(i) of containing speech for each frame of audio i, where 0 ≤ V(i) ≤ 1. If the VAD operates on a mono signal, a downmix such as l(t) + r(t) is used instead of s(t).

[0022] [Self and External Speech Classification] In step S4, the self-external classification unit 5 receives the features E(i,b), SS(i,b), and IC(i,b) from the feature extraction unit 3 and generates a binary classification result C(i), i.e., C(i)=1 for self-speech and C(i)=0 for external speech. The classification is performed by a trained classifier, such as a support vector machine (SVM). Training of the classifier can be performed on a set of labeled content, where the input is the aforementioned feature vector and the output class is given in advance for each frame of audio. SVM is chosen because it is a powerful nonlinear classifier that requires less training data than deep neural networks.

[0023] For improved performance, only frames containing audio are passed to the SVM during both training and classification. In the illustrated example, the classification unit 5 also receives the speech probability V from the VAD 4. This allows the classifier 5 to pass only frames with a probability V above a given threshold to the SVM.

[0024] The accuracy of the classifier may vary depending on the presence of noise, different speaker types, etc. Being a frame-by-frame decision, a method may be provided to segment the signal based on this classification.

[0025] Alternatively or additionally, the self-external classification unit 5 receives bone conduction vibration sensor data from a bone conduction sensor (not shown) and generates a binary classification result C(i) based at least in part on the bone conduction vibration sensor data. For example, classification based on the bone conduction vibration sensor data may be performed by determining whether the bone conduction vibration sensor data exceeds a predetermined threshold, which indicates that the audio is self-speech, while bone conduction vibration sensor data that does not exceed the predetermined threshold may indicate external speech. The bone conduction vibration sensor data can be used as a substitute for or a supplement to the features output from the feature extraction unit 3 and the speech probability V output from the VAD 4.

[0026] [Dissimilarity Segmentation] This dissimilarity segmentation unit 6 also receives the MFCC(i,b) features and the VAD information V(i), and in step S5, defines a threshold th for voice detection such that all frames where V(i) < th are discarded. The rows k of the discarded frames are removed from the matrix MFCC(i,b), and the Bayesian Information Criterion (BIC) method is applied to the remaining frames j, and a dissimilarity function D(j) is obtained according to the conventional notation. A BIC window length corresponding to the minimum length (e.g., 2s) to be segmented can be used. Then, the transitions of the speech signal are: i) The peak must be higher than a pre-defined threshold th D and ii) The peak should be separated by a minimum number of frames Δj, usually corresponding to the BIC window length and is obtained by finding the peaks of D(j) under these conditions.

[0027] After finding the peaks in the speech-only frames, their positions are mapped back to the full set of frames, and as a result, the transitions are based on the time of the original signal.

[0028] Note that dissimilarity segmentation detects not only transitions between the speaker and external speakers but also any other changes in the speaker or acoustic conditions; even for transitions between self-speech and external speech, it does not provide information about which is which.

[0029] [Segmentation] Segmentation unit 7 receives the self and external speech classification for each frame C(i) from classification unit 5 and the set of frames j for which speech transitions have been identified by dissimilarity segmentation unit 6. In step S6, unit 7 divides the binaural audio into segments based on the transition frames j. Then, in step S7, unit 7 provides a final segmentation of the audio into self and external speech segments of sufficient length and classification confidence.

[0030] For each segment k provided by the dissimilarity segmentation unit 6, multiple frames belonging to the segment are considered to determine whether the segment is considered to be self-speech.

[0031] For example, if a "majority vote" is applied to the frame-by-frame classification, and the number of frames classified as self-speech CS(k) is greater than the number of frames classified as external speech CE(k), then segment k can be considered self-speech, and vice versa. The confidence σ(k) for segment k is determined based on the relative difference between the number of frames CS(k) in segment k that are classified as self-speech and the number of frames CE(k) in segment k that are classified as external speech: σ(k)=|CS(k)-CE(k)| / N(k) where N(k) is the total number k of frames in segment k, including non-speech frames, i.e., N(k)=CS(k)+CE(k).

[0032] Threshold th σ is σ <th σ segments are defined to be considered uncertain.

[0033] The segmentation unit 7 may further merge adjacent segments in certain situations. For example, adjacent segments classified into the same class (self or external) and deemed certain by a confidence criterion may be merged into a single segment. Similarly, adjacent uncertain segments may be merged to form a single uncertain segment. Segments shorter than a predetermined duration may be merged with larger adjacent frames. An uncertain segment surrounded by two certain segments of the same class may be merged with the adjacent segments into one segment. An uncertain segment surrounded by two certain segments of different classes (i.e., one self-speech and one external speech) may be merged into the longest adjacent segment.

[0034] Furthermore, an uncertain segment surrounded by two certain segments of different classes (i.e., one self-speech and one external speech) can be used as a transition region in a subsequent speech enhancement chain. For example, a short uncertain segment can be used as a cross-fade region for transitioning between different treatments applied to adjacent segments.

[0035] The final segmentation obtained by unit 7 is passed to the speech enhancement unit in a format that includes the transition points and inferred class (self-speech or external speech) of each segment. Alternative representations such as segment start points and durations are also possible.

[0036] [Segmentation-based speech enhancement] The speech enhancement chain 8 may comprise signal processing blocks that perform sibilance reduction, equalization, dynamic range compression, noise reduction, de-reverberation, and other processing. Often, the optimal amount and settings of each processing block may vary depending on the characteristics of the signal: self and outside speech typically benefit from different equalization, independent leveling, different amounts of reverberation suppression, etc.

[0037] Therefore, the segmentation into self and external speech provided by the segmentation unit 7 can be used to process the two classes of speech separately to achieve optimal sound quality.

[0038] Examples of segmentation-based processing include: - Equalization to compensate for high frequency losses in recordings of self-speech; the correction curves (gain per frequency band) can be measured, estimated or obtained by simulation and then applied to the self-speech segments only. - Leveling: Matching the level and dynamic range of self and external speech can be difficult when the content is considered as a whole unit. Segmentation allows each segment to be leveled independently, thus ensuring the required volume and dynamic range for each speaker. Ambience suppression: Ambience typically increases immersion but reduces intelligibility. Ambience and reverberation suppression can be applied heavily to external speech to increase intelligibility, and lightly to self-speech to preserve immersion. - Binaural signal rotation to stabilize the perceived image by compensating for the effects of head movements during recording: self-speech does not require stabilization (which is actually perceived as unwanted rotation), but external speech benefits from stabilization. - Channel imbalance correction: Earphones can have inter-channel imbalance in the high frequency range, depending on how firmly the buds are positioned in each ear canal. This causes the unvoiced parts of self-speech (sibilance) to be positioned slightly off-center toward the sound source, resulting in a less solid sound than a mono recording. Compensating for the level difference between the left and right channels in the affected high frequency range can improve self-speech quality, but applying the same processing to external speech may affect its spatial cues.

[0039] Once segmentation data is available, the entire signal is divided into segments, and each segment is processed according to its inferred class. Segments may contain extra frames at their boundaries for crossfading due to overlap when recombining processed segments. The settings used to process each frame are either based on different presets for the self and outside speech classes (e.g., for processes where different processing is required for self and outside speech, such as ambience suppression), or based on the same settings (e.g., for instances where the goal is to obtain uniform results, such as in the case of leveling).

[0040] In some implementations, Classification of self and external speech can be achieved by a bone conduction vibration sensor. In such implementations, the classifier can perform classification based on bone conduction vibration sensor data in addition to or instead of using features. For example, the classifier can classify audio as self-speech in response to detecting bone vibrations corresponding to speech, or as external speech in response to detecting the absence of bone vibrations based on data from the bone conduction vibration sensor. Thus, bone conduction vibration sensor data can complement or replace features. The frame sizes used for MFCC, VAD, and other features may be different; in such cases, when combining different features, or different metrics derived from features, the coarser feature may be "upsampled" to the resolution of the finest feature by interpolation or simple nearest-neighbor repetition.

[0041] Aspects of the systems described herein can be implemented in a suitable computer-based sound processing network environment for processing digital or digitized audio files. Portions of an adaptive audio system can include one or more networks comprising any desired number of individual machines, including one or more routers (not shown) that serve to buffer and route data transmitted between computers. Such networks may be built on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0042] One or more of the components, blocks, processes, or other functional components may be implemented through a computer program that controls the execution of a processor-based computing device of the system. It should also be noted that the various functions disclosed herein may be described using any number of combinations of hardware, firmware, and / or in terms of their behavior, register transfers, logic components, and / or other characteristics as data and / or instructions embodied in various machine-readable or computer-readable media. Computer-readable media on which such formatted data and / or instructions may be embodied include various forms of physical (non-transitory) non-volatile storage media, such as optical, magnetic, or semiconductor storage media.

[0043] While one or more embodiments have been described by way of example and in terms of specific embodiments, it is to be understood that the one or more embodiments are not limited to the disclosed embodiments. To the contrary, the intention is to cover various modifications and similar arrangements as would be apparent to those skilled in the art. Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.

[0044] [Appendix 1] 10. A method comprising: - dividing the binaural audio signal into frames; - applying a time-frequency transform to each frame; - calculating features of said frames based on a time-frequency representation; - classifying, by a classifier, each frame as self-speech or foreign speech based at least in part on the subset of features; - calculating a dissimilarity function based on a subset of features; - segmenting the binaural audio signal at a peak of the dissimilarity function; - for each segment, determining an overall class of own speech or external speech, respectively, by aggregating classifier data of the frames belonging to said segment; - processing each segment through a speech enhancement chain, the configuration of said speech enhancement chain being based on the overall class determined for that segment; method. [Appendix 2] calculating a respective speech probability for each frame using voice activity detection (VAD), wherein only frames for which the speech probability is greater than a predetermined value are considered for classification and segmentation; The method described in Appendix 1. [Appendix 3] the features include at least one of energy per frequency band, spectral slope in a predetermined frequency range, inter-channel coherence per frequency band, or Mel-frequency cepstral coefficients; The method described in Appendix 1. [Appendix 4] the classifier is a support vector machine; 4. The method of any one of appendices 1 to 3. [Appendix 5] The dissimilarity function is obtained by applying the Bayes Information Criterion (BIC) to the subset of features. 4. The method of any one of appendices 1 to 3. [Appendix 6] retaining a peak of the dissimilarity function if the value of the dissimilarity function is greater than a predetermined value and the distance to the nearest peak is greater than another predetermined value; The method described in Appendix 5. [Appendix 7] The steps to determine each overall class are: Calculating the number of frames classified as external speech (CE) in the segment; Calculating the number of frames classified as self-speech (CS) in the segment; assigning class self-speech if CS≧CE, and assigning class outside-speech if CE>CS; The method described in Appendix 1. [Appendix 8] assigning a respective classification confidence value to each segment using the formula abs(CE-CS) / N, where N is the total number of frames in the segment; The method described in Appendix 7. [Appendix 9] designating as uncertain any segment having a confidence value below a predetermined value; The method described in Appendix 8. [Appendix 10] merging adjacent segments of the same class into a single segment of said same class; merging a segment designated as uncertain and surrounded by two segments of the same class (self or foreign) with the surrounding segments; The method described in Appendix 9. [Appendix 11] The steps for processing each segment in the speech enhancement chain are: noise estimation and noise reduction; Equalization including specific filters for self-speech and external speech; Leveling, including specific target levels and dynamic ranges for self-speech and external speech; Ambience balance, including different amounts of boost or attenuation for self-speech and external speech; spatial rotation, including different amounts of rotation for self-speech and external speech; and channel imbalance correction, including different amounts of correction for self-speech and external speech; 11. The method of any one of appendices 1 to 10. [Appendix 12] - One segment is - Designated as uncertain; - surrounded by two segments of different classes (one self, one external); and - shorter than the specified length; It is processed with both settings and used as a crossfade area. - One segment is: - Designated as uncertain; - surrounded by two segments of different classes (one self, one external); and - longer than the specified length, processed in a neutral setting or merged with the longest adjacent segment, The method described in Appendix 11. [Appendix 13] recombining the processed segments into sequences according to their order in the original input; and reducing the audible discontinuity by applying a cross-fade at the transition point. The method described in Appendix 11. [Appendix 14] 1. A system comprising: one or more processors; a non-transitory computer-readable storage medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 13; system. [Appendix 15] A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of clauses 1 to 13. A non-transitory computer-readable storage medium.

Claims

1. 1. A method comprising: receiving a binaural audio signal; - dividing the binaural audio signal into frames; - applying a time-frequency transformation to each frame; - calculating features of said frames based on their time-frequency representation; classifying, by a classifier, each frame as own speech or external speech based at least in part on the first subset of features, where own speech is speech of a user of equipment associated with the recording of the binaural audio signal and external speech is speech other than that of a user of equipment associated with the recording of the binaural audio signal; - calculating a dissimilarity function based on said second subset of features; - segmenting the binaural audio signal at the peak of the dissimilarity function; - for each segment, determining the overall class of own speech or external speech, respectively, by aggregating the classifier data of the frames belonging to said segment; - processing each segment through a speech enhancement chain, the configuration of said speech enhancement chain being based on the overall class determined for that segment; method.

2. calculating a respective speech probability for each frame using voice activity detection (VAD), wherein only frames for which the speech probability is greater than a predetermined value are considered for classification and segmentation; The method of claim 1.

3. the features include at least one of energy per frequency band, spectral slope in a predetermined frequency range, inter-channel coherence per frequency band, or Mel-frequency cepstral coefficients; The method of claim 1.

4. the classifier is a support vector machine; 4. The method according to any one of claims 1 to 3.

5. the dissimilarity function is obtained by applying a Bayesian Information Criterion (BIC) to the second subset of features; 4. The method according to any one of claims 1 to 3.

6. the first subset of features includes a first set of Mel-frequency cepstral coefficients; the second subset of features comprises a second set of Mel-frequency cepstral coefficients that is a subset of the first set of Mel-frequency cepstral coefficients.

6. The method according to any one of claims 1 to 5.

7. the second set of Mel-frequency cepstral coefficients corresponds to frames that do not contain detected own speech and detected external speech; The method of claim 6.

8. retaining a peak of the dissimilarity function if the value of the dissimilarity function is greater than a predetermined value and the distance to the nearest peak is greater than another predetermined value; The method of claim 7.

9. The steps to determine each overall class are: Calculating the number of frames classified as external speech (CE) within the segment; Calculating the number of frames classified as self-speech (CS) in the segment; assigning the class self-speech if CS≧CE, and assigning the class outside-speech if CE>CS; The method of claim 1.

10. assigning a respective classification confidence value to each segment using the formula abs(CE-CS) / N, where N is the total number of frames in the segment; 10. The method of claim 9.

11. designating as uncertain any segment having a confidence value below a predetermined value; The method of claim 10.

12. merging adjacent segments of the same class into a single segment of said same class; merging a segment designated as uncertain and surrounded by two segments of the same class (self or foreign) with the surrounding segments; The method of claim 11.

13. The steps for processing each segment in the speech enhancement chain are: noise estimation and noise reduction; Equalization including specific filters for self-speech and external speech; Leveling, including specific target levels and dynamic ranges for self-speech and external speech; Ambience balance, including different amounts of boost or attenuation for self-speech and external speech; Spatial rotation, including different amounts of rotation for self-speech and external speech; and channel imbalance correction, including different amounts of correction for self-speech and external speech; 13. The method of any one of claims 1 to 12.

14. - One segment is: - designated as uncertain; - surrounded by two segments of different classes (one self, one external); and - shorter than a predetermined length; It is processed with both settings and used as a crossfade area. - One segment is: - designated as uncertain; - surrounded by two segments of different classes (one self, one external); and - longer than a given length, processed in a neutral setting or merged with the longest adjacent segment, 14. The method of claim 13.

15. recombining the processed segments into sequences according to their order in the original input; and reducing the audible discontinuity by applying a cross-fade at the transition point.

14. The method of claim 13.

16. 1. A system comprising: one or more processors; a non-transitory computer-readable storage medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 15. system.

17. 16. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 15. A non-transitory computer-readable storage medium.

Images