Detection and enhancement of speech in binaural recordings

Abstract

This paper discloses methods, systems, and computer program products for segmenting binaural recordings of speech into portions containing the speaker's own speech and portions containing external speech, and processing each type using different settings to obtain an enhanced overall presentation. Segmentation is based on a combination of: i) feature-based frame-by-frame classification, and ii) dissimilarity detection using statistical methods. The segmentation information is then used in a speech enhancement chain, where independent settings are used to process the speaker's own speech portion and the external speech portion.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Applications No. 63 / 162,289 and No. 63 / 245,548, filed March 17, 2021 and September 17, 2021, respectively, and Spanish Patent Application No. P202130013, filed January 12, 2021, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to a method for enhancing speech in binaural recordings, a system for performing the method, and a non-transitory computer-readable medium storing instructions for performing the method. Background Technology

[0004] Earbuds are wireless in-ear headphones that pair with smart devices such as phones and tablets; they are becoming a popular choice for smartphone users to listen to audio, and thanks to the addition of a built-in microphone, they can also capture the user's voice for real-time communication or recording voice messages. For those who want to conduct interviews, create video logs or podcast content, or simply record voice notes, earbuds are a convenient alternative to recording voice without a dedicated microphone. Summary of the Invention

[0005] In this disclosure, the term "self-voice" is used to refer to the voice of the person wearing the earpiece, while the term "external voice" is used to refer to the voice of someone other than the person wearing the earpiece.

[0006] Because the microphone is located inside the earpiece, when recording one's own speech, the sound travels from the mouth to the earpiece. Combined with the directionality of the mouth, this significantly alters the speech spectrum, resulting in increased high-frequency energy loss compared to a traditional microphone placed in front of the mouth. When recording external speech, the distance of each external speaker compared to the loudness of one's own speech causes a level loss. Both of these factors (level loss and high-frequency loss) contribute to significant differences in loudness and pitch between one's own and external speech. Compensation for these effects relies on recognizing both one's own and external speech, segmenting the recording, and processing each segment with optimal settings.

[0007] Speaker segmentation and clustering have been active research areas for years, with well-established statistical methods such as the Bayesian Information Criterion (BIC) and more recently, AI-based techniques. While these techniques are effective at detecting changes in speaker or acoustic conditions, they fail to provide additional information, such as whether the speech is self-speech or external speech. In particular, they process monophonic signals (monophonic recordings), and therefore do not consider spatial aspects of sound embedded in binaural recordings. Spatial aspects such as direction of arrival and the similarity between the left and right binaural microphone signals have proven to contain important information for distinguishing self-speech from external speech tasks, but such cues are often ignored for segmentation purposes.

[0008] While automatic solutions for speech enhancement exist, they do not detect or use speaker segmentation information, thus they do not allow for optimization or customized processing of their own speech and external speech to achieve a balanced tone and loudness.

[0009] This disclosure describes a method for improving binaural recordings of speech by identifying portions corresponding to one's own speech and external speech, segmenting the recording accordingly, applying independent enhancements to each segment, and optimizing settings based on either one's own speech conditions or external speech conditions.

[0010] Using binaural signals as input, a time-frequency transformation is applied to divide the signal into frequency bands. Simultaneously, the signal is sent to a speech activity detector to identify which parts of the signal contain speech, thus avoiding processing non-speech components.

[0011] Spectral features are extracted from the time-frequency representation of the signal and used for frame-by-frame classification into self-speech and external speech. Simultaneously, some of these features are sent to a dissimilarity segmentation unit, which uses statistical methods to identify frames where speaker identity or acoustic conditions have changed. The segmentation unit receives information from the classification and dissimilarity segmentation units and combines it into a decision for each segment (self-speech or external speech) through majority voting. Segmentation is used to process the recording into multiple independent recordings, each with appropriate settings derived from the self-speech and external speech classification. Attached Figure Description

[0012] Embodiments of the present invention will be described in more detail with reference to the accompanying drawings.

[0013] Figure 1 A schematic block diagram of a speech enhancement system according to an embodiment of the present invention is provided.

[0014] Figure 2 This is a flowchart illustrating an example process of segmenting a signal according to an embodiment of the present invention. Detailed Implementation

[0015] Time-frequency transformation and feature extraction

[0016] exist Figure 1-2 In the process, a binaural signal s(t) consisting of a left signal l(t) and a right signal r(t) has been obtained. The binaural signal can be acquired in various ways, including recording via earbuds worn by the user. The binaural signal is then received by a device running a speech enhancement system. This device can be part of the user-worn device or a separate device. In the latter case, the binaural signal has already been transmitted to that separate device.

[0017] Figure 1 The system includes a frame divider 1 connected to receive binaural signals and divide them into frames. A time-frequency conversion unit 2 is connected to receive frames, followed by a feature extraction unit 3. A voice activity detector (VAD) 4 is connected in parallel to units 2 and 3 and is also connected to receive binaural signal frames. The outputs of both feature extraction unit 3 and VAD 4 are connected to two blocks: a self-classification block 5 and a dissimilarity segmentation block 6. The outputs of blocks 5 and 6 are provided to a segmentation unit 7. Furthermore, a speech enhancement chain 8 is connected to receive binaural signal frames from the frame divider 1 and the output from the segmentation unit 7. The speech enhancement chain 8 outputs a modified binaural signal. More details will follow, with reference to... Figure 2 The flowchart describes the operation of the system and its various components.

[0018] In step S1, the binaural signal s(t) is divided into frames i by frame divider 1. Then, in step S2, time-frequency conversion unit 2 receives the frames and generates signals L(i,f) and R(i,f), where the frame indices i = 1:N and the frequencies f = 1:M. The time-frequency conversion can be, for example, a discrete Fourier transform, a QMF filter bank, or other transforms.

[0019] In step S3, the framed signals L(i,f) and R(i,f) are grouped into frequency bands, and in each frequency band b, the feature extraction unit 3 calculates the following features:

[0020] - The energy of each frequency band is E(i,b)=∑ f∈b (L 2 (i,f)+R 2 (i,f));

[0021] - Interchannel coherence IC(i,b);

[0022] -Mel frequency cepstral coefficients MFCC(i,b)

[0023] Since current analysis focuses on speech, it typically only retains frequency bands within the speech frequency range, such as between 80 Hz and 4 kHz.

[0024] In addition, the spectral slope SS(i) is calculated as the slope of the linear fit of E(i,b) within the frequency range of interest.

[0025] Spectral slope is a measure of the degree of high-frequency attenuation, making it suitable for tasks that distinguish one's own speech from external speech.

[0026] Inter-channel coherence is a measure of the similarity between L and R; given the propagation symmetry from the mouth to the L and R microphones, it can be expected that L and R are almost identical for their own speech, while under typical conditions L and R are different for external speech.

[0027] MFCC is a feature commonly used in speech correlation analysis and classification.

[0028] Parallel to step S3 ( Figure 2 (not shown in the image), the frames of s(t) are sent to VAD 4, which outputs the probability V(i) containing speech for each frame of audio i, where 0 ≤ V(i) ≤ 1. If the VAD is operating on a mono signal, a downmixing such as l(t) + r(t) is used instead of s(t).

[0029] Classification of self-speech and external speech

[0030] In step S4, the self-external classification unit 5 receives features E(i,b), SS(i,b), and IC(i,b) from the feature extraction unit 3 and produces a binary classification result C(i), where C(i) = 1 for self-speech and C(i) = 0 for external-speech. Classification is performed by a trained classifier (such as a Support Vector Machine (SVM)). For each frame of the audio, the classifier can be trained using a set of labeled content, where the input is the aforementioned feature vector and the output class is the given prior. SVM is chosen because it is a powerful non-linear classifier that requires less training data compared to deep neural networks.

[0031] To improve performance, only frames containing audio are passed to the SVM during both training and classification. In the example shown, classification unit 5 also receives the speech probability V from VAD 4. This allows classification unit 5 to pass only frames with probability V exceeding a given threshold to the SVM.

[0032] The accuracy of the classifier can vary depending on the presence of noise, different speaker types, etc. As a frame-by-frame decision, a method for segmenting the signal based on this classification can be provided.

[0033] Alternatively or additionally, the self-vs-external classifier 5 receives bone-conducted vibration sensor data from a bone-conducted sensor (not shown) and generates a binary classification result C(i) based at least in part on the bone-conducted vibration sensor data. For example, classification based on the bone-conducted vibration sensor data can be performed by determining whether the bone-conducted vibration sensor data exceeds a predetermined threshold. The bone-conducted vibration sensor data exceeding the predetermined threshold can indicate that the audio is self-speech, while the bone-conducted vibration sensor data not exceeding the predetermined threshold can indicate that the audio is external speech. The bone-conducted vibration sensor data can be used as an alternative or supplement to the features output from the feature extraction unit 3 and the speech probability V received from the VAD 4.

[0034] Dissimilarity segmentation

[0035] The dissimilarity segmentation unit 6 also receives the MFCC(i,b) features and the VAD information V(i), and defines a threshold th for voice detection in step S5, thereby discarding all frames where V(i) < th. The k-th row of the discarded frames is removed from the matrix MFCC(i,b), and the Bayesian Information Criterion (BIC) method is applied to the remaining frames j to obtain a dissimilarity function D(j) according to the conventional notation. The BIC window length corresponding to the minimum length of interest for segmentation (e.g., 2s) can be used. Then, under the following conditions, transitions in the speech signal are obtained by finding the peaks in D(j): i) the peak should be higher than a predefined threshold th D , and ii) the peaks should be separated by a minimum number of frames Δj typically corresponding to the BIC window length.

[0036] After finding the peaks in only the speech frames, their positions are mapped back to the complete set of frames such that the transitions refer to the time of the original signal.

[0037] Note that dissimilarity segmentation not only detects transitions between the self-speaker and the external speaker but also any other changes in the speaker or acoustic conditions; even for transitions between self-speech and external speech, it does not provide information on which is the self-speech and which is the external speech.

[0038] Segmentation

[0039] The segmentation unit 7 receives the self-speech and external-speech classification C(i) for each frame from the classification unit 5, and the set of frames j where the dissimilarity segmentation unit 6 identifies speech transitions. In step S6, the unit 7 segments the binaural into segments based on the transition frames j. Then, in step S7, the unit 7 finally segments the audio into self-speech segments and external-speech segments with sufficient length and classification confidence.

[0040] For each segment k provided by the dissimilarity segmentation unit 6, multiple frames belonging to the segment are considered to determine whether the segment is its own speech.

[0041] For example, "majority voting" can be applied to the classification of each frame, such that if the number of frames classified as self-speech (CS(k)) is greater than the number of frames classified as external speech (CE(k)), then segment k is considered self-speech, and vice versa. The confidence level σ(k) of segment k is determined based on the relative difference between the number of frames in segment k classified as self-speech (CS(k)) and the number of frames in segment k classified as external speech (CE(k)).

[0042]

[0043] Where N(k) is the total number of frames in segment k, including non-speech frames, i.e., N(k) = CS(k) + CE(k).

[0044] Define threshold th σ Make σ <th σ The fragments are considered indeterminate.

[0045] Segmentation unit 7 can also merge adjacent segments under certain circumstances. For example, adjacent segments classified into the same category (self or external) and considered definite according to confidence criteria can be merged into a single segment. Similarly, adjacent uncertain segments can be merged to form a single uncertain segment. Segments shorter than a predefined duration can be merged with larger adjacent frames. Uncertain segments surrounded by two definite segments of the same category can be merged with adjacent segments into a single segment. Uncertain segments surrounded by two definite segments of different categories (i.e., one self-speech and one external speech) can be merged into the longest adjacent segment.

[0046] Furthermore, uncertain segments surrounded by two deterministic segments of different categories (i.e., one self-speech and one external speech) can be used as transition regions in the following speech enhancement chains. For example, short uncertain segments can be used as cross-fade regions for transitions between different processes applied to adjacent segments.

[0047] The final segments obtained by Unit 7 are passed to the speech enhancement unit in a format that includes the transition point of each segment and the inferred category (self-speech or external speech). Alternative representations such as the start point and duration of segments are also possible.

[0048] Segmented speech enhancement

[0049] The speech enhancement chain 8 may include signal processing blocks that perform sibilance reduction, equalization, dynamic range compression, noise reduction, dereverberation, and other processing. Typically, the optimal amounts and settings for each processing block can vary depending on the characteristics of the signal: usually, the speaker and the external speaker will benefit from different equalization, independent leveling, different amounts of reverberation suppression, etc.

[0050] Therefore, the segmentation of its own speech and external speech provided by segmentation unit 7 can be used to process the two types of speech differently and achieve optimal audio quality.

[0051] Examples of segmentation-based processing include:

[0052] - Equalization to compensate for high-frequency losses in one's own speech recordings; the correction curve (gain for each frequency band) can be measured, estimated, or obtained through simulation and then applied only to one's own speech segments.

[0053] - Leveling: Aligning the level and dynamic range of one's own speech with that of external speech can be challenging if the content is considered as a whole. By segmenting, each segment can be leveled independently, thus ensuring the loudness and dynamic range required by each speaker.

[0054] - Ambient suppression: Ambient usually enhances immersion but reduces intelligibility. Ambient and reverberation suppression can be applied heavily to external speech to improve intelligibility and lightly to one's own speech to maintain immersion.

[0055] - Binaural signal rotation is used to stabilize the perceived image by compensating for the effects of head movements during recording: self-speech does not need stabilization (which would actually be perceived as an unwanted rotation), while external speech benefits from it.

[0056] - Channel Imbalance Correction: Earplugs may have inter-channel imbalance in the high-frequency range, depending on how firmly the earplug is placed in each ear canal. This can cause unvoiced sibilants (dental sounds) in one's own speech to be slightly off-center from the source direction and sound less clear than a mono recording. Compensating for the level difference between the left and right channels in the affected high-frequency band can improve the quality of one's own speech, while applying the same processing to external speech may affect its spatial cues.

[0057] Once segmented data is available, the entire signal is divided into multiple segments, each processed according to the inferred category. These segments may include additional frames at the boundaries, used for alternating fading through overlap when reassembling the processed segments. The settings for processing each frame are either based on different presets for the self-speech category and the external speech category (e.g., for processes that require different processing of the self-speech and the external speech, such as ambience suppression), or based on the same settings (e.g., for examples where the goal is to obtain the same result, such as in the case of leveling).

[0058] In some implementations:

[0059] - Classification of self-speech and external speech can be achieved using bone conduction vibration sensors. In such implementations, in addition to using features or as an alternative, the classifier can perform classification based on bone conduction vibration sensor data. For example, the classifier can classify audio as self-speech in response to the detection of bone vibration corresponding to speech, or classify audio as external speech in response to the detection of no bone vibration based on data from the bone conduction vibration sensor. Bone conduction vibration sensor data can thus supplement or replace the aforementioned features.

[0060] - The frame size used for MFCC, VAD and other features may vary; in this case, when different features are combined or different metrics are derived from the features, the coarser features can be "upsampled" to the resolution of the finest features by interpolation or simple nearest neighbor repetition.

[0061] The aspects of the system described herein can be implemented in a suitable computer-based audio processing network environment to process digital or digitized audio files. Parts of the adaptive audio system may include one or more networks comprising any desired number of independent machines, including one or more routers (not shown) for buffering and routing data transmitted between computers. Such networks can be built on a variety of different network protocols and can be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0062] One or more components, blocks, processes, or other functional components may be implemented by a computer program executed by a processor-based computing device controlling the system. It should also be noted that any number of combinations of hardware, firmware, and / or data and / or instructions embodied in various machine-readable or computer-readable media may be used to describe the various functions disclosed herein in terms of behavior, register transfers, logical components, and / or other characteristics. Computer-readable media that may embody such formatted data and / or instructions include, but are not limited to, various forms of physical (non-transitory), non-volatile storage media, such as optical, magnetic, or semiconductor storage media.

[0063] While one or more embodiments have been described by way of example and specific examples, it should be understood that one or more embodiments are not limited to the disclosed embodiments. Rather, they are intended to cover various modifications and similar arrangements that will be obvious to those skilled in the art. Therefore, the scope of the appended claims should be given the broadest interpretation in order to cover all such modifications and similar arrangements.

Claims

1. A method comprising: Receives binaural speech signals recorded through a device worn by the user; Divide the binaural speech signal into frames; Apply time-frequency transformation to each frame; The features of the frame are calculated based on the time-frequency representation; Each frame is classified into self-speech or external speech by a classifier, at least in part based on a subset of features, wherein self-speech is the user's speech and external speech is the speech of someone other than the user. Calculate the dissimilarity function based on feature subsets; The signal is segmented at the peak of the dissimilarity function; For each segment, the corresponding overall category in its own speech or external speech is determined by aggregating classifier data from frames belonging to that segment; and Each segment is processed using a speech enhancement chain, the settings of which are based on an overall category determined for that segment.

2. The method of claim 1, further comprising using Voice Activity Detection (VAD) to calculate the corresponding speech probability for each frame, wherein, Only frames with a speech probability greater than a predefined value are considered for classification and segmentation.

3. The method as described in claim 1, wherein, The features include at least one of the following: energy of each frequency band, spectral slope within a predefined frequency range, interchannel coherence of each frequency band, or Mel frequency cepstral coefficients.

4. The method according to any one of claims 1-3, wherein, The classifier is a support vector machine.

5. The method according to any one of claims 1-3, wherein, The dissimilarity function is obtained by applying the Bayesian Information Criterion (BIC) to a subset of the features.

6. The method of claim 5, further comprising retaining the peak value of the dissimilarity function, provided that the value of the dissimilarity function is greater than a predefined value and the distance to the nearest peak value is greater than another predefined value.

7. The method of claim 1, wherein, The corresponding overall category is determined as follows: Calculate the number of frames (CE) in the segment that are classified as external speech; Calculate the number of frames CS in a segment that are classified as its own speech; and If CS >= CE, then assign the speech class itself; or if CE > CS, then assign the external speech class.

8. The method of claim 7, further comprising assigning a corresponding classification confidence value to each segment using the expression abs(CE-CS) / N, where N is the total number of frames in the segment.

9. The method of claim 8, further comprising designating segments with confidence values ​​less than a predefined value as indeterminate.

10. The method of claim 9, comprising: Merge adjacent segments of the same category into a single segment of the same category; as well as Fragments that are specified as indeterminate and are surrounded by two fragments of the same category will be merged with the surrounding fragments.

11. The method according to any one of claims 1-10, wherein, Processing each segment using a speech enhancement chain includes one or more of the following: Noise estimation and noise reduction; Equalization includes specific filtering of one's own speech and external speech; Leveling includes the specific target level and dynamic range of both the speaker's own voice and external speech; Atmosphere balance includes varying degrees of boosting or attenuation of one's own voice and external voice; Spatial rotation, including rotations of varying amounts on one's own speech and external speech; and Vocal tract imbalance correction includes correction of different amounts of one's own speech and external speech.

12. The method of claim 11, wherein: The following excerpt: - Designated as indeterminate; - Surrounded by two fragments of different categories; and - Shorter than the predefined length; Two settings were used for processing and were employed as alternating fade regions; and The following excerpt: - Designated as indeterminate; - Surrounded by two fragments of different categories; and - Longer than the predefined length; It is processed using a neutral setting, or merged with the longest neighboring segment.

13. The method of claim 11, comprising: The processed fragments are reassembled into a sequence according to their order in the original input. as well as Auditory discontinuities are reduced by applying alternating fades at transition points.

14. A system comprising: One or more processors; as well as A non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations of any one of claims 1-13.

15. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause one or more processors to perform the operations of any one of claims 1-13.

Images