Time alignment of signals using attention

Abstract

This document relates to alignment of time-varying signals, such as sound. One example includes a method or technique executable on a computing device. The method or technique can include inputting a first time-varying signal and a second time-varying signal into a trained model. The method or technique can also include performing time alignment of the first time-varying signal with the second time-varying signal using an attention layer of the trained model to obtain a first time-varying signal aligned in time. The method or technique can also include enhancing the second time-varying signal based at least on the first time-varying signal aligned in time, and outputting the enhanced second time-varying signal.

Description

Background Art

[0001] Machine learning can be used to perform a wide range of tasks, such as natural language processing, information retrieval, and image processing. One important application of machine learning involves the processing of time-varying signals, such as audio or video signals. However, conventional machine learning techniques for processing time-varying signals have various drawbacks, as discussed in detail below. Summary of the invention

[0002] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] Techniques for alignment of time-varying signals are generally described. One example includes a method or technique that can be performed on a computing device. The method or technique can include inputting a first time-varying signal and a second time-varying signal into a training model. The method or technique can also include performing time alignment of the first time-varying signal with the second time-varying signal using an attention layer of the training model to obtain a time-aligned first time-varying signal. The method or technique can also include enhancing the second time-varying signal based at least on the time-aligned first time-varying signal, and outputting the enhanced second time-varying signal.

[0004] Another example includes a system having a hardware processing unit and a storage resource storing computer-readable instructions. When executed by the hardware processing unit, the computer-readable instructions enable the system to access training data including a first time-varying signal, a second time-varying signal, and a target time-varying signal. The computer-readable instructions also enable the system to input the first time-varying signal and the second time-varying signal into a model having an attention layer, and the attention layer aligns the first time-varying signal with the second time-varying signal to obtain a first time-varying signal that is aligned in time. The computer-readable instructions also enable the system to enhance the second time-varying signal based on at least the time-aligned first time-varying signal to obtain an enhanced second time-varying signal. The computer-readable instructions also enable the system to modify the parameters of the model based on at least the difference between the enhanced second time-varying signal and the target time-varying signal.

[0005] Another example includes a computer-readable storage medium. The computer-readable storage medium can store instructions that, when executed by a computing device, cause the computing device to perform an action. The action can include receiving a first time-varying signal and a second time-varying signal. The action can also include aligning the first time-varying signal with the second time-varying signal using an attention layer of a neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The detailed description is described with reference to the accompanying drawings. In the drawings, the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. The use of like reference numbers in different instances in the description and the drawings may indicate similar or identical items.

[0007] Figure 1 An example workflow for using a model to align and enhance time-varying signals, consistent with some implementations of the present concepts, is illustrated.

[0008] Figure 2 An example workflow for training a model to align and enhance time-varying signals, consistent with some implementations of the present concepts, is illustrated.

[0009] Figure 3 An example echo cancellation scenario consistent with some implementations of the disclosed technology is illustrated.

[0010] Figure 4 An example echo cancellation neural network consistent with some implementations of the present concepts is illustrated.

[0011] Figure 5 Example alignment blocks consistent with some implementations of the present concepts are illustrated.

[0012] Figure 6 An example system consistent with some implementations of the disclosed technology is illustrated.

[0013] Figure 7 Example methods or techniques for aligning and enhancing time-varying signals consistent with some implementations of the disclosed technology are illustrated.

[0014] Figure 8 An example method or technique for training a model to perform alignment and enhancement of time-varying signals, consistent with some implementations of the disclosed technology, is illustrated.

[0015] Fig. 9 Example user experiences consistent with some implementations of the present concepts are illustrated.

[0016] Fig.10 An example user interface consistent with some implementations of the present concepts is illustrated. DETAILED DESCRIPTION

[0017] Overview

[0018] Some signal processing applications involve time-aligning two or more time-varying signals, such as audio or video signals. One way to align time-varying signals is to use cross-correlation techniques to estimate the delay between the time-varying signals. The estimated delays can then be used to time-align the time-varying signals before augmentation using a machine learning model.

[0019] For example, consider that a microphone picks up a far-end signal from a speaker in the same room as the microphone, which results in an echo. One way to remove the echo from the microphone signal is to use a digital signal processing technique based on cross-correlation analysis to estimate the delay between the microphone signal and the far-end signal. An external delay module can be used to delay the far-end signal so that it is aligned in time with the microphone signal before the echo removal by the machine learning model. Then, the machine learning model can be used to perform echo cancellation processing on the aligned signals.

[0020] Using external delay modules to implement cross-correlation alignment of time-varying signals generally works well when the delay between the varying signals is predictable. However, for many applications, the delays can vary unpredictably. For example, hardware delays and / or software buffering for voice calls can cause relatively large variations in the delay between the microphone and the far-end signal. Additionally, an individual may change rooms during a voice call, which can also change the delay between the microphone and the far-end signal.

[0021] In these scenarios, conventional cross-correlation alignment techniques often fail to estimate delays well. When the estimated delay between two signals is inaccurate, the signals cannot be accurately aligned. Therefore, signal enhancement models that rely on time-aligned signals (such as echo cancellation models) tend to perform poorly when the delay between the two signals is difficult to predict.

[0022] The disclosed implementation scheme generally provides a technique for aligning two or more time-varying signals using an attention layer of a machine learning model (e.g., a deep neural network), which also performs signal enhancement. For example, the time-varying signal can be projected into a latent space. The attention layer can perform an alignment process based on the projection. Subsequent operations, such as enhancement processes for removing echoes or noise, can be performed after the alignment process.

[0023] By performing the alignment inside a machine learning model that also performs the enhancement processing, the layer that performs the signal alignment can be jointly trained with the layer that implements the enhancement processing. In addition, as discussed below, the disclosed implementation allows for "soft" alignment, in which case multiple delay estimates can be weighted relative to each other rather than using a single hard delay estimate from an external delay module. These characteristics of attention-based alignment allow for smooth processing of delay scenarios that often present difficulties for conventional cross-correlation-based techniques. Therefore, the disclosed attention-based alignment techniques can improve the results of subsequent enhancement processing, as will be discussed in more detail below.

[0024] definition

[0025] For the purposes of this document, the term "signal" refers to a function that varies over time or space. A signal can be represented digitally using data samples, such as an audio sample, a video sample, or one or more pixels of an image. A "time-varying signal" refers to a function that varies over time, such as an audio, video, sonar, or radar signal. A time-varying signal can be represented in the time domain, for example, using samples to represent the amplitude of the signal when it is sampled. A time domain signal can also be represented in the frequency domain, for example, using frequency information to represent the range of the time domain signal within different frequency bands. In some cases, the time domain representation can be converted to a frequency domain representation that contains phase information, which can allow the original time domain representation to be restored.

[0026] The term "microphone signal" refers to the audio signal captured by the microphone. The term "far-end signal" refers to the signal played by the speaker in the room. The microphone can pick up the far-end signal distorted by the room impulse response. The term "near-end signal" refers to the ideal microphone signal, that is, the audio signal that the ideal microphone will pick up without noise and echo without introducing distortion. The "predicted near-end signal" is an estimate of the near-end signal. The predicted near-end signal can be obtained by removing the echo caused by the far-end signal and / or removing the noise introduced by hardware or software artifacts or other noise sources. The "target signal" is a training signal (e.g., an ideal signal) that can be generated by training a model. In the case of audio processing, the target signal can be an example near-end signal, which represents the enhanced audio signal that the model is trained to produce assuming that the microphone signal and the far-end signal are input.

[0027] A "data enhancement model" refers to a model that processes data samples from an input signal to enhance the perceived quality of the signal. For example, a data enhancement model may remove noise or echo from audio data, or a data enhancement model may sharpen image or video data. The term "quality estimation model" refers to a model that evaluates an input signal to estimate how a human might rate the perceived quality of the signal. For example, a quality estimation model may estimate the quality of an unprocessed or raw audio signal, and may output a synthetic label that characterizes the signal quality relative to impairments such as device distortion, background noise, and / or room reverberation. The quality estimation model may also be able to evaluate a processed audio signal that is output by a particular data enhancement model to remove noise and / or echo from a noisy input signal, and the quality estimation model may output a synthetic label that reflects the effectiveness of the particular data enhancement model in removing noise and / or echo, and the extent to which the particular data enhancement model may have introduced undesirable artifacts when removing noise. Herein, the term "synthetic label" means a label that is at least partially generated by a machine, while a "manual" label is provided by a human.

[0028] The term "model" as used herein refers to a range of processing techniques and includes models trained using machine learning as well as hand-coded (e.g., heuristic-based) models. For example, a machine learning model can be a neural network, a support vector machine, a decision tree, etc. Whether machine trained or not, a data enhancement model can include a codec or other compression mechanism, an audio noise suppressor, an echo remover, a distortion remover, an image / video healer, a low light enhancer, an image / video sharpener, an image / video denoiser, etc., which will be discussed in detail below.

[0029] The term "impairment" as used in this article refers to any feature of a signal that reduces the perceived quality of the signal. Thus, for example, impairments can include noise or echo that occurs when recording an audio signal, or blur or low light conditions for an image or video. One type of impairment is an artifact, which a data augmentation model introduces when removing impairments from the original data samples. From one perspective, an artifact can be an impairment introduced by processing the input signal to remove other impairments. Another type of impairment is a recording device impairment introduced into the original input signal by a recording device such as a microphone or camera. Another type of impairment is a capture condition impairment introduced by the conditions when the original input signal was captured, for example, room reverberation for audio, low light conditions for images / videos, etc.

[0030] Machine Learning Overview

[0031] There are various types of machine learning frameworks that can be trained to perform a given task, such as estimating the quality of a signal or enhancing a signal. Support vector machines, decision trees, and neural networks are just a few examples of machine learning frameworks that have been used in a variety of applications such as image processing and natural language processing. Some machine learning frameworks, such as neural networks, use multiple layers of nodes to perform specific operations.

[0032] In a neural network, nodes are interconnected via one or more edges. A neural network can include an input layer, an output layer, and one or more intermediate layers. Individual nodes can process their respective inputs according to a predefined function, and provide the output to subsequent layers, or in some cases to previous layers. The input to a given node can be multiplied by the corresponding weight value for the edge between the input and the node. In addition, a node can have an individual bias value, which is also used to generate an output. Various training processes can be applied to learn edge weights and / or bias values. The term "internal parameter" used in this article refers to a learnable value, such as edge weights and bias values, which can be learned by training a machine learning model (such as a neural network). The term "hyperparameter" used in this article refers to the features of model training, such as learning rate, batch size, number of training epochs, number of hidden layers, activation function, etc.

[0033] A neural network structure can have different layers that perform different specific functions. For example, one or more layers of nodes can jointly perform specific operations, such as pooling, encoding, decoding, alignment, prediction, or convolution operations. For the purposes of this document, the term "layer" refers to a group of nodes that share inputs and outputs, for example, sharing inputs and outputs with external sources or other layers in the network. The term "operation" refers to a function that can be performed by one or more layers of nodes. The term "model structure" refers to the overall architecture of a hierarchical model, including the number of layers, the connectivity of the layers, and the types of operations performed by individual layers. The term "neural network structure" refers to the model structure of a neural network. The terms "trained model" and / or "adjusted model" refer to the model structure together with the internal parameters for the model structure, which have been trained or adjusted. It should be noted that two trained models can share the same model structure, but the internal parameter values ​​can be different, for example, if the two models are trained on different training data, or if there is an underlying random process in the training process.

[0034] Overview of Signal Enhancement Using Alignment Processing

[0035] Figure 1An example of a signal enhancement workflow 100 with an internal alignment process is shown. A first input signal 102 and a second input signal 104 are input to a preprocessing 106. The preprocessing can output features representing the input signals to an encoding / alignment process 108, wherein the input signals can be encoded and aligned, as discussed below. For example, the features can include Mel-spectrogram or linear audio spectrogram features representing frequency components of the input signals. These features can be calculated over a frame of each of the input signals, wherein each frame includes a specified number of amplitude samples for each input signal. For example, as discussed below, an implementation uses a frame of 320 amplitude samples for each of the input signals, sampled at 16 khz, wherein each frame is 20 milliseconds long. A hop length of 10 milliseconds can be used so that each subsequent frame includes 160 new samples and 160 old samples.

[0036] Features for each frame can be projected into a linear space during the encoding / alignment process 108. For example, projections can be performed on features of each frame of the first input signal and each frame of the second input signal using a convolution operation. For example, linear convolutions with nonlinear activation functions can be used to obtain projections, thereby producing features that can be input to the next layer. The projections or "encodings" of each frame of the second input signal can be compared with encodings of multiple frames of the first input signal. For example, as discussed below, the encodings of one second's worth of frames (100 hops) of the first input signal can be compared with the encodings of a single frame of the second input signal. The encodings of the frames of the first input signal can be weighted based on their respective similarities to the encoded single frames of the second input signal. The weights can be used to construct a weighted encoding of the first input signal frame, which represents the alignment of the first input signal with the second input signal. As described more below, further encoding processes can be performed on the weighted first input signal frames and second input signal frames using further convolution operations. Each convolution operation can reduce the number of frequency bands of the input it receives.

[0037] After encoding and alignment, the recursive process 110 can process the current encoding of the first input signal and the second input signal based on the hidden state maintained by the previous encoding to produce an output that is sent to the decoder 112. In general, using recursive processing allows the model to take into account previous inputs when processing the current input. This can be very useful in signal processing scenarios where certain artifacts often exhibit memory effects, such as vibrations transmitted from a speaker to a microphone. The recursive process can manipulate the projection of the received frame to provide input to the decoder 112.

[0038] At decoding 112, the encoding is decoded, for example, using more transposed convolution operations. The transposed convolution operations can produce outputs with more frequency bands than the inputs received by the transposed convolution operations. In addition, decoding can also involve jump connections between transposed convolution operations, as discussed more below.

[0039] The output of the decoding process 112 can be input to a prediction process 114. In general, the prediction process can predict values ​​that can be used to enhance the second input signal 104. For example, for audio applications, the predicted values ​​can include suppression masks, where each mask is a coefficient for a corresponding frequency band.

[0040] The values ​​predicted during the prediction process 114 can be used during the enhancement process 116 to obtain an enhanced signal 118. For example, the suppression mask described above can be used to suppress individual frequency bands of the second input signal 104, producing an enhanced audio signal. In some implementations, the enhanced audio signal can have reduced echo and / or noise components that are removed by the enhancement process.

[0041] In general, some or all of the processing of the signal enhancement workflow 100 can be performed within a machine learning model such as a deep neural network. As discussed more below, incorporating alignment processing into a model for signal enhancement can allow an alignment layer (e.g., an attention layer) of the model to be jointly trained with other layers of the model. Thus, the alignment layer can be trained to align the input signals 102 and 104 in a manner that facilitates subsequent enhancement processing. In addition, a weighted frame for representing the first input signal 102 can include components of multiple actual frames. Thus, for example, if the correct alignment of the signal involves two frames of the first input signal spanning a particular frame of the second input signal, the alignment layer can learn to represent it as a weighted combination of the two frames. In contrast, previous schemes using separate delay modules simply delay the signal by a specified amount, and thereby "hard" align a single frame of one signal with a single frame of another signal.

[0042] Training Overview

[0043] Figure 2 An example of a training workflow 200 is shown. The training workflow 200 generally relates to the above Figure 1 Signal enhancement as described, and additional functionality as described below.

[0044] As described above, the first input signal 102 and the second input signal 104 can be aligned before enhancing the second input signal 104, producing an enhanced signal 118. The target signal 120 can represent an ideal signal that the enhancement is trained to replicate. For example, the target signal can be a clean microphone signal with little or no impairments such as echo or noise.

[0045] The loss determination 122 can be used to calculate the loss based on the difference between the enhanced signal 118 and the target signal 120. Based on the loss, parameter adjustment 124 can be performed on any or all internal parameters of the encoding / alignment process 108, the recursive process 110, the decoding process 112, and / or the prediction process 114. In general, the greater the difference between the target signal and the enhanced signal, the greater the loss will be. The parameter adjustment can be proportional to the loss gradient relative to the individual parameters. Examples of loss functions and parameter adjustment techniques are described in detail below.

[0046] Specific implementation for echo cancellation

[0047] The signal enhancement and training workflow described above can be used in a wide variety of applications involving different types of time-varying signals. Specific implementations that can be used for echo cancellation of audio signals are provided below.

[0048] Figure 3 is a schematic diagram of an acoustic echo cancellation scenario 300. Scenario 300 shows that a far-end room 302 communicates with a receiving room 304 using acoustic echo cancellation to enhance call quality. The acoustic echo cancellation system 306 can be formally described as follows. A far-end reference signal f(t) is transmitted to a receiving room, played through a speaker 308, and then picked up by a microphone 310 via an acoustic echo path as a microphone signal m(t). A linear echo path can be simulated by a room impulse response, and a nonlinear distortion model can be used to simulate a nonlinear echo path. The captured microphone signal m(t), which includes a near-end signal s(t), background noise z(t), and echo e(t), is further received by the far-end user as It is important to note that the echo component e(t) from the microphone signal m(t) is a delayed version of the received reference far-end signal f(t) due to the echo propagation path (from the speaker to the microphone) and hardware or software related delays. Therefore, between the far-end user and the near-end user, an acoustic echo cancellation system can be integrated to remove the unwanted echo, using the microphone and the reference far-end signal as input to output an enhanced microphone signal.

[0049] Specific network architecture. Figure 4 A neural network model 400 is illustrated that can be used to output a suppression mask, which in turn can be used to predict an enhanced microphone signal, as described more below. To obtain input features for the neural network model, the audio signal can be sampled at 16kHz, and preprocessing is performed on reference far-end and microphone signals. The input features to the neural network model can include a log power spectrum calculated using a square root Hann window.

[0050] The disclosed implementation can be used to adapt an audio enhancement model to perform acoustic echo cancellation using a self-attention mechanism with built-in deep alignment. As discussed further below, the neural network model 400 can include an encoder, a decoder, and a recursive module. The following uses the notation Represents an arbitrary axis length. Additional information about neural network techniques for enhancing audio signals (which can be adapted as described herein) can be found in Braun et al., “Towards Efficient Models for Real-Time Deep Noise Suppression,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) IEEE, 2021, pp. 656-660.

[0051] Encoder. In the encoding stage, the network includes two branches: the first branch receives the remote features 402 as input, and the second branch receives the microphone features 404 as input. Each convolution block 406, 408, 410 and 412 can include a convolution layer, a batch normalization layer and an exponential linear unit (ELU) activation function. (Clevert et al., "Fast and AccurateDeep Network Learning by Exponential Linear Units (ELUS)", arXiv preprint, arXiv:1511.07289, 2015). The remote branch can include two convolution blocks 406 and 408, followed by an alignment block 414, which takes the remote feature map and the depth corresponding feature map from the microphone branch as input. Therefore, the aligned remote map can be linked 416 to the microphone branch and processed by another two convolution blocks 418 and 420. The example number of cores for the microphone convolution block can be 16, 40, 72 and 32, respectively. The distal branch can consist of 8 and 24 filters. The convolution kernel can have a size of 4×3 and a stride of 1×2 to reduce the number of frequency bins along the frequency axis. Each convolution can be causal, for example, padding can be performed so that lookahead is not used.

[0052] Recursive Block. There is a recursive layer between the encoder and the decoder, which includes a gated recursive unit 422, which is fed with feature maps flattened along the channel and frequency dimensions. Formally, the input can be flattened into Replacing GRU layers with LSTM layers may not necessarily lead to significant performance improvements, and in some cases, GRU layers can provide a beneficial trade-off between performance and computational complexity. The output of the recurrent layer is then reshaped back to

[0053] Skip blocks. Replacing classical skip connections based on cascading or summing with trainable channel-wise scaling and biasing can improve network performance at negligible additional cost in some cases. Moreover, this scheme allows the use of asymmetric encoder-decoder blocks by scaling the number of encoding channels to the number of corresponding decoding channels. Considering the decoder input and the corresponding encoder input Jump blocks can be defined as follows:

[0054] Z=X+Conv(Y),

[0055] in, and Conv(·) has a c1 kernel of size 1×1 with stride 1×1 applied.

[0056] Decoder. The decoding stage can include alternating jump blocks 424, 428, and 432, and convolutional transpose blocks 426, 430, and 434, followed by a final jump block 436 and mask block 438. Reshaped output from GRU Combined with the corresponding features from the encoder (into a jump block), and fed to the transposed convolution block. Each transposed convolution block can include a transposed convolution layer, followed by a batch norm and an ELU activation function. For each transposed convolution layer, a non-causal kernel of size 1×3 can be used to calculate features along the frequency dimension. The span for the encoder can be as described above, and the number of filters for the convolution block can be 32, 48, and 48. Subsequently, the output of the last jump block 436 can be processed by a mask block 438. The mask block can include a convolution layer having a single filter of 1×3, followed by a sigmoid activation. In addition, a learnable parameter can be added to the mask block, which can be multiplied by the output mask. This allows the model to have an internal learnable control gain that can compensate for the excessive suppression that may be caused by the mask block.

[0057] Alignment blocks. Figure 5 An example alignment block 414 is shown in FIG. 4, which can be used to synchronize microphone and far-end latent features. Let the microphone feature be and distal features are First, the feature map can be reduced along the frequency dimension using max pooling layers 502 and 504 (e.g., with a kernel size of 1×4) to reduce the computational cost of alignment. Next, the features can be reshaped 506 and 508 so that and Next, linear projection 510 can project the microphone features to the query and linear projection 512 can project the distal features onto the key The tensor K can be zero-padded at the beginning and clipped with the same d value at the end to generate the synthetic delay. Afterwards, a time axis dot product 514 can be performed on the delayed K and Q. This process can be calculated for each delay index d from a specific interval, determined by the maximum supported delay d max Given, the derived length is d max The vector is further used for softmax activation 516 to predict the most likely delay distribution. Afterwards, the aligned distal features are calculated by weighted sum 518 on the time axis and the corresponding delayed probabilities from D More precisely, X f is delayed, multiplied by the corresponding weight factor from D, and added to the final result X f Note that making a weighted sum instead of a hard choice can improve the robustness to erroneous delay estimates by allowing a flat delay distribution.

[0058] Forecast. Return to reference Figure 4 , the output of the neural network model 400 is a suppression mask 440, which can be used to enhance the microphone signal by removing unwanted components. The output suppression mask can be applied to the complex spectrum of the microphone signal 442, resulting in an enhanced composite microphone signal 444. Next, an inverse short-time Fourier transform (STFT) 446 can be calculated to obtain an enhanced time-domain microphone signal 448.

[0059] Loss Function: The neural network model 400 can be trained by re-propagating the enhanced time-domain microphone signal 448 through the STFT 450 to promote STFT consistency. (Wisdom et al., "Differentiable Consistency Constraints for Improved Deep Speech Enhancement" in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2019, pp. 900-904). During the training phase, the reconstructed complex spectrum of the enhanced signal is fed into the loss function 452 together with the target signal 454 (in the time domain) processed by the STFT 456. The network can be optimized by minimizing a composite compressed mean square error loss that mixes amplitude and phase awareness terms, which was found to outperform other losses in the experiments described below (Ephrat et al., "Looking to Listen at the Cocktail Party: a Speaker-Independent Audio-Visual Model for Speech Separation", ACM Transactions on Graphics (TOG), Vol. 37, No. 4, pp. 1-11, 2018). Formally, the loss function can be given by the following formula:

[0060]

[0061] where c = 0.3 is the compression factor and β = 0.7 is the weighting factor between the composite loss and the amplitude-based loss. For brevity, the dependencies S(k,n) and Other loss functions having other amplitude terms and / or phase-aware terms may also be employed.

[0062] Experimental Results

[0063] The following sections describe the use of Figure 4 Experiments performed on the network shown in and the results obtained through the experiments. To ensure generalization ability, the training data described below is synthesized online using random parameters (e.g., signal-to-noise ratio, distortion, gain, signal-to-echo ratio) for each sample.

[0064] Training set. The network is trained on data from the AEC challenge (Cutler et al., "Interspeech 2021 Acoustic Echo Cancellation Challenge" in Proc. Interspeech, 2021). The training set contains more than 10,000 real audio scenes (e.g., speech clips) collected in diverse environments using different devices.

[0065] Test set. Testing was performed on a blind test set from the "Interspeech 2021 Acoustic Echo Cancellation Challenge" by Cutler et al. in Proc. Interspeech, 2021, which contains real-world recordings in diverse scenarios. The far-end one-way call blind test set ("Interspeech 2021 Acoustic Echo Cancellation Challenge" by Cutler et al. in Proc. Interspeech, 2021) is divided into two parts: FEST-HD, which contains 27 samples in difficult delay estimation situations (e.g., long delays or variable delays as pointed out by the authors); FEST-GEN, which contains 273 samples in other types of scenarios (e.g., nonlinear distortion, static noise). In addition, two synthetic datasets, each containing 500 samples, were generated to specifically address long delay situations. In LD-300-500, the delay is randomly and evenly distributed from 300-500ms, while for LD-500-1000, the delay is distributed from 500-1000ms.

[0066] Evaluation Metrics. AeCMOS was used to test the ability to remove echo (Purin et al., “Aecmos: A SpeechQuality Assessment Metric for Echo Impairment,” arXiv preprint, arXiv:2110.03010, 2021). The metric reflects the pseudo-subjective quality of the sample and is highly correlated with human subjective opinion. To evaluate the echo cancellation capability in the far-end one-way call scenario, echo return loss enhancement (ERLE) was also used. The augmented samples were also submitted to human raters in order to obtain MOS scores (K. Sridhar, R. Cutler, A. Saabas, T. Parnamaa, M. Loide, H. Gamper, S. Braun, R. Aichner, and S. Srinivasan, "Icassp2021 acoustic echo cancellation challenge: Datasets, testing framework, and results" in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE 2021, pp. 151-155). Each sample was annotated by 5 different raters.

[0067] Hyperparameter tuning. For feature generation, a square root Hann window of length 20ms, a jump length of 10ms, and a discrete Fourier transform of length 320 were used. The following results use the name “Align-CRUSE” to refer to the Figure 4 The name “CRUSE” is used to refer to the model shown in “Towards Efficient Models for Real-Time Deep Noise Suppression” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) IEEE, 2021, pp. 656-660.

[0068] To obtain the following results, the CRUSE and Align-CRUSE models were trained in the same way. The Adam optimizer was used to train all networks with batches of 400 samples for 150 epochs and a learning rate of 1.5 10 -4, and the weight decays to 5·10 -6 The maximum delay of one second is set to d max =100.

[0069] Results. Below we provide the results of the Align-CRUSE model on the synthetically generated test sets LD-300-500, LD-500-1000, and the AEC Challenge (Cutler et al., "Interspeech 2021 AcousticEcho Cancellation Challenge" in Proc. Interspeech, 2021) one-way call test set, divided into FEST-HD and FEST-GEN. The baseline CRUSE model performs well on non-aligned data (CRUSE), online aligned data and global alignment data The inference time is in milliseconds per frame. For both AECMOS and ERLE metrics, larger values ​​mean better results. Table 1 shows the performance of the Align-CRUSE model on the AEC (double talk) challenge test set compared to the Comparison results with baseline methods.

[0070]

[0071]

[0072] Table 1 shows the difference between non-aligned and real-time alignment. and global alignment data Experiments were performed on the CRUSE architecture tested on the CNN, highlighting the importance of alignment. It should be noted that global alignment methods are based on the entire audio sample and are therefore generally not suitable for real-time applications. Nevertheless, these results are included as a strong baseline. As shown above, Align-CRUSE significantly outperforms the baseline models, including the globally aligned On the synthetic LD-300-500 and LD-500-1000 datasets, Align-CRUSE outperforms Up to 0.6 AECMOS and 7 dB ERLE. The difference is more significant compared to other baseline models.

[0073] To measure the generalization ability of Align-CRUSE, the model was tested on real-world records from the FEST-HD and FEST-GEN datasets. For both FEST-GEN and FEST-HD datasets, Align-CRUSE achieved better results than the baseline. The improvement is 7.22 dB ERLE and 0.24 AECMOS for FEST-HD, and 2.44 dB ERLE and 0.08 AECMOS for FEST-GEN. The results show that it is important to have a robust delay estimator for difficult delay estimation cases. The self-alignment scheme of Align-CRUSE achieves excellent results in all AEC datasets, consistently outperforming the previous two approaches in difficult delay estimation cases. baseline, while also improving overall performance.

[0074] In addition, Table 1 shows the number of parameters and inference time per frame on a CPU Intel Core i7 10600K@3.8 GHz. Align-CRUSE is about 10% faster than the baseline at each frame with only 0.01M more parameters. Therefore, Align-CRUSE provides significantly better results, especially in the case of long delays and hard delay estimates, while ensuring low inference time, which is a critical aspect for real-time processing. Overall, Align-CRUSE improves AEC in both one-way and two-way talk scenarios. Considering that the performance improvement is accompanied by a simplification of the communication pipeline and a 10% reduction in inference time, Align-CRUSE can be successfully applied to real-time applications. An additional observation is that during the experiments, Align-CRUSE predicted a constant delay most of the time, but sometimes a softer delay estimate was provided by the model.

[0075] In addition to the far-end one-way talk test, the models were also compared on a blind two-way talk dataset from the AEC Challenge (Cutler et al., "Interspeech 2021 Acoustic Echo Cancellation Challenge" in Proc. Interspeech, 2021), measuring AECMOS and MOS. These results are shown in Table 2 below:

[0076]

[0077] Here, only is included in these results as a baseline because CRUSE tested on non-aligned data performs quite poorly (see Table 1), and Not applicable for real-time applications. As shown in Table 2, the Align-CRUSE model outperforms the CRUSE model in both AEC performance metrics in the double talk scenario. network.

[0078] Example System

[0079] This implementation can be executed in various scenarios on various devices. Figure 6 An example system 600 is shown in which the present implementations can be employed, as discussed more below.

[0080] As in Figure 6 As shown in FIG. 6 , system 600 includes client device 610, server 620, server 630, and server 640 connected via one or more networks 650. It should be noted that the client device can be embodied as a mobile device such as a smart phone or a tablet computer, or as a fixed device such as a desktop computer, a server device, etc. Similarly, the server can be implemented using various types of computing devices. In some cases, Figure 6 Any of the devices shown in , especially the servers, can be implemented in a data center, a server farm, etc.

[0081] exist Figure 6 Specific components of the devices shown in 600 and 610 may be referred to herein by reference numbers in parentheses. For purposes of the following description, parentheses (1) indicate that a given component is present on client device 610, (2) indicates that a given component is present on server 620, (3) indicates that a given component is present on server 630, and (4) indicates that a given component is present on server 640. Unless a specific instance of a given component is specified, this document will generally refer to the component without parentheses.

[0082] In general, devices 610, 620, 630, and / or 640 may have respective processing resources 601 and storage resources 602, which will be discussed in detail below. These devices may also have various modules that use processing resources and storage resources to perform the techniques discussed herein. Storage resources can include both: persistent storage resources, such as magnetic or solid-state drives; and volatile storage, such as one or more random access memory devices. In some cases, modules are provided as executable instructions that are stored on a persistent storage device, loaded into a random access memory device, and read from the random access memory by a processing resource for execution.

[0083] The client device 610 can include a preprocessing module 603(1), an alignment module 604(1), and an enhancement module 605(1). The preprocessing module 603(1) can preprocess the signal to obtain features. The features can be input to the alignment module 604(1), and the alignment module 604(1) can align the signal, for example, using the encoding techniques described above. The enhancement module 605(1) can be used to enhance the aligned signal using the recursive processing, decoding, and prediction / applying masks to the signal described above. The servers 620 and 630 can have respective instances of these modules, for example, the preprocessing module 603(2), alignment module 604(2), and enhancement module 605(2) on the server 620, and the preprocessing module 603(3), alignment module 604(3), and enhancement module 605(3) on the server 630. In general, the preprocessing, alignment, and enhancement modules can implement the above-described Figure 1 A signal enhancement workflow 100 is described.

[0084] Server 640 can include a training module 641 that can train respective instances of the preprocessing, alignment, and / or enhancement modules described elsewhere herein. For example, training module 641 can implement the above description of Figure 2 A training workflow 200 is described.

[0085] Example Alignment Method

[0086] Figure 7 An example method 700 consistent with some implementations of the present concepts is illustrated. The method 700 can be implemented on many different types of devices, for example, by one or more cloud servers, by a client device such as a laptop, tablet, or smartphone, or by a combination of one or more servers, client devices, and the like.

[0087] The method 700 begins at block 702, where the first and second time-varying signals are input into a trained model. For example, the trained model can be a deep neural network with an internal alignment layer. In addition, the model can also perform encoding, decoding, recursive and / or mask prediction processing.

[0088] The method 700 continues at block 704, where the first time-varying signal is aligned in time with the second time-varying signal. For example, the encoder of the trained model can include an internal alignment layer. The internal alignment layer can align the signals by generating a weighted representation of the first time-varying signal. The internal alignment layer can generate the weighted representation using an attention technique that weights based on a similarity between an embedding representing the first time-varying signal and an embedding representing the second time-varying signal.

[0089] The method 700 continues at block 706, where the second time-varying signal is enhanced. For example, recursion, decoding and / or mask prediction processing can be performed after time alignment. The second time-varying signal can be enhanced using a prediction mask, for example, by selectively suppressing different frequency components to remove echo and / or noise.

[0090] The method 700 continues at block 708 where the enhanced second time-varying signal is output. For example, the enhanced second time-varying signal can be played back through a speaker, sent to another device for playback over a network, and / or stored in a persistent memory for future playback.

[0091] Block 702 can be performed by the pre-processing module 603 , block 704 can be performed by the alignment module 604 , and blocks 706 and 708 can be performed by the enhancement module 605 .

[0092] Example Training Method

[0093] Figure 8 An example method 800 consistent with some implementations of the present concepts is illustrated. The method 800 can be implemented on many different types of devices, for example, by one or more cloud servers, by a client device such as a laptop, tablet, or smartphone, or by a combination of one or more servers, client devices, and the like.

[0094] Method 800 begins at block 802, where training data is accessed. In some cases, block 802 can include accessing previously generated training data, or can include generating previously described training data. The training data can include a first time-varying signal, a second time-varying signal, and a target time-varying signal. For an audio enhancement scenario, the first time-varying signal can include a far-end signal for playback by a speaker, the second time-varying signal can include a microphone signal detected by a microphone that picks up the far-end signal played back by the speaker, and the target time-varying signal can include a near-end signal representing an ideal microphone signal, the ideal microphone signal having little or no echo or noise.

[0095] Method 800 continues at block 804, where the first time-varying signal and the second time-varying signal are input into a model having an attention layer. For example, the attention layer can perform alignment of the first time-varying signal with the second time-varying signal to produce a first time-varying signal that is aligned in time. For example, the attention layer can perform weighting of multiple remote frames based on similarity to microphone frames to obtain weighted remote frames, for example, using a linear projection (embedding) of the microphone frame as a query and a linear projection of the remote frame as a key. In some cases, the weighting can be performed using a softmax layer to determine a weight vector, and the weighted remote frame can include a sum of remote frames weighted by the weight vector.

[0096] The method 800 continues at block 806, where the second time-varying signal is enhanced based on the first time-varying signal aligned in time. For example, the model can have subsequent layers after the attention layer that produce values, such as a suppression mask, that can be used to enhance the second time-varying signal. In some cases, the suppression mask can be determined in a decoding layer.

[0097] The method 800 continues at block 808, where internal parameters of the model are updated based on the difference between the target signal and the enhanced second time-varying signal (e.g., by modifying the previous internal parameters). For example, if the target signal represents an ideal near-end microphone signal and the enhanced second time-varying signal represents a predicted near-end microphone signal, the internal parameters can be modified so that the model removes echo and / or noise, thereby causing the predicted near-end microphone signal to become relatively more similar to the ideal near-end microphone signal.

[0098] Blocks 802 , 804 , 806 , and 808 can be performed by the training module 641 .

[0099] First Example User Experience

[0100] The disclosed techniques enable improved sound quality via echo (and possibly noise) cancellation. Fig. 9A video call GUI 900 is illustrated that includes a sound quality estimate 902 that conveys a five-star value out of five for an audio signal for a video call. In some cases, a user estimate of signal quality can be used to manually tag the audio or video content of a call for subsequent training and / or adjustment of various enhancement models that employ the signal alignment techniques described herein. For example, a user estimate of a five-star signal quality of a microphone signal enhanced during a call can be used as an example of a target near-end signal. Corresponding original microphone and far-end signals recorded during the call can be provided as training examples along with the enhanced microphone signal. However, in other cases, training examples can be generated by synthesis or by using a trained quality estimation model to assess the quality of a training signal based on the presence or absence of impairments in the training signal.

[0101] Second Example User Experience

[0102] Fig.10 An example configuration graphical user interface ("GUI") 1000 is illustrated that can be presented via the training module 641 to configure specific aspects of training. For example, a feature type element 1001 allows a user to select a feature type for training. Here, the user has selected a logarithmic spectrum, and the GUI may provide other feature type options, such as linear audio spectrogram features. A training budget element 1002 allows a user to select a training budget. Here, the user has selected 150 training epochs. The GUI may provide other options, such as options to use all available training data, specify convergence conditions, etc. In some cases, a text editor may be employed to edit the text of the training code and / or configuration files.

[0103] Recursive module element 1003 allows the user to select the type of recursive module to use. Here, the user has selected a gated module, such as a gated recursive unit. The GUI may provide other options, such as an option to select an LSTM recursive module. Training data path element 1004 allows the user to select a training data path. Optimizer element 1005 allows the user to select an optimizer. Here, the user has selected the Adam optimizer, and the GUI may provide other options, such as stochastic gradient descent.

[0104] When the user clicks submit, the training module 641 can configure itself according to the user selections entered into the configuration GUI 1000. Fig.10The examples of elements shown in are merely illustrative, and other types of elements can be used to specify other aspects of training. For example, elements for configuring the architecture of the neural network can be provided to the configuration GUI, for example, by specifying the number and / or type of layers and how the layers are connected. For example, the configuration GUI can be used to add or remove connections for convolutional layers, pooling layers, transposed convolutional layers, recursive layers, and / or skip models. The configuration GUI can also be used to select the number of cores for each convolutional layer, the pooling type for the pooling layer, etc. In addition, the user can configure other training aspects, such as maximum delay, frame size, hop size, etc.

[0105] Technical Effects

[0106] The disclosed implementations provide several technical improvements over conventional signal alignment techniques, which use separate delay blocks to align signals prior to enhancement processing. By incorporating an alignment module into a machine learning model with subsequent layers that perform signal enhancement, the internal parameters of the alignment module can be jointly trained with the internal parameters of the layers in the model that enhance the signal. As a result, the alignment layers can learn to produce signals that are aligned in time, thereby facilitating subsequent enhancement processing.

[0107] As a related point, conventional individual delay blocks result in a "hard" delay. That is, one time-varying signal is delayed by a specified amount to best match the other time-varying signal. However, as previously described, in some cases, the current frame of a given signal can span two adjacent frames in time in the other signal, e.g., one signal may be delayed by a fraction of a frame relative to the other. For example, assuming a 10 millisecond jump, if the first signal is delayed by 15 milliseconds than the second signal, conventional methods can only delay the first signal by 10 milliseconds or 20 milliseconds, and select a particular frame from the first signal that occurs 10 or 20 milliseconds before the current frame of the second signal. The disclosed implementation can use the weighted self-attention mechanism described above to mix the two closest frames in the first signal to more closely match the current frame of the second signal.

[0108] Several technical problems are solved by using the internal alignment layer described above. First, software caching and hardware delays can cause significant delay differences between the far-end signal and the microphone signal. The disclosed implementation can accurately estimate the delay even when caching or hardware problems cause the delay to vary significantly during the course of a one-way call. For example, the internal alignment layer using attention can adapt to changes in delay more quickly than the smoothing and / or averaging operations typically used in correlation analysis. In addition, the user may sometimes change his or her physical location during a call, moving to a room with a different microphone and / or speaker. Or the user may choose to use a different microphone and / or speaker during a call, even without leaving the room, for example, by putting on a USB headset during a call after initially using the built-in speakers and microphone on his or her laptop. In these cases, the internal alignment layer can quickly adapt to changes in the far-end signal-to-microphone delay.

[0109] Additionally, as described above, the enhanced model utilizing the internal alignment layer has been shown to improve echo cancellation relative to other schemes using different evaluation metrics (such as AECMOS, MOS, and ERLE). Furthermore, in addition to improving the model's ability to cancel echo and / or reduce noise, the use of the internal alignment layer also reduces the model's execution time by approximately 10% compared to other techniques. Thus, the timing signal alignment provided by the disclosed implementation not only improves the model's ability to align and enhance signals, but also does so in a computationally efficient manner.

[0110] Additional Applications

[0111] The specific examples discussed above relate to echo cancellation for use in audio processing applications. However, any two time-varying signals can be aligned consistently with the disclosed implementations. For example, consider two microphones located at different distances from a speaker. The audio signal from one microphone can be delayed using the internal alignment module described herein to align the two microphone signals in time. Even if the microphone worn by the speaker moves farther or closer during the course of the discussion, the disclosed signal alignment techniques can quickly adapt to align the two signals.

[0112] As another example, consider two radar signals detected by an antenna. One signal may be an echo of the other, e.g., a fighter radar may have line of sight with the antenna, while the other signal may be reflected off a nearby mountain and arrive at the antenna slightly later than the line of sight signal. The disclosed implementations can be used to accurately estimate the delay between the two signals. Using the estimated delay, the echo and / or noise introduced by the reflected radar signal can be removed from the line of sight signal to obtain an enhanced signal that is less prone to problems such as false target identification.

[0113] Device Implementation

[0114] As mentioned above about Figure 6 As described, system 600 includes multiple devices, including client device 610, server 620, server 630 and server 640. It should also be noted that not all device implementations can be described, and other device implementations should be obvious to skilled technicians from the above and below descriptions.

[0115] As used herein, the terms "device," "computer," "computing device," "client device," and / or "server device" may refer to any type of device having a certain amount of hardware processing capability and / or hardware storage / memory capability. The processing capability may be provided by one or more hardware processors (e.g., hardware processing units / cores) that may execute computer-readable instructions to provide functionality. The computer-readable instructions and / or data may be stored on a storage device, such as storage / memory and / or a database. As used herein, the term "system" may refer to a single device, multiple devices, and the like.

[0116] The storage resources may be inside or outside of the associated devices. The storage resources may include any one or more of volatile or non-volatile memory, hard disk drives, flash memory devices, and / or optical storage devices (such as CDs, DVDs, etc.). The term "computer-readable medium" as used herein may include signals. In contrast, the term "computer-readable storage medium" does not include signals. Computer-readable storage media include "computer-readable storage devices". Examples of computer-readable storage devices include volatile storage media (such as RAM) and non-volatile storage media (such as hard disk drives, optical disks, and flash memory), etc.

[0117] In some cases, the device is configured with a general-purpose hardware processor and storage resources. In other cases, the device may include a system-on-chip (SOC) type design. In a SOC design implementation, the functions provided by the device can be integrated on a single SOC or multiple coupled SOCs. One or more associated processors may be configured to coordinate with shared resources (such as memory, storage, etc.) and / or one or more dedicated resources (such as hardware blocks configured to perform certain specific functions). Therefore, the terms "processor", "hardware processor" or "hardware processing unit" used in this article may also refer to a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a controller, a microcontroller, a processor core, or other types of processing devices suitable for implementation in conventional computing architectures and SOC designs.

[0118] Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, and not limitation, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0119] In some configurations, any modules / codes discussed herein may be implemented in software, hardware, and / or firmware. In any case, these modules / codes may be provided during device manufacturing or by a middleman who is preparing to sell the device to an end user. In other cases, the end user may later install these modules / codes, such as by downloading an executable code and installing the executable code on the corresponding device.

[0120] Also note that devices generally have input and / or output capabilities. For example, a computing device can have various input mechanisms, such as a keyboard, mouse, touchpad, voice recognition, gesture recognition (e.g., using a depth camera, such as a stereo or time-of-flight camera system, an infrared camera system, an RGB camera system, or using an accelerometer / gyroscope, facial recognition, etc.). A device can also have various output mechanisms, such as a printer, a display, etc.

[0121] It is also noted that the devices described herein can function independently or in cooperation to implement the techniques. For example, the methods and functions described herein can be executed on a single computing device and / or distributed across multiple computing devices communicating via a network 650. Without limitation, the network 650 can include one or more local area networks (LANs), wide area networks (WANs), the Internet, etc.

[0122] Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims, and other features and acts that may be recognized by those skilled in the art are intended to fall within the scope of the claims.

[0123] Various examples are described above. Additional examples are described below. One example includes a method that includes inputting a first time-varying signal and a second time-varying signal into a trained model, using an attention layer of the trained model to perform time alignment of the first time-varying signal with the second time-varying signal to obtain a first time-varying signal that is aligned in time, enhancing the second time-varying signal based at least on the first time-varying signal that is aligned in time, and outputting the enhanced second time-varying signal.

[0124] Another example can include any of the above and / or following examples, wherein the first time-varying signal and the second time-varying signal include audio signals.

[0125] Another example can include any of the above and / or following examples, wherein the first time-varying signal comprises a far-end signal and the second time-varying signal comprises a microphone signal.

[0126] Another example can include any of the above and / or following examples, wherein the enhanced second time-varying signal comprises a predicted near-end signal, and the enhancing comprises removing at least one of the following from the microphone signal: noise, distortion, or echo.

[0127] Another example may include any one of the above and / or following examples, wherein performing time alignment includes: weighting the multiple far-end frames based at least on the similarity between the multiple far-end frames of the far-end signal and the microphone frames of the microphone signal to obtain weighted far-end frames, and the weighted far-end frames represent the alignment of the far-end signal with the microphone signal.

[0128] Another example can include any of the above and / or following examples, wherein the weighting includes using a linear projection generated from the microphone frame as a query and using a linear projection generated from the plurality of far-end frames as a key.

[0129] Another example can include any of the above and / or following examples, wherein the weighting includes calculating a weight vector having weights corresponding to a similarity between the query and each of the keys.

[0130] Another example can include any of the above and / or following examples, wherein the weighting is performed using a softmax layer and the similarity is calculated using a dot product operation on the query and each of the keys.

[0131] Another example can include any one of the above and / or following examples, wherein the weighted far-end frame includes a sum of the plurality of far-end frames weighted by the weight vector.

[0132] Another example can include any of the above and / or following examples, wherein the enhancing includes determining a suppression mask and modifying the microphone signal using the suppression mask.

[0133] Another example can include any of the above and / or following examples, wherein the suppression mask is determined by one or other layer of the trained model that has been jointly trained with the attention layer.

[0134] Another example includes a system comprising a processor and a storage medium storing instructions that, when executed by the processor, cause the system to access training data comprising a first time-varying signal, a second time-varying signal, and a target time-varying signal, input the first time-varying signal and the second time-varying signal into a model having an attention layer that aligns the first time-varying signal with the second time-varying signal to obtain a first time-varying signal that is temporally aligned, enhance the second time-varying signal based at least on the temporally aligned first time-varying signal to obtain an enhanced second time-varying signal, and modify parameters of the model based at least on a difference between the enhanced second time-varying signal and the target time-varying signal.

[0135] Another example can include any one of the above and / or following examples, wherein the target time-varying signal includes a speech segment, the first time-varying signal includes a far-end signal generated based on the speech segment, and the second time-varying signal includes a microphone signal generated based on the speech segment.

[0136] Another example can include any of the above and / or following examples, wherein the modification of the parameter is based at least on a loss function calculated using a frequency domain representation of the enhanced second time-varying signal and the target time-varying signal.

[0137] Another example can include any of the above and / or following examples, wherein the loss function has an amplitude term and a phase-aware term.

[0138] Another example can include any of the above and / or following examples, wherein the model outputs a suppression mask, and the enhanced second time-varying signal is obtained by modifying the microphone signal using the suppression mask.

[0139] Another example can include any of the above and / or following examples, wherein the instructions, when executed by the processor, cause the system to jointly train the attention layer and one or more other layers in the model that output the suppression mask using the training data.

[0140] Another example can include any of the above and / or following examples, wherein the model is trained to suppress echo.

[0141] Another example includes a computer-readable storage medium storing instructions that, when executed by a computing device, cause the computing device to perform actions including receiving a first time-varying signal and a second time-varying signal, and aligning the first time-varying signal with the second time-varying signal using an attention layer of a neural network.

[0142] Another example can include any of the above and / or following examples, wherein the first time-varying signal and the second time-varying signal include different audio signals picked up by different microphones.

Claims

1. A method comprising: inputting the first time-varying signal and the second time-varying signal into the trained model; performing time alignment of the first time-varying signal with the second time-varying signal using an attention layer of the trained model to obtain a first time-varying signal that is aligned in time; enhancing the second time-varying signal based at least on the time-aligned first time-varying signal; as well as An enhanced second time-varying signal is output. 2 . The method of claim 1 , wherein the first time-varying signal and the second time-varying signal comprise audio signals. 3 . The method of claim 2 , the first time-varying signal comprising a far-end signal and the second time-varying signal comprising a microphone signal. 4 . The method of claim 3 , the enhanced second time-varying signal comprising a predicted near-end signal, the enhancing comprising removing at least one of the following from the microphone signal: noise, distortion, or echo.

5. The method according to claim 4, wherein: The execution time alignment comprises: weighting the plurality of far-end frames of the far-end signal based at least on similarities with the microphone frame of the microphone signal to obtain a weighted far-end frame, and The weighted far-end frame represents an alignment of the far-end signal with the microphone signal.

6. The method according to claim 5, wherein: The weighting includes using a linear projection generated from the microphone frame as a query and using a linear projection generated from the plurality of far-end frames as a key.

7. The method according to claim 6, wherein: The weighting includes calculating a weight vector having weights corresponding to similarities between the query and each of the keys.

8. The method according to claim 7, wherein: The weighting is performed using a softmax layer, and the similarity is calculated using a dot product operation on the query and each of the keys.

9. The method according to claim 8, wherein: The weighted far-end frame comprises a sum of the plurality of far-end frames weighted by the weight vector.

10. The method according to claim 9, wherein: The augmentation includes determining a suppression mask and modifying the microphone signal using the suppression mask, and the suppression mask is determined by one or other layer of the trained model that has been jointly trained with the attention layer.

11. A system comprising: processor; as well as a storage medium storing instructions that, when executed by the processor, cause the system to: accessing training data including a first time-varying signal, a second time-varying signal, and a target time-varying signal; inputting the first time-varying signal and the second time-varying signal into a model, the model having an attention layer, the attention layer aligning the first time-varying signal with the second time-varying signal to obtain a first time-varying signal that is aligned in time; enhancing the second time-varying signal based at least on the time-aligned first time-varying signal to obtain an enhanced second time-varying signal; as well as Parameters of the model are modified based at least on a difference between the enhanced second time-varying signal and the target time-varying signal.

12. The system according to claim 11, wherein: The target time-varying signal includes a speech segment, the first time-varying signal includes a far-end signal generated according to the speech segment, and the second time-varying signal includes a microphone signal generated according to the speech segment.

13. The system of claim 12, the parameter being modified based on at least a loss function computed using a frequency domain representation of the enhanced second time-varying signal and a frequency domain representation of the target time-varying signal.

14. The system according to claim 13, wherein: The model outputs a suppression mask, and the enhanced second time-varying signal is obtained by modifying the microphone signal using the suppression mask.

15. The system of claim 14, wherein: When the instructions are executed by the processor, the system: The attention layer is jointly trained with one or more other layers in the model that output the suppression mask to suppress echo using the training data.

Images