Deep source separation architecture

Abstract

A speech separation server includes a deep learning encoder with non-linear activation. The encoder is programmed to: obtain a mixed audio waveform in a time domain; learn generalized patterns from the mixed audio waveform; and generate an encoded representation that effectively characterizes the mixed audio waveform for speech separation.

Description

[0001] Cross-references to related applications

[0002] This patent application claims priority to Spanish Patent Application No. P201930932, filed October 21, 2019; U.S. Provisional Patent Application No. 62 / 957,870, filed January 7, 2020; and U.S. Provisional Patent Application No. 63 / 087,788, filed October 5, 2020, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] This application relates to speech recognition and deep machine learning. More specifically, the example embodiments described below relate to improving neural network architectures for better separation of speech sources. Background Technology

[0004] The methods described in this section are traceable, but not necessarily methods that were previously conceived or pursued. Therefore, unless otherwise specified, no method described in this section should be considered prior art simply by virtue of its inclusion in this section.

[0005] Identifying individual speech sources from mixed speech is challenging. Learning from large amounts of data has led to some progress in this area. Further leveraging deep machine learning could be helpful in improving speech source separation. Attached Figure Description

[0006] The accompanying drawings illustrate, by way of example and not limitation, several exemplary embodiments of the invention, in which:

[0007] Figure 1 An example component of a voice separation server computer according to a disclosed embodiment is illustrated.

[0008] Figure 2 The diagram illustrates the Conv-TasNet neural network.

[0009] Figure 3 An example neural network for speech separation according to a disclosed embodiment is illustrated.

[0010] Figure 4 An example neural network for speech separation according to a disclosed embodiment is illustrated.

[0011] Figure 5 An example convolutional layer with modified gated linear units according to a disclosed embodiment is illustrated.

[0012] Figure 6 An example neural network for speech separation according to a disclosed embodiment is illustrated.

[0013] Figure 7 An example convolutional layer with skip connections and residual connections according to the disclosed embodiments is illustrated.

[0014] Figure 8 The illustration shows an example process performed by a voice separation server computer according to some embodiments described herein.

[0015] Figure 9 The illustration shows an example process performed by a voice separation server computer according to some embodiments described herein.

[0016] Figure 10 This is a block diagram illustrating a computer system on which embodiments of the present invention can be implemented. Detailed Implementation

[0017] In the following description, numerous specific details are set forth for illustrative purposes in order to provide a thorough understanding of the example embodiments(s). However, it will be apparent that the example embodiments(s) ...

[0018] The embodiments are described in the following subsections based on the following summary:

[0019] 1. General Overview

[0020] 2. Example Computer Components

[0021] 3. Function Description

[0022] 3.1.CONV-TASNET

[0023] 3.2. Deep Nonlinear Encoder and Decoder

[0024] 3.2.1.1. Architecture

[0025] 3.2.1.2. Training

[0026] 4. Example Process

[0027] 5. Experimental Results

[0028] 6. Hardware Implementation

[0029] 7. Extensions and Alternatives

[0030] **

[0031] 1. General Overview

[0032] A speech separation server computer (“Server”) and related methods are disclosed. In some embodiments, the Server includes a deep learning encoder with non-linear activation, the deep learning encoder being programmed to: acquire a mixed audio waveform in the time domain; learn a generalized pattern from the mixed audio waveform; and generate an encoded representation that effectively characterizes the mixed audio waveform for speech separation. The mixed audio waveform includes utterances from multiple sound sources over a period of time. The Server also includes a deep learning decoder with non-linear activation, the deep learning decoder being programmed to: acquire encoded representations of individual waveforms corresponding to different speech sources; and generate individual waveforms.

[0033] In some embodiments, the encoder is a convolutional network comprising multiple convolutional layers. At least one of these convolutional layers includes a relatively small one-dimensional (1-D) filter. At least one of these convolutional layers includes a nonlinear activation function such as a parameter-corrected linear unit (PReLU) or a gated linear unit (GLU).

[0034] In some embodiments, the server is programmed to receive a mixed audio waveform spanning a time period in the time domain. For example, the mixed audio waveform might be a mixture of two utterances from two different speakers within a ten-minute interval. The server is programmed to further generate waveform segments from the mixed audio waveform that span overlapping time ranges within that time period. For example, four waveform segments could be obtained from a ten-minute mixed audio waveform, each spanning ten milliseconds and overlapping the time range of the preceding waveform segment by two milliseconds.

[0035] In some embodiments, the server is programmed to run each waveform segment via an encoder to generate an encoded representation of that waveform segment.

[0036] In some embodiments, the server is programmed to apply a separator to a set of coded representations to generate separate coded representations, each spanning a time period and corresponding to a different speech source.

[0037] In some embodiments, the decoder is a convolutional network comprising multiple convolutional layers. At least one of these convolutional layers includes a relatively small 1-D filter. At least one of these convolutional layers includes a non-linear activation function such as PReLU or GLU.

[0038] In some embodiments, the server is programmed to run each separate encoded representation to generate individual waveforms across the time period and corresponding to different speech sources.

[0039] This server offers several technical advantages. The architecture, which combines small contexts and nonlinear transformations for the encoder or decoder, enables the learning of generalized patterns from the mixed audio waveforms of the input, resulting in better characterization of the individual component waveforms and thus better separation of speech sources. Small contexts allow for efficient implementation, even when multiple small contexts are combined. Nonlinearity allows for the creation of complex mappings between input and output, which is beneficial for learning and modeling complex data.

[0040] 2. Example Computer Components

[0041] Figure 1 The illustration shows example components of a voice separation server computer according to a disclosed embodiment. This figure is for illustrative purposes only, and server 102 may include fewer or more functional components or storage components. Each functional component may be implemented as a software component, a general-purpose or special-purpose hardware component, a firmware component, or any combination thereof. Storage components may be implemented using any of a relational database, an object database, a flat file system, or JavaScript Object Notation (JSON) storage. Storage components may be connected to functional components locally or via a network using programming calls, Remote Procedure Call (RPC) facilities, or a message bus. A component may or may not be standalone. Depending on implementation-specific or other considerations, these components may be functionally or physically centralized or distributed.

[0042] In some embodiments, server 102 may include data collection instructions 104, neural model management instructions 106, neural model training instructions 108, and neural model execution instructions 110. Additionally, server 102 may include server database 120.

[0043] In some embodiments, data collection instruction 102 enables the collection of training data, validation data, or actual data, including mixed audio waveforms (from which individual waveforms corresponding to different speech sources are identified). Training data and validation data, including both mixed audio waveforms and corresponding individual waveforms, can be obtained from public data sources. Data collection instruction 102 further enables the collection of various user preferences or system configurations related to the generation, processing, or use of training data, validation data, or actual data.

[0044] In some embodiments, neural model management instructions 106 enable the management of different neural models, including storing or updating multiple convolutional neural networks with different structures. In this context, each neural “model” refers to a set of interrelated executable instructions and data values ​​stored electronically and digitally, capable of receiving and responding to programmatic or other digital invocation, retrieval, or parsing requests based on specified input values ​​to produce one or more stored or computed output values ​​that can serve as the basis for computer-implemented recommendations, output data display, or machine control. Management of different neural models includes acquiring and storing system configurations or user preferences related to the training or execution of the neural models, such as the selection of training algorithms or objective functions.

[0045] In some embodiments, neural model training instruction 108 enables the training or construction of different neural models. Based on data obtained via data collection instruction 104 and relevant parameter values ​​for the training process obtained via neural model management instruction 106, different neural models can be trained, and the training results, including the values ​​of various parameters of the neural models, can be stored via neural model management instruction 106 for future use.

[0046] In some embodiments, neural model execution instructions 110 enable the execution of different neural models. By using the trained neural model obtained via neural model training instructions 108 based on relevant parameter values ​​for the execution process obtained via neural model management instructions 106, the actual data of the mixed audio waveforms can be separated into individual waveforms corresponding to different speech sources. Neural model execution instructions 110 further enable the storage or transmission of the results of executing the neural models, such as individual waveforms.

[0047] In some embodiments, server database 220 is programmed or configured to manage the storage and access to related data such as training data, validation data, actual data, neural models, the composition of the training process, system configuration, user preferences, or the results of training or executing neural models.

[0048] 3. Function Description

[0049] 3.1CONV-TASNET

[0050] Recent publications have reported promising results for handling source separation in the waveform domain. Conv-TasNet is an end-to-end neural network for speech source separation in the waveform domain, as described by Yi Luo and Nima Mesgarani in the following paper: "Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation", IEEE / ACM Transactions on Audio, Speech and Language Processing, Vol. 27, No. 8, pp. 1256-1266, 2019.

[0051] Figure 2 The diagram illustrates a Conv-TasNet neural network. The Conv-TasNet neural network (“the system” or “Conv-TasNet”) includes an encoder 204, a splitter 206, and a decoder 208. The system is programmed to take a mixed audio waveform 202 as input and produce individual waveforms 210. Specifically, the system is programmed to handle single-channel speech mixtures... (where T represents the length of the waveform and) Separate C individual speech sources from ) ,in, .

[0052] Encoder 204 linearly maps the mixed audio waveform 202 into the learned latent space. Specifically, the mixed audio waveform 202 is segmented into K overlapping frames. Each frame of length L has an overlap of S with the previous frame. Then, the linear transformation is defined as:

[0053]

[0054] in, Each row contains N learnable fundamental filters or kernels of size L. Store K frames of length L in columns, and The latent space representation of the mixed audio waveform 202 is given, where each frame is represented by N values. Therefore, the encoder can be implemented as a 1-D convolution with N kernels. In one example, N = 512, L = 16, and S = 8, corresponding to a 2 ms fundamental filter and a 1 ms overlap at a sampling rate of 8 kHz.

[0055] Separator 206 predicts the representation of each speech source by learning a mask in this latent space. Separator 206 comprises stacked dilated convolutional blocks with exponentially increasing dilation factors, and each stack can be repeated multiple times. The deep stacking of dilated convolutions allows the separator to have a large temporal context and a compact model size. Element-wise multiplication 214 then applies the individual masks produced by separator 206 to the mixed audio waveform 202 to generate individual latent space representations.

[0056] The decoder linearly transforms the latent space representation of each estimated clean source c = 1, 2, ..., C to the time domain:

[0057]

[0058] in, It contains N decoder base filters (not bound to encoder U). It is the representation of the c-th estimated source predicted by the separator, and K frames containing the reconstructed signal. The entire time-domain waveform. Through The result is obtained by summing the overlapping rows. Similar to the encoder, the decoder can be implemented as a 1-D (transposed) convolution. All Conv-TasNet building blocks are jointly optimized.

[0059] 3.2 Deep Nonlinear Encoder and Decoder

[0060] 3.2.1 Architecture

[0061] Figure 3 An example neural network for speech separation according to the disclosed embodiments is illustrated. In some embodiments, the neural network includes an encoder and a decoder. The encoder or decoder may be combined with a separator in Conv-TasNet to form an end-to-end neural network for performing speech separation, or it may be combined with another separator (such as in a later improvement to Conv-TasNet) configured to receive input data of the same type as the separator in Conv-TasNet and produce output data of the same type.

[0062] In some embodiments, the encoder includes a 1-D convolution 304, which may be the same as or similar to the encoder 204 in Conv-TasNet that receives a mixed audio waveform and generates a latent space representation of the mixed audio waveform. The 1-D convolution 304 is followed by a stack of first convolutional layers, such as convolutional layers 312, 314, and 316. Each convolutional layer includes a 1-D convolution and a non-linear activation function. For example, each convolutional layer may have a 1-D convolution followed by PReLU, which is known to those skilled in the art.

[0063]

[0064] Where * denotes the convolution operator, i = 2, 3, ..., and I denotes the convolutional layer index. Represents N learnable kernels, It is the output of a 1-D convolution 304, and This is the output of the i-th convolutional layer. Since each... Both represent the latent space representation of the mixed audio waveform 202 (where each frame is represented by N values), and each kernel is actually of size N × g. Convolutions with respect to the mixed audio waveform 202 continue in the time dimension. A stride of S can be used in the convolutions. For example, the value of g could be 3, the value of S could be 1, and the stack size could be 3. This first stack of convolutional layers transforms the mixed audio waveform 202 hierarchically into a non-linear latent space.

[0065] Because the kernels in these convolutional layers have relatively small convolutional sizes in the temporal dimension, the initial convolutional layer extracts more high-resolution or concrete patterns from the mixed audio waveform 202, while subsequent convolutional layers extract more low-resolution or abstract patterns. This abstract, non-linear latent representation allows the separator to make predictions in a space where it is easier to distinguish and extract the significant components of the mixture. Small strides tend to improve temporal resolution and can be combined with smaller kernel sizes to achieve good coverage. The controlled number of filters and the size of the first stack (the number of convolutional layers) then help to achieve a good trade-off between accuracy and speed. For example, the relatively simple structure of each convolutional layer and the entire first convolutional layer stack compared to the convolutional blocks in the separator 206 in Conv-TasNet helps to maintain the performance advantage of the encoder, leaving more execution time for the separator module. Previous encoders or decoders lacked this kind of deep learning that effectively and efficiently represents speech signals for speech separation.

[0066] In some embodiments, the decoder's structure mirrors the encoder's structure. The predicted representation generated from a separator 306, which may be identical or similar to separator 206 in Conv-TasNet, is first processed by a stack of second convolutional layers (such as convolutional layers 322, 324, and 326). Each convolutional layer includes a dimension-preserving 1-D transposed convolution with a non-linear activation function and otherwise operates in the same manner as the convolutional layers in the encoder. The kernels in these convolutional layers of the decoder may be learned separately from the kernels in these convolutional layers of the encoder. Finally, the stack of second convolutional layers is followed by a 1-D transposed convolution 308, which may be identical or similar to decoder 308 in Conv-TasNet, to produce a temporal estimate of the source signal.

[0067] Figure 4An example neural network for speech separation according to a disclosed embodiment is illustrated. In some embodiments, the neural network includes an encoder and a decoder, and... Figure 3 The neural network shown is similar. One difference is that each PReLU is replaced with a GLU or a modified GLU known to those skilled in the art as described below. GLUs typically have more learnable parameters than PReLUs. Similar to attention, GLUs rely on the learned gates to model the relative importance of the kernels. Therefore, the encoder includes 1-D convolutions 404, which may be the same as or similar to encoder 204 in Conv-TasNet, followed by a stack of first convolutional layers, such as convolutional layers 412, 414, and 416. Each convolutional layer includes a 1-D convolution and a non-linear activation function, such as GLU. The structure of the decoder reflects the structure of the encoder. The predicted representation from separator 406, which may be the same as or similar to separator 206 in Conv-TasNet, is first processed by a stack of second convolutional layers (such as 422, 424, and 426). Each convolutional layer includes a dimension-preserving 1-D transposed convolution with a non-linear activation function, such as GLU. Finally, the second convolutional layer is stacked followed by a 1-D transposed convolution 408, which serves as the decoder in Conv-TasNet.

[0068] Figure 5 An example convolutional layer with a modified GLU according to a disclosed embodiment is illustrated. In some embodiments, the convolutional layer (such as...) Figure 4 412 shown includes a GLU having a first path via 1-D convolution 502 and a second path via 1-D convolution 504 and a sigmoid activation function 508 to implement a gating mechanism 510. The convolutional layer further includes a global layer normalization 506 known to those skilled in the art prior to the sigmoid activation function 508, thereby forming a modified GLU to accelerate training.

[0069] In some embodiments, the neural network includes encoders or decoders (such as...) Figure 3 or Figure 4The number of convolutional layers with nonlinear activations in the encoder or decoder shown may be fewer or more than three. In one embodiment, the nonlinear activation function included in the neural network may alternatively be a leak-corrected linear unit (ReLU), a sigmoid function, a TanH function, a Gaussian error linear unit (GELU), a softplus function, an exponential linear unit (ELU), an arcTan function, a squared nonlinear function (SQNL), or other nonlinear activation functions known to those skilled in the art. In one embodiment, multiple convolutional layers have different nonlinear activation functions. In one embodiment, each convolutional layer has a nonlinear activation. In one embodiment, not every convolutional layer after the first convolutional layer with linear activation or without linear activation has a nonlinear activation.

[0070] Figure 6 An example neural network for speech separation according to the disclosed embodiments is illustrated. In some embodiments, the neural network includes an encoder and a decoder. The encoder includes a 1-D convolution 604, which may be the same as or similar to the encoder in Conv-TasNet. The 1-D convolution 604 is followed by a stack of first convolutional layers, such as 612, 614, and 616, regulated by a nonlinear unit (such as PReLU or GLU 618). Each convolutional layer in the encoder establishes skip connections and residual connections, which are known to those skilled in the art. Skip connections are known to be used to avoid gradient vanishing or eliminate singularities in deep learning networks, thereby making the learning process possible or easier. Residual connections lead to the next convolutional layer, while skip connections lead to a summation 620 fed to the nonlinear unit 618. The structure of the decoder reflects the structure of the encoder. The predicted representation from a separator 606, which may be the same as or similar to the separator in Conv-TasNet, is first processed by a stack of second convolutional layers, such as 622, 624, and 626, regulated by a nonlinear unit 628. Each convolutional layer in the decoder establishes skip connections and residual connections, just as in the encoder. Finally, the second convolutional layer is stacked followed by a 1-D transposed convolution 608 that can be the same as or similar to the decoder 208 in Conv-TasNet.

[0071] Figure 7 An example convolutional layer with skip connections and residual connections according to a disclosed embodiment is illustrated. In some embodiments, the convolutional layer (such as...) Figure 6The diagram (612) includes a depthwise convolution 704 (depthwise convolution or depthwise separable convolution known to those skilled in the art), followed by a non-linear activation function 706, such as PReLU. The depthwise convolution 704 can have the same parameters as the 1-D convolution discussed. For example, the kernel size used in each convolution operation is g. The non-linear activation function 706 is then followed by a first path for residual connections 714 via a first pointwise convolution 710 (which is known to those skilled in the art) and a second path for skip connections 712 via a second pointwise convolution 708. In other embodiments, the depthwise convolution 704 PReLU 706 can be composed of another non-linear activation function (such as PReLU). Figure 5 (The GLU shown or the modified GLU) is used instead.

[0072] 3.2.2 Training

[0073] In some embodiments, the encoder and decoder are trained together with the separator. This end-to-end training process is described in the Conv-TasNet disclosure cited above. Instead of the scale-invariant signal-to-noise ratio (SI-SNR) applied in this process, a power-law term is used as the objective function. Specifically, to limit the scale of the prediction sources from the deep encoder or decoder (according to the amplitude of the waveform), a power-law term is added to the SI-SNR objective function to encourage the trained model to predict spectra with amplitudes similar to the ground truth. Power laws are known to be relevant to human perception. The added objective function is written as:

[0074]

[0075] in,

[0076] .

[0077] and This has already been discussed above. STFT stands for Short-Time Fourier Transform. It is the perceptual index that maps the original signal energy in each time-frequency interval to the perceptual domain. This means mapping the spectrum to a domain where human perception of loudness is more linear; for example, doubling the value in this domain should double the perceived loudness of the signal. It is a weighting factor that represents the weight of P-law in the total loss. or The value of is between 0 and 1. In some embodiments, Set to 0.01 and Set it to 0.5.

[0078] In some embodiments, training data spans a specific duration, such as 30 hours, and validation data spans a specific duration, such as 10 hours. Training or validation data can be obtained by randomly mixing utterances from multiple speakers at a randomly selected signal-to-noise ratio (SNR). For example, training and validation data can be generated by randomly mixing utterances from 100 speakers at a randomly selected SNR between -5 dB and 5 dB. The waveform can be sampled at a specific frequency, such as 8,000 Hz.

[0079] 4. Example Process

[0080] Figure 8 and Figure 9 Each illustration depicts an example process performed by a voice separation server computer according to some embodiments described herein. Figure 8 and Figure 9 Each is shown in a simplified illustrative format for the purpose of clear illustration, and other embodiments may include more, fewer, or different elements connected in various ways. Figure 8 and Figure 9 Each aims to disclose an algorithm, plan, or outline that can be used to implement one or more computer programs or other software elements, which, when executed, enable the functional improvements and technological advancements described herein. Furthermore, the flowcharts herein are described with the same level of detail as those typically used by those skilled in the art to communicate with each other regarding the algorithms, plans, or specifications that form the basis of software programs they intend to write or implement using their accumulated technology or knowledge.

[0081] refer to Figure 8 In some embodiments, in step 802, the server is programmed to receive a mixed audio signal comprising audio signals from multiple audio sources. Each of the multiple audio signals may include a waveform in the time domain.

[0082] In some embodiments, in step 804, the server is programmed to transform the mixed audio signal into an encoded representation using an encoder convolutional neural network (CNN) with multiple convolutional layers and non-linear activation.

[0083] In some embodiments, the encoder CNN includes convolutional layers with linear activation or no activation. In other embodiments, the encoder CNN includes convolutional layers performing 1-D convolution operations.

[0084] In some embodiments, nonlinear activations include PReLU, GLU, normalized GLU, leaky ReLU, sigmoid function, or TanH function. In other embodiments, the encoder CNN includes three convolutional layers, each of which has a nonlinear activation.

[0085] In some embodiments, the encoder CNN includes one or more convolutional layers having residual connections and skip connections. The encoder CNN may further include nonlinear activation following the one or more convolutional layers having residual connections and skip connections. Additionally, the convolutional layers within the one or more convolutional layers may include one or more nonlinear activation functions.

[0086] In some embodiments, in step 806, the server is programmed to separate the coded representation into multiple individual representations corresponding to multiple audio sources. The separation can be performed by a separator CNN comprising stacked dilated convolutional blocks.

[0087] In some embodiments, in step 808, the server is programmed to transform multiple individual representations into multiple audio signals corresponding to multiple audio sources using a decoder CNN having multiple convolutional layers and nonlinear activations. The structure of the decoder CNN can correspond to the structure of the encoder CNN.

[0088] In some embodiments, the server is programmed to receive multiple sample audio signals corresponding to multiple audio sources. The server is further programmed to construct an encoder CNN from the multiple sampled audio signals using a permutation-invariant training method that includes an objective function including SI-SNR.

[0089] refer to Figure 9 In some embodiments, in step 902, the server is programmed to receive a mixed audio signal comprising audio signals from multiple audio sources, the mixed audio signal spanning a certain time range in the time domain.

[0090] In some embodiments, in step 904, the server is programmed to divide the mixed audio signal into multiple segments of a specific length, at least two of which span overlapping subranges of time ranges in the time domain.

[0091] In some embodiments, in step 906, the server is programmed to apply an encoder CNN to each of the plurality of segments to generate an encoded representation of that segment, the encoder CNN having multiple convolutional layers and nonlinear activations.

[0092] In some embodiments, nonlinear activations include PReLU, GLU, normalized GLU, leaky ReLU, a sigmoid function, or a TanH function. In other embodiments, the encoder CNN includes three convolutional layers, each of which has a nonlinear activation. In still other embodiments, the encoder CNN includes one or more convolutional layers with residual connections and skip connections.

[0093] In some embodiments, in step 908, the server is programmed to transmit multiple coded representations of multiple segments to a splitter to generate multiple individual representations corresponding to multiple audio sources.

[0094] In some embodiments, the server is programmed to receive multiple individual representations from a separator. The server is further programmed to apply a decoder CNN to each of the multiple individual representations to generate an audio signal spanning a time range; the decoder CNN has multiple convolutional layers and non-linear activations. The server is then programmed to transmit multiple audio signals.

[0095] 5. Experimental Results

[0096] Experiments were conducted using an exemplary nonlinear depth encoder / decoder with I layers. The first layer is equivalent to the original Conv-TasNet encoder, where a linear transformation is applied to a frame of length L and stride S. It is implemented via a 1-D convolutional layer with N kernels. The first layer is followed by a stack of I-1 1-D convolutional layers, each with N kernels of size N × 3 and a PReLU as described above according to Equation 3 (shown below).

[0097]

[0098] Where * denotes the convolution operator, i = 2, 3, ..., and I denotes the convolutional layer index. Represents N learnable kernels, It is the output of a 1-D convolution 304, and It is the output of the i-th convolutional layer.

[0099] The first variant of the exemplary nonlinear depth encoder / decoder adds temporal context to the deep layers by employing dilated convolutions.

[0100] As described above, the second variant of the exemplary nonlinear deep encoder / decoder further increases capacity by using GLU instead of PReLU as the activation function. Furthermore, a global layer normalization is inserted in the GLU before the sigmoid nonlinearity to accelerate training.

[0101] In the first experiment, the performance of nonlinear deep encoders / decoders and their variants for speech source separation was evaluated on the WSJ0 2-speaker (WSJ0-2mix) database, which is widely known to those skilled in the art. The results of the first experiment are summarized in Table 1 below.

[0102]

[0103] The training set (30 hours) and validation set (10 hours) were created by randomly mixing utterances from 100 speakers with an SNR between -5 dB and 5 dB at randomly selected frequencies. All waveforms were sampled at 8000 Hz. Table 1 summarizes the SI-SNR improvements (SI-SNRi) before and after speech separation on the test set (5 hours, 16 unseen speakers). First, the original Conv-TasNet was successfully replicated to build upon their results (Table 1, row 1). Second, note that BigConv-TasNet (Table 1, row 2), i.e., the modified Conv-TasNet with double the number of kernel filters and four (instead of three) temporal convolutional networks (TCNs) in the separator, does not outperform the original Conv-TasNet. And third, the deep encoder / decoder (Table 1, row 3) provides a 0.7 dB improvement over the baseline. This result highlights the importance of the architecture itself, as simply increasing the model's capacity does not improve objective metrics. Therefore, as can be seen in Table 1, the nonlinear depth encoder / decoder and its variants significantly outperform the original Conv-TasNet and BigConv-TasNet.

[0104] 6. Hardware Implementation

[0105] According to one embodiment, the techniques described herein are implemented by at least one computing device. These techniques can be implemented, in whole or in part, using a combination of at least one server computer and / or other computing devices coupled via a network (such as a packet data network). The computing device may be hardwired to perform these techniques, or may include digital electronic devices such as at least one application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA) persistently programmed to perform these techniques, or may include at least one general-purpose hardware processor programmed to perform these techniques according to program instructions in firmware, memory, other storage devices, or combinations thereof. Such a computing device may also combine custom hardwired logic, ASIC, or FPGA with custom programming to implement these described techniques. The computing device may be a server computer, workstation, personal computer, portable computer system, handheld device, mobile computing device, wearable device, body-installed or implantable device, smartphone, smart appliance, network-connected device, autonomous or semi-autonomous device such as a robot or unmanned ground or air vehicle, any other electronic device incorporating hardwired and / or program logic to implement the described techniques, one or more virtual computing machines or instances in a data center, and / or a network of server computers and / or personal computers.

[0106] Figure 10This is a block diagram illustrating an example computer system that can be implemented using this embodiment. Figure 10 In the example, the computer system 1000 and the instructions for implementing the disclosed technology in hardware, software or a combination of hardware and software are schematically represented, for example, by boxes and circles, with the same level of detail as commonly used by those skilled in the art to which this disclosure pertains in communicating computer architecture and computer system implementations.

[0107] Computer system 1000 includes an input / output (I / O) subsystem 1002, which may include a bus and / or other communication mechanisms for transmitting information and / or instructions between components of computer system 1000 via electronic signal paths. I / O subsystem 1002 may include an I / O controller, a memory controller, and at least one I / O port. Electronic signal paths are schematically represented in the figures as, for example, lines, unidirectional arrows, or bidirectional arrows.

[0108] At least one hardware processor 1004 is coupled to an I / O subsystem 1002 for processing information and instructions. The hardware processor 1004 may include, for example, a general-purpose microprocessor or microcontroller and / or a dedicated microprocessor such as an embedded system, graphics processing unit (GPU), digital signal processor, or ARM processor. The processor 1004 may include an integrated arithmetic logic unit (ALU) or may be coupled to a separate ALU.

[0109] Computer system 1000 includes one or more units of memory 1006, such as main memory, coupled to I / O subsystem 1002 to electronically and digitally store data and instructions to be executed by processor 1004. Memory 1006 may include volatile memory (such as various forms of random access memory (RAM)) or other dynamic storage devices. Memory 1006 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 1004. When stored in a non-transitory computer-readable storage medium accessible to processor 1004, such instructions can transform computer system 1000 into a dedicated machine customized to perform the operations specified in the instructions.

[0110] Computer system 1000 further includes non-volatile memory (such as read-only memory (ROM) 1008) or other static storage devices coupled to I / O subsystem 1002 to store information and instructions of processor 1004. ROM 1008 may include various forms of programmable ROM (PROM), such as erasable PROM (EPROM) or electrically erasable PROM (EEPROM). Persistent storage unit 1010 may include various forms of non-volatile RAM (NVRAM), such as flash memory or solid-state storage devices, magnetic disks or optical disks (such as CD-ROM or DVD-ROM), and may be coupled to I / O subsystem 1002 for storing information and instructions. Storage device 1010 is an example of a non-transitory computer-readable medium that can be used to store instructions and data that are executed by a computer-implemented method performing the techniques described herein when executed by processor 1004.

[0111] Instructions in memory 1006, ROM 1008, or storage device 1010 may include one or more sets of instructions organized as modules, methods, objects, functions, routines, or calls. Instructions may be organized as one or more computer programs, operating system services, or application programs including mobile applications. Instructions may include operating system and / or system software; one or more libraries supporting multimedia, programming, or other functions; data protocol instructions or stacks for implementing TCP / IP, HTTP, or other communication protocols; file processing instructions for interpreting and rendering files encoded using HTML, XML, JPEG, MPEG, or PNG; user interface instructions for rendering or interpreting commands for graphical user interfaces (GUIs), command-line interfaces, or text user interfaces; and application software such as office suites, internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games, or other applications. These instructions may implement web servers, web application servers, or web clients. Instructions may be organized as a presentation layer, an application layer, and a data storage layer such as a relational database system using Structured Query Language (SQL) or NoSQL, object storage, graphical databases, flat file systems, or other data storage.

[0112] Computer system 1000 may be coupled to at least one output device 1012 via I / O subsystem 1002. In one embodiment, output device 1012 is a digital computer display. Examples of displays that may be used in various embodiments include touch screen displays, light-emitting diode (LED) displays, liquid crystal displays (LCDs), or electronic paper displays. Computer system 1000 may include other types of output devices 1012 as alternatives to or supplements to the display device. Examples of other output devices 1012 include printers, ticket printers, plotters, projectors, sound cards or video cards, speakers, buzzers or piezoelectric devices or other audible devices, lamps or LED or LCD indicators, haptic devices, actuators, or servers.

[0113] At least one input device 1014 is coupled to the I / O subsystem 1002 for transmitting signals, data, command selections, or gestures to the processor 1004. Examples of input devices 1014 include touchscreens, microphones, still and video digital cameras, alphanumeric keys and other keys, keypads, keyboards, graphics tablets, image scanners, joysticks, clocks, switches, buttons, dials, sliders, and / or various types of sensors such as force sensors, motion sensors, thermal sensors, accelerometers, gyroscopes, and inertial measurement unit (IMU) sensors, and / or various types of transceivers such as wireless (e.g., cellular or Wi-Fi) transceivers, radio frequency (RF) transceivers or infrared (IR) transceivers, and global positioning system (GPS) transceivers.

[0114] Another type of input device is a control device 1016, which can perform cursor control or other automatic control functions, such as navigation in a graphical interface on the display screen, as an alternative to or supplement to the input function. The control device 1016 can be a touchpad, mouse, trackball, or cursor arrow keys to transmit directional information and command selections to the processor 1004 and to control cursor movement on the display 1012. This input device can have at least two degrees of freedom on two axes (a first axis (e.g., the x-axis) and a second axis (e.g., the y-axis)), allowing the device to specify orientation in a plane. Another type of input device is a wired control device, a wireless control device, or an optical control device, such as a joystick, stick, console, steering wheel, pedals, gear shift mechanism, or other types of control devices. Input device 1014 can include a combination of multiple different input devices, such as a camera and a depth sensor.

[0115] In another embodiment, the computer system 1000 may include Internet of Things (IoT) devices, wherein one or more of the output device 1012, input device 1014, and control device 1016 are omitted. Alternatively, in such an embodiment, the input device 1014 may include one or more cameras, motion detectors, thermometers, microphones, seismic detectors, other sensors or detectors, measuring devices, or encoders, and the output device 1012 may include a dedicated display such as a single-line LED or LCD display, one or more indicators, display panels, instruments, valves, solenoids, actuators, or servers.

[0116] When computer system 1000 is a mobile computing device, input device 1014 may include a Global Positioning System (GPS) receiver coupled to a GPS module capable of triangulating multiple GPS satellites, determining, and generating geographic location or location data, such as the latitude-longitude values ​​of the geophysical location of computer system 1000. Output device 1012 may include hardware, software, firmware, and interfaces for generating, alone or in combination with other application-specific data, location report packets, notifications, pulse or heartbeat signals, or other repetitive data transmissions specifying the location of computer system 1000, pointing to host 1024 or server 1030.

[0117] Computer system 1000 may implement the techniques described herein using custom hardwired logic, at least one ASIC or FPGA, firmware, and / or program instructions or logic, which, when loaded and used or executed, combine with the computer system to enable the computer system to operate as a dedicated machine. According to one embodiment, computer system 1000 executes the techniques herein in response to processor 1004 executing at least one sequence of at least one instruction contained in main memory 1006. Such instructions may be read into main memory 1006 from another storage medium (such as storage device 1010). Execution of the instruction sequence contained in main memory 1006 causes processor 1004 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0118] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that enable a machine to operate in a particular manner. Such storage media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs or magnetic disks, such as storage device 1010. Volatile media include dynamic memory, such as memory 1006. Common forms of storage media include, for example, hard disks, solid-state drives, flash drives, magnetic data storage media, any optical data storage media or physical data storage media, memory chips, etc.

[0119] Storage media differ from transmission media but can be used in conjunction with them. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including the conductors of bus 1002 containing the I / O subsystem. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0120] Various forms of media can involve loading at least one sequence of at least one instruction to processor 1004 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit these instructions using a modem via a communication link such as fiber optic or coaxial cable or telephone line. A modem or router local to computer system 1000 may receive data on the communication link and convert the data for computer system 1000 to read. For example, a receiver such as a radio frequency antenna or infrared detector may receive data carried in a wireless or optical signal, and appropriate circuitry may provide the data to I / O subsystem 1002, such as by placing the data on a bus. I / O subsystem 1002 loads the data into memory 1006, from which processor 1004 retrieves and executes the instructions. The instructions received by memory 1006 may optionally be stored on storage device 1010 before or after execution by processor 1004.

[0121] Computer system 1000 also includes a communication interface 1018 coupled to bus 1002. Communication interface 1018 provides bidirectional data communication coupling to network links(s)(s)(s)(s), which(s)(s)) are directly or indirectly connected to at least one communication network, such as network 1022 or a public or private cloud on the Internet. For example, communication interface 1018 may be an Ethernet network interface, an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem to provide data communication connectivity to a corresponding type of communication line (e.g., Ethernet cable or any kind of metal cable or fiber optic cable or telephone line). Network 1022 broadly represents a local area network (LAN), a wide area network (WAN), a campus network, the Internet, or any combination thereof. Communication interface 1018 may include a LAN card providing data communication connectivity to a LAN-compatible network, or a cellular wireless telephone interface for wired transmission or reception of cellular data according to a cellular wireless telephone network standard, or a satellite radio interface for wired transmission or reception of digital data according to a satellite wireless network standard. In any such implementation, the communication interface 1018 transmits and receives electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information via a signal path.

[0122] Network link 1020 typically uses technologies such as satellite, cellular, Wi-Fi, or Bluetooth to provide electrical, electromagnetic, or optical data communication directly or via at least one network to other data devices. For example, network link 1020 can provide a connection to host computer 1024 via network 1022.

[0123] Furthermore, network link 1020 can provide connections to other computing devices via network 1022 or via Internet service provider (ISP) 1026 operating Internet devices and / or computers. ISP 1026 provides data communication services through a worldwide packet data communication network (represented as Internet 1028). Server computer 1030 may be coupled to Internet 1028. Server 1030 broadly represents any computer, data center, virtual machine or virtual computing instance with or without a hypervisor, or computer running containerized programs such as DOCKER or KUBERNETES. Server 1030 may represent electronic digital services implemented using more than one computer or instance and accessed and used by transmitting web service requests, Uniform Resource Locator (URL) strings with parameters in the HTTP payload, API calls, application service calls, or other service calls. Computer system 1000 and server 1030 may form elements of a distributed computing system that includes other computers, processing clusters, server groups, or other computer organizations collaborating to perform tasks or execute applications or services. Server 1030 may include one or more sets of instructions organized as modules, methods, objects, functions, routines, or calls. Instructions may be organized as one or more computer programs, operating system services, or applications including mobile applications. Instructions may include operating system and / or system software; one or more libraries supporting multimedia, programming, or other functions; data protocol instructions or stacks for implementing TCP / IP, HTTP, or other communication protocols; file format processing instructions for interpreting or rendering files encoded using HTML, XML, JPEG, MPEG, or PNG; user interface instructions for rendering or interpreting commands for graphical user interfaces (GUIs), command-line interfaces, or text user interfaces; and application software such as office suites, internet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games, or other applications. Server 1030 may include a web application server hosting a presentation layer, an application layer, and a data storage layer such as a relational database system using Structured Query Language (SQL) or NoSQL, object storage, graph database, flat file system, or other data storage.

[0124] Computer system 1000 can send messages and receive data and instructions, including program code, through networks (multiple), network links 1020, and communication interfaces 1018. In the Internet example, server 1030 can transmit application request codes through the Internet 1028, ISP 1026, local network 1022, and communication interface 1018. The received code can be executed by processor 1004 upon receipt and / or stored in storage device 1010 or other non-volatile memory for later execution.

[0125] The execution of the instructions described in this section can be implemented as a process, which is an instance of a computer program being executed and consisting of program code and its current activity. Depending on the operating system (OS), a process can consist of multiple execution threads executing instructions concurrently. In this context, the computer program is a passive collection of instructions, while the process can be the actual execution of those instructions. Several processes can be associated with the same program; for example, opening several instances of the same program typically means executing more than one process. Multitasking can be implemented to allow multiple processes to share processor 1004. While each processor 1004 or processor core executes a single task at a time, the computer system 1000 can be programmed to implement multitasking to allow each processor to switch between executed tasks without waiting for each task to complete. In embodiments, switching can occur when a task performs an input / output operation, when a task indicates it can be switched, or when a hardware interrupt occurs. Time-sharing can be implemented to allow for rapid response in interactive user applications by rapidly performing context switching to allow multiple processes to execute concurrently. In embodiments, for security and reliability, the operating system can prevent direct communication between independent processes, providing strictly mediated and controlled inter-process communication functionality.

[0126] 7. Extensions and Alternatives

[0127] In the foregoing description, numerous specific details have been described with reference to embodiments of this disclosure, which may vary depending on the implementation. Therefore, this specification and drawings should be viewed in an illustrative rather than restrictive sense. The sole and exclusive indication of the scope of this disclosure and the applicant's inventive intent within it is the scope of the claims arising from this application, in the specific form of the claims, including any subsequent modifications, both literal and equivalent.

Claims

1. A computer-implemented method for separating audio signals from different speech sources, comprising: The processor receives a mixed audio signal that includes audio signals from multiple speech sources; The processor transforms the mixed audio signal into an encoded representation using an encoder convolutional neural network (CNN) with multiple convolutional layers and non-linear activation. The transformation of the mixed audio signal into an encoded representation using the encoder convolutional neural network includes: The processor uses at least a first convolutional layer to transform the mixed audio signal into an intermediate coded representation, the intermediate coded representation being N-dimensional; The processor transforms the intermediate encoded representation into the encoded representation using at least three subsequent convolutional layers, each layer performing a one-dimensional convolutional operation with N learnable kernels and outputting an output representation with N dimensions, wherein the output representation of the last of the at least three subsequent convolutional layers is the encoded representation, and wherein the at least three subsequent convolutional layers have non-linear activations. The encoded representation is separated into multiple individual representations corresponding to the multiple speech sources; The multiple individual representations are transformed into multiple audio signals corresponding to the multiple speech sources by a decoder CNN, the structure of which corresponds to the structure of the encoder CNN, wherein the decoder CNN includes at least three subsequent decoder convolutional layers and a final decoder convolutional layer, each of the at least three subsequent decoder convolutional layers performing a one-dimensional convolution operation and a non-linear activation.

2. The computer-implemented method of claim 1, wherein each of the plurality of audio signals comprises a waveform in the time domain.

3. The computer-implemented method as described in claim 1 or 2, wherein the first convolutional layer is a convolutional layer with linear activation.

4. The computer-implemented method as described in claim 1 or 2, wherein the nonlinear activation includes a parameter-corrected linear unit (PReLU), a gated linear unit (GLU), or a normalized GLU.

5. The computer-implemented method of claim 1, wherein the encoder CNN comprises one or more convolutional layers having residual connections and skip connections.

6. The computer-implemented method of claim 5, wherein the encoder CNN further includes nonlinear activation following the one or more convolutional layers having residual connections and skip connections.

7. The computer-implemented method of claim 1 or 2, wherein the separation is performed by a separator CNN comprising stacked dilated convolutional blocks.

8. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the following operations: Receive a mixed audio signal comprising audio signals from multiple speech sources, the mixed audio signal spanning a time range in the time domain; The mixed audio signal is divided into multiple segments of a specific length; An encoder CNN is applied to each of the plurality of segments to generate an encoded representation of the segment, wherein the encoder CNN has multiple convolutional layers and nonlinear activations, wherein... The encoder CNN includes: At least a first convolutional layer, the at least first convolutional layer being configured to transform the mixed audio signal into an intermediate coded representation, the intermediate coded representation being N-dimensional; and At least three subsequent convolutional layers performing one-dimensional convolution operations, the at least three subsequent convolutional layers being configured to transform the intermediate encoded representation into the encoded representation for each of the N dimensions using N learnable kernels, wherein each of the at least three subsequent convolutional layers is configured to output an output representation having N dimensions, wherein the output representation of the last layer of the at least three subsequent convolutional layers is the encoded representation, and wherein the at least three subsequent convolutional layers have non-linear activations; The multiple encoded representations of the multiple segments are transmitted to a separator, which generates multiple individual representations corresponding to the multiple speech sources; Receive the plurality of individual representations from the separator; A decoder CNN is applied to each of the plurality of individual representations to generate an audio signal across the time range, the structure of the decoder CNN corresponding to the structure of the encoder CNN, wherein the decoder CNN includes at least three subsequent decoder convolutional layers and a final decoder convolutional layer, each of the at least three subsequent decoder convolutional layers performing a one-dimensional convolution operation and a non-linear activation; Transmit the multiple audio signals.

9. The non-transitory computer-readable storage medium of claim 8, wherein the nonlinear activation includes PReLU, GLU, or normalized GLU.

10. The non-transitory computer-readable storage medium of any one of claims 8 to 9, wherein the encoder CNN comprises one or more convolutional layers having residual connections and skip connections.

11. A system for separating audio signals from different sources, comprising: One or more processors; A memory storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform the following operations: Receive multiple individual representations corresponding to multiple audio signals spanning a time range in the time domain, wherein the multiple separate audio signals correspond to multiple different speech sources and each individual representation is represented by N dimensions; A decoder CNN is applied to each of the plurality of individual representations to generate a corresponding audio signal across the time range, the decoder CNN having multiple convolutional layers and nonlinear activations, wherein each decoder CNN includes: At least three transposed convolutional layers, configured to transform individual representations into intermediate representations using N learnable kernels, wherein each of the at least three transposed convolutional layers is configured to output an N-dimensional representation, wherein the output representation of the last layer of the at least three transposed convolutional layers is the intermediate representation; and At least three subsequent transposed convolutional layers perform one-dimensional convolution operations, the subsequent transposed convolutional layers being configured to transform the intermediate representation to generate a corresponding audio signal across the time range, wherein the at least three transposed convolutional layers have non-linear activation. Transmit the multiple audio signals.

12. The system of claim 11, wherein the decoder CNN comprises one or more convolutional layers having residual connections and skip connections.

13. A computer program product comprising a computer program that, when executed by a processor, causes the processor to perform a computer-implemented method according to any one of claims 1-7.

Images