Speech separation method, device and storage medium

Abstract

The present disclosure relates to the technical field of computers, comprising a voice separation method, device and storage medium. A mixed signal to be separated is obtained; the mixed signal is converted into a time-frequency feature representation by an encoder; a separation model is used to perform feature separation on the time-frequency feature representation to obtain feature sequences corresponding to different output channels respectively; the separation model comprises a buffer unit and at least one separation unit connected in sequence; each separation unit comprises a frequency domain modeling subunit, a time domain modeling subunit and a causal attention refinement subunit connected in sequence, and the time domain modeling subunit in each separation unit is connected with the buffer unit; a decoder is used to decode the feature sequences corresponding to different output channels respectively to obtain time domain voice signals corresponding to each output channel; the buffer unit additionally provided can store historical information during global modeling, so that the separation model can perform feature processing based on the historical information, and the overall voice separation performance of the model can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a speech separation method, device, and storage medium. Background Art

[0002] Speech recognition technology completes the speech recognition task by inputting speech signals acquired by the front-end into the processing back-end. Currently, more advanced voice assistants, or mainstream speech recognition methods, have relatively high requirements for the input speech environment. In actual speech recognition scenarios, the input speech signal environment is often noisy or contains multiple speakers, which affects speech recognition accuracy. Based on this, speech separation technology can be used to separate the speech signals of different speakers from mixed speech including multiple speakers. The separated single speaker signal is then input into the processing back-end for recognition, thereby improving recognition accuracy. Speech separation technology is crucial for the development of fields such as automatic conference transcription, automatic captioning of audio / video, and multi-party human-computer interaction.

[0003] In recent years, with the rapid development of deep learning technology, various deep learning models have emerged, improving speech separation performance. Traditional speech separation tasks are generally divided into two categories: offline speech separation tasks and real-time speech separation tasks. Offline speech separation tasks have been widely studied due to their more comprehensive information and better results. Generally, speech separation models used to handle offline speech separation tasks are also called non-causal models. Conversely, real-time speech separation tasks, such as meeting transcription, require real-time speech separation. Future speech information cannot be used for speech separation at the current moment. Generally, models used to handle real-time speech separation tasks are also called causal models. Causal models require the system to obtain and process information only from the current and previous time frames. Generally, causal models lag behind non-causal models in performance.

[0004] To narrow the performance gap between causal and non-causal speech separation models, causal models typically consider extending the look-ahead window or implementing a joint training strategy. For example, using a Skipping Memory LSTM (SKiM) or Parallel Skipping Memory LSTM (pSKiM) network to extract past audio feature representations and integrate historical hidden layer states to enhance the causal model, thereby reducing latency and improving efficiency.

[0005] However, current causal models cannot effectively retain historical information when processing time series, resulting in poor speech separation performance of causal models. Summary of the Invention

[0006] In view of this, the present disclosure proposes a speech separation method, device and storage medium, which can store historical information during global modeling by additionally setting a cache unit in the separation model, so that the separation model can perform feature processing based on historical information, thereby improving the overall speech separation performance of the model.

[0007] According to one aspect of the present disclosure, a speech separation method is provided, the method comprising:

[0008] Acquire a mixed signal to be separated;

[0009] Converting the mixed signal into a time-frequency feature representation through an encoder;

[0010] The time-frequency feature representation is feature separated by a separation model to obtain feature sequences corresponding to different output channels; wherein the separation model includes a cache unit and at least one separation unit connected in sequence; each separation unit includes a frequency domain modeling subunit, a time domain modeling subunit and a causal attention refinement subunit connected in sequence, and the time domain modeling subunits in each separation unit are connected to the cache unit; for the i-th separation unit, the frequency domain modeling subunit in the i-th separation unit is used to perform feature segmentation on the input feature data along the frequency dimension, and perform frequency domain modeling on the first feature after segmentation based on the first LSTM network to obtain frequency domain modeling features; the time domain modeling subunit in the i-th separation unit is used to perform feature segmentation on the frequency domain modeling features along the time domain dimension, and Obtain the network features corresponding to the i-1th separation unit stored in the cache unit, process the segmented second features and the network features based on the second LSTM network to obtain time-domain modeling features and updated network features; the updated network features are processed and stored by the cache unit to obtain the network features corresponding to the i-th separation unit for use by the time-domain modeling subunit in the i+1th separation unit when performing time-domain modeling; the causal attention refinement subunit in the i-th separation unit is used to extract the time-domain modeling features based on the attention weight and output them to the i+1th separation unit for processing to enhance the time-frequency features; wherein, i is a positive integer, and the network features include the hidden state features and cell state features of the second LSTM network;

[0011] The feature sequences corresponding to different output channels are decoded by a decoder to obtain a time domain speech signal corresponding to each output channel; wherein each output channel corresponds to a speech source.

[0012] In a possible implementation, processing the updated network features by the cache unit includes:

[0013] Performing temporal coding on the updated network features through a temporal coding network to obtain encoded network features;

[0014] The encoded network features are realigned to obtain the network features corresponding to the i-th separation unit.

[0015] In one possible implementation, the temporal encoding network includes a third LSTM network that temporally encodes hidden states and a fourth LSTM network that temporally encodes cell states.

[0016] In a possible implementation, the encoded network features are realigned to obtain the network features corresponding to the i-th separation unit, which is expressed by the following formula:

[0017]

[0018] in, Represents the lth hidden state feature in the network feature corresponding to the i-th separation unit, represents the lth cell state feature in the network feature corresponding to the i-th separation unit, h i ' ,l-1 represents the l-1th encoded hidden state feature corresponding to the i-th separation unit, c i ' ,l-1 represents the l-1th encoded cell state feature corresponding to the i-th separation unit, where l is a positive integer less than or equal to L, where L is the number of the second LSTM networks, and the value of L is determined based on the segmentation parameters when performing feature segmentation along the time domain dimension.

[0019] In one possible implementation, the causal attention refinement subunit includes the following network structure:

[0020] Multi-head attention network, used to extract multi-head high-dimensional features of temporal modeling features;

[0021] A first feature extraction path and a second feature extraction path are connected to and parallel to the multi-head attention network. The first feature extraction path is used to extract features from the multi-head high-dimensional features, and the second feature extraction path is used to perform gated selection on the features extracted by the first feature extraction path to obtain features after causal attention refinement.

[0022] In one possible implementation, the multi-head attention network has a diagonal mask structure to mask future frame information.

[0023] In a possible implementation, the first LSTM network and the second LSTM network are unidirectional LSTM networks to conceal future frame information.

[0024] In a possible implementation, the separation model further comprises:

[0025] a first feature mapping unit connected to the encoder, configured to map the time-frequency feature representation into a high-dimensional representation, and input the high-dimensional representation into the first separation unit; and

[0026] a second feature mapping unit connected to the last separation unit, configured to convert the causal attention refined features output by the last separation unit into real number representation and imaginary number representation corresponding to different output channels respectively, to obtain feature sequences corresponding to different output channels respectively, for decoding by the decoder;

[0027] The first feature mapping unit and the second feature mapping unit have a shielding layer to mask future frame information.

[0028] According to another aspect of the present disclosure, a speech separation device is provided, comprising a processor, a memory for storing processor-executable instructions, wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0029] According to another aspect of the present disclosure, a non-volatile computer readable storage medium is provided, which stores computer program instructions, wherein the computer program instructions are executed by a processor to implement the above method.

[0030] According to another aspect of the present disclosure, a computer program product is provided, comprising computer readable code or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in a processor of an electronic device, the processor in the electronic device executes the above method.

[0031] In the present application, a mixed signal to be separated is obtained; the mixed signal is converted into a time-frequency feature representation by an encoder; the time-frequency feature representation is separated by a separation model to obtain feature sequences corresponding to different output channels respectively; wherein the separation model comprises a buffer unit and at least one separation unit connected in sequence; each separation unit comprises a frequency domain modeling subunit, a time domain modeling subunit and a causal attention refinement subunit connected in sequence, and the time domain modeling subunit in each separation unit is connected to the buffer unit; the feature sequences corresponding to different output channels respectively are decoded by a decoder to obtain time domain speech signals corresponding to each output channel; wherein each output channel corresponds to a speech source; since the additional buffer unit can store historical information during global modeling, the separation model can process features based on historical information, which can solve the problem that causal models cannot effectively retain historical information when processing time series, resulting in poor speech separation performance, thereby improving the overall speech separation performance of the model.

[0032] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0034] Figure 1 A flowchart of a speech separation method according to an embodiment of the present disclosure is shown;

[0035] Figure 2 A schematic diagram showing a model structure of a frequency domain modeling subunit according to an embodiment of the present disclosure;

[0036] Figure 3 A schematic diagram showing a model structure of a time domain modeling subunit according to an embodiment of the present disclosure;

[0037] Figure 4 A schematic diagram showing a model structure of a causal attention refinement subunit according to an embodiment of the present disclosure;

[0038] Figure 5 A schematic diagram illustrating a speech separation process according to an embodiment of the present disclosure is shown;

[0039] Figure 6 A block diagram of a speech separation device according to an embodiment of the present disclosure is shown;

[0040] Figure 7 A block diagram of a speech separation device according to another embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0041] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0042] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0043] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0044] First, several terms involved in this application are introduced.

[0045] Short-time Fourier transform (STFT) is used to convert signals from the time domain to the frequency domain. The basic idea is to divide the signal into multiple small time windows, assume that the signal is stationary in each window, and then perform Fourier transform on the signal in each window.

[0046] Inverse Short-Time Fourier Transform (iSTFT): used to convert the signal from the frequency domain back to the time domain.

[0047] Long Short-Term Memory (LSTM) networks are a special type of recurrent neural network (RNN) capable of learning long-term dependencies. LSTM networks address the vanishing or exploding gradient problems encountered by traditional RNNs when processing long sequences of data by introducing a gating mechanism. These gating mechanisms, including a forget gate, an input gate, and an output gate, collectively control the flow of information within the network, allowing the LSTM to retain or discard information as necessary. Therefore, LSTM networks are well-suited for processing and predicting important events in time series data, particularly when long-term dependencies exist.

[0048] The network parameters of an LSTM network include the hidden state and the cell state. The hidden state can be represented by the feature vector H, and the cell state can be represented by the feature vector C. The hidden state H is the output of the LSTM network at each time step and can be viewed as the memory or representation of the LSTM network at that time step. The cell state C is used to store and transmit important information in the LSTM network and can remain unchanged between time steps, which enables the LSTM to learn long-term dependencies.

[0049] Time-Frequency Grid-like Neural Network (TF-GridNet): A non-causal model that typically includes an encoder, a separation model, and a decoder. The encoder is based on the STFT, and the separation model consists of multiple sequentially connected intra-frame spectral modules, sub-band temporal modules, and full-band self-attention modules.

[0050] Assume that for the speaker signals of C speakers acquired under echo-free conditions and noise signal The mixed signal obtained by mixing the speaker signal and the noise signal can be expressed as: in, L0 represents the length of the original signal.

[0051] Encoder: used to convert the mixed signal Y into time-frequency feature representation E∈R through STFT 2×F×T , where "2" represents the feature dimension. Since the features are mapped to a complex spectrum in the encoder, the time-frequency feature representation is obtained by superimposing the real and imaginary parts of the complex number. Therefore, the feature dimension is 2. F and T represent the number of frequency bands and the number of time domain steps of the time-frequency feature representation, respectively.

[0052] The encoder is generally connected to the separation model through a two-dimensional convolutional layer (Conv2D) and global normalization (Global Layer Normalization, gLN) to represent the time-frequency features E∈R 2×F×T Convert to high-dimensional feature representation.

[0053] Intra-frame spectrum module: This module processes the spectrum information within each time frame and uses a bidirectional long short-term memory (BiLSTM) network to model the local spectrum information within the frame. In this way, the module can capture local features within the spectrum domain.

[0054] The Subband Time Domain Module treats the input tensor obtained from the Intra Spectrum Module as multiple subbands, each containing time series data. It uses a BiLSTM to model the time domain information within each subband. This allows the module to capture local features within the time domain.

[0055] Full-band self-attention module: This module uses the self-attention mechanism to capture long-range global information across the entire frequency band. The self-attention mechanism enables the model to establish connections between different time frames, thereby capturing global time-frequency features.

[0056] The global time-frequency features output by the full-band self-attention module in the last separation model are output to the decoder for decoding after passing through the two-dimensional deconvolution layer (Deconv2D). The decoder reconstructs the processed features into the time domain signals corresponding to each output channel through iSTFT. The number of output channels is the same as the number of speakers C.

[0057] The above-mentioned TF-GridNet cannot retain historical information, so it performs poorly when processing real-time speech separation tasks. In this application, TF-GridNet is creatively improved to obtain a time-frequency domain attention cache memory model (Time-Frequency-Based Attention Cache Memory model, TF-ACM). TF-ACM can effectively retain historical information, solve the problem of information loss in long-term temporal processing, and improve the speech separation performance of the model. In this application, TF-ACM includes an encoder, a separation model, and a decoder, wherein the separation model of TF-ACM is different from the separation model of TF-GridNet. Below, the specific implementation method of TF-ACM is introduced.

[0058] Figure 1 The flowchart of the speech separation method according to an embodiment of the present disclosure is shown. In this embodiment, the method is described by taking the method as an example of an electronic device with computing capabilities such as a user terminal or a server, wherein the user terminal includes but is not limited to: a mobile phone, a computer, a tablet computer, etc. This embodiment does not limit the device types of the user terminal and the electronic device. Figure 1 As shown, the method includes:

[0059] Step 101: Obtain a mixed signal to be separated.

[0060] The mixed signal includes speech signals emitted by multiple speech sources. In some embodiments, the mixed signal also includes a noise signal. For example, the mixed signal is the in, represents the speech signal emitted by C speech sources; represents the noise signal, L0 represents the length of the original mixed signal, and C represents the number of speech sources. The speech source can be a real person or an audio playback device. This embodiment does not limit the type of speech source.

[0061] Optionally, the mixed signal may be collected in real time by the electronic device, or may be collected and sent in real time by other devices connected to the electronic device. In this embodiment, the mixed signal obtained in real time is used to perform speech separation, that is, perform a real-time speech separation task.

[0062] Step 102: Convert the mixed signal into a time-frequency feature representation through an encoder.

[0063] In one example, the encoder converts the mixed signal Y into a time-frequency feature representation through STFT, E∈R 2×F×T , where "2" represents the feature dimension, F and T represent the number of frequency bands and the number of time domain steps represented by the time-frequency feature, respectively.

[0064] Step 103, feature separation is performed on the time-frequency feature representation through the separation model to obtain feature sequences corresponding to different output channels; wherein the separation model includes a cache unit and at least one separation unit connected in sequence; each separation unit includes a frequency domain modeling subunit, a time domain modeling subunit and a causal attention refinement subunit connected in sequence, and the time domain modeling subunits in each separation unit are connected to the cache unit; for the i-th separation unit, the frequency domain modeling subunit in the i-th separation unit is used to perform feature segmentation on the input feature data along the frequency dimension, and perform frequency domain modeling on the first feature after segmentation based on the first LSTM network to obtain frequency domain modeling features; the time domain modeling subunit in the i-th separation unit The subunit is used to perform feature segmentation on the frequency domain modeling features along the time domain dimension, and obtain the network features corresponding to the i-1th separation unit stored in the cache unit, and process the segmented second features and network features based on the second LSTM network to obtain the time domain modeling features and the updated network features; the updated network features are processed and stored by the cache unit to obtain the network features corresponding to the i-th separation unit for use by the time domain modeling subunit in the i+1th separation unit when performing time domain modeling; the causal attention refinement subunit in the i-th separation unit is used to extract the time domain modeling features based on the attention weight and output them to the i+1th separation unit for processing to enhance the time-frequency features.

[0065] Where i is a positive integer, and the network features include the hidden state features and cell state features of the second LSTM network.

[0066] In this embodiment, the features output by the encoder are input to the separation model for processing. Each separation unit in the separation model processes the input features G in sequence. Each separation unit includes a frequency domain modeling subunit, a time domain modeling subunit and a causal attention refinement subunit. Specifically, G i Input into the i-th separation unit to obtain the causal attention refined feature G i+1 , G i+1 Input to the i+1th separation unit to obtain the causal attention refined feature G i+2 , ..., and so on, until the last separation unit is processed. These sub-units aggregate local and global time-frequency information, enhance the time-frequency representation, and improve the separation ability of the model.

[0067] Below, the frequency domain modeling subunit (F-Local), time domain modeling subunit (T-Local) and causal attention refinement subunit (CAR) in the same separation unit are introduced respectively.

[0068] 1. Reference Figure 2 The frequency domain modeling subunit in each separation unit shown, assuming that the characteristic data input by the frequency domain modeling subunit is G i ∈R N×F×T , indicating that the feature data input to the frequency domain modeling subunit in the i-th separation unit is a three-dimensional real number matrix of N×F×T, with the sizes of each dimension being N, F, and T, where N represents the number of feature channels, and F and T represent the number of frequency bands and the number of time domain steps represented by the time-frequency feature, respectively.

[0069] For the frequency domain modeling subunit in the first separation unit, its input feature data is obtained by processing the time-frequency feature representation output by the encoder. The specific processing method can be high-dimensional mapping through the first feature mapping unit below. In other embodiments, the processing method can also be other methods, which are not listed one by one in this embodiment. For the frequency domain modeling subunit in each separation unit after the first separation unit, its input feature data is the feature output by the causal attention refinement subunit in the previous separation unit, that is, the causal attention refinement subunit in each separation unit is connected to the frequency domain modeling subunit in the next separation unit, so that each separation unit is connected in sequence.

[0070] Accordingly, in the frequency domain modeling subunit, the input feature data is segmented along the frequency dimension, including: segmenting the feature data G according to the frequency domain dimension using a fixed width W1 and a stride S1 through an expansion operation. i ∈R N×F×T , combined with the folding process, the feature data G i Converted to sub-band features (i.e. the first feature after segmentation) in,

[0071] Accordingly, frequency domain modeling is performed on the first feature after segmentation based on the first LSTM network to obtain frequency domain modeling features, including: performing layer normalization (LN) processing on the first feature after segmentation, inputting the normalized features into the first LSTM network (i.e. Figure 2 LSTM-F) to process the historical information interaction between different frequency sub-bands and obtain the interaction features The interaction features are integrated across sub-bands through the feature integration network to obtain the frequency domain modeling features.

[0072] Here, H represents the hidden layer dimension of the first LSTM network, or the number of units in the LSTM network layer, that is, the number of LSTM units in the LSTM network that can process one time step in the sequence data. At this time, in the first LSTM network, the feature dimension of the feature data is W1×N, and the sequence length is L F .

[0073] Schematically, the feature integration network can be implemented by one-dimensional causal deconvolution (CDeconv).

[0074] Schematically, layer normalization can be achieved through a two-dimensional causal convolution (CConv) layer and a normalization layer.

[0075] In this embodiment, the first LSTM network is a unidirectional LSTM network to mask future frame information, thereby ensuring that the model does not utilize future frame information after the current time frame at all, so as to implement the model as a causal model.

[0076] 2. Reference Figure 3 The time domain modeling subunit in each separation unit shown is assumed to input characteristic data of the time domain modeling subunit as the frequency domain modeling feature output by the frequency domain modeling subunit. N represents the number of feature channels, F and T represent the number of frequency bands and the number of time domain steps represented by the time-frequency feature, respectively.

[0077] Accordingly, in the time domain modeling subunit, the frequency domain modeling features are segmented along the time domain dimension, including: splitting the frequency domain modeling features along the time domain dimension using a fixed width W2 and a stride S2 through an expansion operation. Combined with the folding process, the frequency domain modeling feature G i Converted to sub-band features (i.e. the second feature after segmentation) in,

[0078] Accordingly, the second feature after segmentation and the network feature are processed based on the second LSTM network to obtain the time domain modeling feature and the updated network feature, including: and network characteristics Input the second LSTM network (i.e. Figure 3 LSTM-T in ), and obtain the time domain modeling features and the updated network features (H i ,C i ).

[0079] Network characteristics Model the network characteristics corresponding to the subunits in the time domain of the previous separation unit.

[0080] In the second LSTM network, each feature data has a feature dimension of N, a sequence length of W2, and F features at each time step. At this time, the actual composition of the second LSTM network is equivalent to an L LSTM with shared parameters, and different LSTMs have the same parameters and process a sequence segment with a sequence length of W2.

[0081] In this embodiment, the second LSTM network is a unidirectional LSTM network to mask future frame information, so that the model can not use future frame information after the current time frame at all, and thus can be implemented as a causal model.

[0082] Correspondingly, the cache unit obtains the updated network feature (H i ,C i ), processes the updated network feature, and obtains the network feature corresponding to the i-th separation unit.

[0083] In this embodiment, in order to solve the problem of information loss in long time dimension T, a cache unit is proposed. The cache unit is a shared module independent of the plurality of separation units. In an example, the cache unit processes the updated network feature, including: performing time sequence encoding on the updated network feature through a time sequence encoding network to obtain an encoded network feature; and performing realignment processing on the encoded network feature to obtain the network feature corresponding to the i-th separation unit.

[0084] Optionally, as shown in Figure 3 , the time sequence encoding network includes a third LSTM network (i.e., LSTM-H in Figure 3 ) for time sequence encoding of the updated network feature and a fourth LSTM network (i.e., LSTM-C in Figure 3 ) for time sequence encoding of the cell state, so as to aggregate and process the historical information interaction across modules by using the updated network feature. At this time, for each updated network feature of the i-th separation unit, the time sequence encoding process can be represented by the following formula:

[0085] (H i ′,C i ′)=CacheMemory(H i ,C i )

[0086] wherein CacheMemory represents encoding H i using the third LSTM network (i.e., LSTM-H) and encoding C i using the fourth LSTM network (i.e., LSTM-C). By reprocessing the updated network feature, it is helpful to save and enhance the time information.

[0087] In other embodiments, the temporal coding network may also be a temporal network such as a Transformer network. This embodiment does not limit the implementation method of the temporal coding network.

[0088] Since the hidden layer parameters of each second LSTM network encapsulate the complete network feature information of its corresponding time period, in order to ensure (H i ′,C i ′) When it is transmitted to the (i+1)th separation module, no information leakage occurs. In this embodiment, the encoded network features need to be realigned. Specifically, the realignment of the encoded network features can be expressed by the following formula:

[0089]

[0090] in, Represents the lth hidden state feature in the network feature corresponding to the i-th separation unit, represents the lth cell state feature in the network feature corresponding to the i-th separation unit, h i ' ,l-1 represents the l-1th encoded hidden state feature corresponding to the i-th separation unit, c i ' ,l-1 represents the l-1th encoded cell state feature corresponding to the i-th separation unit, l is a positive integer less than or equal to L, L is the number of second LSTM networks, and the value of L is determined based on the segmentation parameters when performing feature segmentation along the time domain dimension.

[0091] Afterwards, the network features corresponding to the ith separation unit composed of each hidden state feature and cell state feature are The summary of the global information is returned to the second LSTM network in the i+1th separation module so that the second LSTM network in the i+1th separation module can perform the same time domain modeling processing as the second LSTM network in the i-th separation module. The specific process is described above and will not be repeated in this embodiment.

[0092] 3. Reference Figure 4 The causal attention refinement subunit in each separation unit shown in the figure includes the following network structure: a multi-head attention network for extracting multi-head high-dimensional features of temporal modeling features; a first feature extraction path and a second feature extraction path connected to and parallel to the multi-head attention network. The first feature extraction path is used to extract features from the multi-head high-dimensional features, and the second feature extraction path is used to perform gated selection on the features extracted by the first feature extraction path to obtain features after causal attention refinement.

[0093] In one example, a multi-head attention network has a diagonal mask structure to mask out future frame information.

[0094] Specifically, the causal attention refinement subunit extracts features from the temporal modeling features based on the attention weights, including:

[0095] Input the time domain modeling features into the feature extraction network in the multi-head attention network to obtain the high-dimensional features Q∈R of the single-head attention network N×F×T , K∈R N×F×T and V∈R N×F×T ; The attention output value is determined by combining the attention mechanism in the multi-head attention network with a diagonal mask; the cross-head information is aggregated through two-dimensional CConv, parametric rectified linear unit (PReLU) and LN in turn to obtain multi-head high-dimensional features.

[0096] Schematically, the feature extraction network is implemented by two-dimensional CConv and LN.

[0097] Optionally, the first feature extraction path and the second feature extraction path respectively include one-dimensional point-by-point dot products and one-dimensional depth-wise dot products connected in sequence, so that fine-grained and broader contextual features can be effectively captured, thereby updating the overall representation of the feature vector before entering the next processing stage and improving the feature extraction effect.

[0098] Gated selection refers to the process of generating a gating signal through an activation function to select some features from the features extracted by the first feature extraction path.

[0099] In one example, there is one second feature extraction path, and gated selection is enabled through an activation function. The gated selected features are multiplied by the features obtained by the first feature extraction path. After that, the causal attention refined features are obtained through one-dimensional point-by-point convolution for subsequent processing, such as outputting to the next separation unit for processing.

[0100] In other embodiments, there may be at least one second feature extraction path, i.e., one or at least two. Different second feature extraction paths enable gated selection through activation functions, and are added to the features obtained by the first feature extraction path to obtain gated features. Subsequently, one-dimensional point-by-point convolution is performed to obtain causal attention refined features for subsequent processing. This embodiment does not limit the implementation method of the second feature extraction path.

[0101] After the above-mentioned separation units complete feature processing in sequence, the causal attention refined features output by the last causal attention refinement sub-unit are processed to obtain feature sequences corresponding to different output channels of the final output of the separation model. Schematically, the processing method of the causal attention refined features output by the last causal attention refinement sub-unit can be: the causal attention refined features are converted into real number representations and imaginary number representations corresponding to different output channels respectively through the second feature mapping unit described below. In other embodiments, the processing method can also be other methods, which are not listed one by one in this embodiment. Schematically, the separation model also includes: a first feature mapping unit connected to the encoder, the first feature mapping unit is used to map the time-frequency feature representation into a high-dimensional representation, and input the high-dimensional representation into the first separation unit; and a second feature mapping unit connected to the last separation unit, the second feature mapping unit is used to convert the causal attention refined features output by the last separation unit into real number representations and imaginary number representations corresponding to different output channels respectively, to obtain feature sequences corresponding to different output channels respectively, for decoding by the decoder.

[0102] The first feature mapping unit and the second feature mapping unit have a shielding layer to cover up future frame information.

[0103] Schematically, the first feature mapping unit is realized by a combination of sequentially connected two-dimensional CConv layers and LNs. At this time, the time-frequency feature representation E∈R 2×F×T Mapping to a high-dimensional representation G∈R N×F×T At this time, the high-dimensional representation is used as the initial condition G0 and directly processed as the input of the first separation unit. Where N represents the number of feature channels, F and T represent the number of frequency bands and the number of time domain steps of the time-frequency feature representation, respectively.

[0104] The second feature mapping unit is implemented by a two-dimensional CDeconv layer with 2×C output channels and a 3×3 convolution kernel, and then by a rectified linear unit (ReLU) activation function. At this time, the causal attention refined features output by the last separation unit can be converted into predicted real and imaginary representations S∈R C×2×F×T . Where C represents the number of speech sources.

[0105] Step 104 : Decode the feature sequences corresponding to different output channels through a decoder to obtain a time domain speech signal corresponding to each output channel; wherein each output channel corresponds to a speech source.

[0106] In one example, the decoder decodes the feature sequences corresponding to different output channels through iSTFT to obtain the time domain speech signal corresponding to each output channel. Among them, c belongs to [1, C], C represents the number of speech sources, L0 represents the length of the original mixed signal, and at this time, the length of the time domain speech signal obtained by speech separation is consistent with the length of the original mixed signal.

[0107] To summarize, the speech separation method provided in this embodiment obtains a mixed signal to be separated; converts the mixed signal into a time-frequency feature representation through an encoder; performs feature separation on the time-frequency feature representation through a separation model to obtain feature sequences corresponding to different output channels; wherein, the separation model includes a cache unit and at least one separation unit connected in sequence; each separation unit includes a frequency domain modeling subunit, a time domain modeling subunit and a causal attention refinement subunit connected in sequence, and the time domain modeling subunits in each separation unit are connected to the cache unit; the feature sequences corresponding to different output channels are decoded through a decoder to obtain a time domain speech signal corresponding to each output channel; wherein, each output channel corresponds to a speech source; since the additional cache unit can store historical information during global modeling, the separation model can perform feature processing based on historical information, which can solve the problem that the causal model cannot effectively retain historical information when processing time series, resulting in poor speech separation performance, thereby improving the overall speech separation performance of the model.

[0108] In addition, by realigning the encoded network features in the buffer unit, the problem of information leakage caused by sharing network parameters across separation units can be avoided, ensuring that information leakage does not occur during the transmission process of network parameters.

[0109] In addition, by temporally encoding the updated network features through the temporal coding network, it is possible to aggregate and process historical information interactions across modules and improve the global representation performance of the features.

[0110] In addition, by connecting the first feature extraction path and the second feature extraction path in parallel in the causal attention refinement sub-unit, the second feature extraction path can perform gated selection on the features extracted by the first feature extraction path, which can effectively capture fine-grained and broader contextual features and improve the feature extraction effect.

[0111] In addition, by setting a shielding layer in the first feature mapping unit and the second feature mapping unit, setting a diagonal mask structure in the multi-head attention network, and setting the first LSTM network and the second LSTM network to a unidirectional LSTM network, the separation model can mask future frame information during the processing process and implement the model as a causal model.

[0112] In order to more clearly understand the speech separation process provided by this application, the following example is used to illustrate the process. Figure 5 , the process includes the following steps:

[0113] Step 51, input the mixed signal Y to be separated into the encoder to obtain the time-frequency feature representation E.

[0114] Step 52, input the time-frequency feature representation into the separation model, and obtain the high-dimensional representation G through the two-dimensional CConv layer and the LN (i.e., the first feature mapping unit) connected in sequence; after the high-dimensional representation G is processed through B separation units, the two-dimensional CDeconv layer and the ReLU activation function (i.e., the second feature mapping unit), the feature sequence S corresponding to different output channels is obtained;

[0115] Among them, each time-domain modeling subunit in the B separation units is connected with the cache unit, and B is a positive integer.

[0116] Step 53, the feature sequence S corresponding to different output channels is processed through the decoder corresponding to each output channel to obtain the time-domain speech signal corresponding to the output channel Figure 5 In the middle That is, C=2 is taken as an example for illustration.

[0117] The speech separation method proposed in the present application exceeds most of the existing methods in the evaluation index. Next, the following settings are used, and the signal-to-interference ratio improvement (SI-SNRi) and the signal-to-noise ratio improvement (SNRi) are used as the evaluation standard of the speech separation performance of the speech separation method proposed in the present application.

[0118] Among them, SI-SNRi is obtained by calculating the difference between the signal-to-interference ratio (SIR) of the processed signal and the SIR of the original signal, and SI-SNRi can be used to evaluate the de-interference ability of the system. SNRi is obtained by calculating the difference between the signal-to-noise ratio (SNR) of the processed signal and the SNR of the original signal, and SNRi can be used to evaluate the noise reduction ability of the system. The larger the values of SI-SNRi and SNRi, the better the performance of the model.

[0119] Assume that the STFT uses a Hann window, with the window size and hop size set to 64 and 8, respectively; the number of feature channels N is set to 128, B = 3; the number of hidden units H in each LSTM network layer within the separation unit is set to 64; and in the CAR module, the number of attention heads is set to 2. Table 1 shows the speech separation results of this application on different datasets, specifically the causal separation performance on the WHAM!, WHAMR!, and Libri2Mix datasets, and uses one second of 8kHz resolution audio to calculate the average inference time on the CPU. As shown in Table 1, the performance of this application is quantitatively compared with existing models: Dual-Path Recurrent Neural Network (DPRNN), SKiM, TF GridNet, and Reversible Separator Feedforward Network (ReSepformer). As shown in Table 1, TFACM outperforms existing models in all indicators, with an improvement of approximately 1.8dB. In addition, TFACM has fewer parameters than all existing causal speech separation models, which can reduce the computing resources occupied by electronic devices.

[0120] Table 1:

[0121]

[0122] Furthermore, this application also evaluates the effectiveness of the cache unit (CM) and the causal attention refinement subunit (CAR) on the WHAM! dataset. For the cache unit, the LSTM-H and LSTM-T modules therein are replaced with misplaced congruent mappings for comparison with this application, effectively passing the state of the current separation unit directly to the next separation unit. For the causal attention refinement subunit, the gated convolution is completely removed for comparison with this application. The results of these ablation studies are shown in Table 2 below. We found that deleting the cache unit will lead to a decline in overall performance, which may be due to the model losing the ability to process internal information, greatly weakening its processing of time information. In addition, removing the causal attention refinement subunit will also lead to a significant decline in performance, mainly because the causal attention refinement subunit provides coarse-grained global time-frequency perception by integrating separately processed frequency and time features. Therefore, it can be seen that incorporating CM and CAR modules into the separation model can effectively improve model performance and improve speech separation results.

[0123] Table 2:

[0124] CM CAR SDRi SI-SNRi exist exist 13.3 13.0 does not exist exist 11.8 11.4 exist does not exist 11.8 11.5 does not exist does not exist 11.2 10.7

[0125] Figure 6 A block diagram of a speech separation device according to an embodiment of the present disclosure is shown. Figure 6 It can be seen that the device includes: a signal acquisition module 610, a signal encoding module 620, a feature separation module 630 and a signal decoding module 640.

[0126] A signal acquisition module 610 is used to acquire a mixed signal to be separated;

[0127] a signal encoding module 620, configured to convert the mixed signal into a time-frequency feature representation through an encoder;

[0128] The feature separation module 630 is used to perform feature separation on the time-frequency feature representation through a separation model to obtain feature sequences corresponding to different output channels; wherein, the separation model includes a cache unit and at least one separation unit connected in sequence; each separation unit includes a frequency domain modeling subunit, a time domain modeling subunit and a causal attention refinement subunit connected in sequence, and the time domain modeling subunits in each separation unit are connected to the cache unit; for the i-th separation unit, the frequency domain modeling subunit in the i-th separation unit is used to perform feature segmentation on the input feature data along the frequency dimension, and perform frequency domain modeling on the first feature after segmentation based on the first LSTM network to obtain frequency domain modeling features; the time domain modeling subunit in the i-th separation unit is used to perform frequency domain modeling on the frequency domain modeling features along the time domain dimension Feature segmentation, and obtaining the network features corresponding to the i-1th separation unit stored in the cache unit, processing the segmented second features and the network features based on the second LSTM network to obtain time domain modeling features and updated network features; the updated network features are processed and stored by the cache unit to obtain the network features corresponding to the i-th separation unit for use by the time domain modeling subunit in the i+1th separation unit when performing time domain modeling; the causal attention refinement subunit in the i-th separation unit is used to extract the time domain modeling features based on the attention weight and output them to the i+1th separation unit for processing to enhance the time-frequency features; wherein, i is a positive integer, and the network features include the hidden state features and cell state features of the second LSTM network;

[0129] The signal decoding module 640 is used to decode the feature sequences corresponding to different output channels through a decoder to obtain a time domain speech signal corresponding to each output channel; wherein each output channel corresponds to a speech source.

[0130] For relevant details, please refer to the above method embodiment.

[0131] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0132] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0133] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0134] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0135] Figure 7 1 is a block diagram of a device 1900 for speech separation according to an exemplary embodiment. For example, the device 1900 can be provided as a server or a terminal device. Figure 7 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0136] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0137] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the apparatus 1900 to perform the above-described method.

[0138] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A speech separation method, characterized in that: The method comprises: Acquire a mixed signal to be separated; Converting the mixed signal into a time-frequency feature representation through an encoder; The time-frequency feature representation is feature separated by a separation model to obtain feature sequences corresponding to different output channels; wherein the separation model includes a cache unit and at least one separation unit connected in sequence; each separation unit includes a frequency domain modeling subunit, a time domain modeling subunit and a causal attention refinement subunit connected in sequence, and the time domain modeling subunits in each separation unit are connected to the cache unit; for the i-th separation unit, the frequency domain modeling subunit in the i-th separation unit is used to perform feature segmentation on the input feature data along the frequency dimension, and perform frequency domain modeling on the first feature after segmentation based on the first LSTM network to obtain frequency domain modeling features; the time domain modeling subunit in the i-th separation unit is used to perform feature segmentation on the frequency domain modeling features along the time domain dimension, and Obtain the network features corresponding to the i-1th separation unit stored in the cache unit, process the segmented second features and the network features based on the second LSTM network to obtain time-domain modeling features and updated network features; the updated network features are processed and stored by the cache unit to obtain the network features corresponding to the i-th separation unit for use by the time-domain modeling subunit in the i+1th separation unit when performing time-domain modeling; the causal attention refinement subunit in the i-th separation unit is used to extract the time-domain modeling features based on the attention weight and output them to the i+1th separation unit for processing to enhance the time-frequency features; wherein, i is a positive integer, and the network features include the hidden state features and cell state features of the second LSTM network; The feature sequences corresponding to different output channels are decoded by a decoder to obtain a time domain speech signal corresponding to each output channel; wherein each output channel corresponds to a speech source.

2. The method according to claim 1, characterized in that The updated network features are processed by the cache unit, including: Performing temporal coding on the updated network features through a temporal coding network to obtain encoded network features; The encoded network features are realigned to obtain the network features corresponding to the i-th separation unit.

3. The method according to claim 2, characterized in that The temporal coding network includes a third LSTM network that temporally encodes hidden states and a fourth LSTM network that temporally encodes cell states.

4. The method according to claim 2, characterized in that The network features after encoding are realigned to obtain the network features corresponding to the i-th separation unit, which is expressed by the following formula: in, Represents the lth hidden state feature in the network feature corresponding to the i-th separation unit, represents the lth cell state feature in the network feature corresponding to the i-th separation unit, h i ′ ,l-1 represents the l-1th encoded hidden state feature corresponding to the i-th separation unit, c i ′ ,l-1 represents the l-1th encoded cell state feature corresponding to the i-th separation unit, l is a positive integer less than or equal to L, and L is the number of the second LSTM networks. The value of L is determined based on the segmentation parameters when performing feature segmentation along the time domain dimension.

5. The method according to claim 1, wherein The causal attention refinement subunit includes the following network structure: Multi-head attention network, used to extract multi-head high-dimensional features of temporal modeling features; A first feature extraction path and a second feature extraction path are connected to and parallel to the multi-head attention network. The first feature extraction path is used to extract features from the multi-head high-dimensional features, and the second feature extraction path is used to perform gated selection on the features extracted by the first feature extraction path to obtain features after causal attention refinement.

6. The method according to claim 5, characterized in that The multi-head attention network has a diagonal mask structure to mask future frame information.

7. The method according to any one of claims 1 to 6, characterized in that: The first LSTM network and the second LSTM network are unidirectional LSTM networks to conceal future frame information.

8. The method according to any one of claims 1 to 6, characterized in that: The separation model also includes: a first feature mapping unit connected to the encoder, configured to map the time-frequency feature representation into a high-dimensional representation, and input the high-dimensional representation into a first separation unit; and a second feature mapping unit connected to the last separation unit, configured to convert the causal attention refined features output by the last separation unit into real number representations and imaginary number representations corresponding to different output channels, respectively, to obtain feature sequences corresponding to different output channels, for decoding by the decoder; The first feature mapping unit and the second feature mapping unit have a shielding layer to cover up future frame information.

9. A speech separation device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method according to any one of claims 1 to 8 when executing the instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images