Voice separation method and device and storage medium

Abstract

The invention relates to the technical field of computers, and discloses a voice separation method and device and a storage medium. The method comprises the steps of obtaining a to-be-separated mixed signal; converting the mixed signal into time-frequency characteristic representation through an encoder; performing feature separation on the time-frequency feature representation through a separation model to obtain feature sequences corresponding to different output channels; the separation model comprises a cache unit and at least one separation unit connected in sequence; each separation unit comprises a frequency domain modeling subunit, a time domain modeling subunit and a causal attention refining subunit which are connected in sequence, and the time domain modeling subunit in each separation unit is connected with the cache unit; decoding the feature sequences corresponding to different output channels through a decoder to obtain a time domain voice signal corresponding to each output channel; the additionally arranged cache unit can store historical information during global modeling, so that the separation model can perform feature processing based on the historical information, and the overall voice separation performance of the model can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a speech separation method, device and storage medium. Background Art

[0002] Speech recognition technology completes the speech recognition task by inputting the speech signal obtained by the front end into the processing back end. At present, the more advanced voice assistants, that is, the mainstream speech recognition methods, have relatively high requirements for the environment of the input speech scene. In actual speech recognition scenarios, the environment of the input speech signal is often noisy or contains multiple speakers, which will affect the accuracy of speech recognition. Based on this, speech separation technology can be used to separate the speech signals of different speakers from the mixed speech of multiple speakers, and then the separated single speaker signal is input into the processing back end for recognition to improve the recognition accuracy. Speech separation technology is crucial to the development of fields such as automatic conference transcription, automatic subtitles of audio / video, and multi-party human-computer interaction.

[0003] In recent years, with the rapid development of deep learning technology, different deep learning models have emerged to improve the performance of speech separation. Traditional speech separation tasks are generally divided into two categories: offline speech separation tasks and real-time speech separation tasks. Among them, offline speech separation tasks are widely studied because they have more comprehensive information and better results. Generally, speech separation models used to process offline speech separation tasks are also called non-causal models. Correspondingly, real-time speech separation tasks, such as conference transcription, require real-time speech separation. Future speech information cannot be used for speech separation at the current moment. Generally, models used to process real-time speech separation tasks are also called causal models. Causal models require the system to obtain and process information only from the current and previous time frames. Usually causal models lag behind non-causal models in performance.

[0004] In order to narrow the performance gap between causal models and non-causal speech separation models, causal models usually need to consider extending the look-ahead window or implementing a joint training strategy. For example, based on the Skipping Memory LSTM (SKiM) or Parallel Skipping Memory LSTM (pSKiM) network, past audio feature representations are extracted to integrate historical hidden layer states to enhance the causal model, thereby reducing latency and improving efficiency.

[0005] However, current causal models cannot effectively retain historical information when processing time series, resulting in poor speech separation performance of causal models. Summary of the invention

[0006] In view of this, the present disclosure proposes a speech separation method, device and storage medium, which can store historical information during global modeling by additionally setting a cache unit in the separation model, so that the separation model can perform feature processing based on the historical information, thereby improving the overall speech separation performance of the model.

[0007] According to one aspect of the present disclosure, a speech separation method is provided, the method comprising:

[0008] Acquire a mixed signal to be separated;

[0009] Converting the mixed signal into a time-frequency feature representation through an encoder;

[0010] The time-frequency feature representation is feature separated by a separation model to obtain feature sequences corresponding to different output channels; wherein the separation model includes a cache unit and at least one separation unit connected in sequence; each separation unit includes a frequency domain modeling subunit, a time domain modeling subunit and a causal attention refinement subunit connected in sequence, and the time domain modeling subunits in each separation unit are connected to the cache unit; for the i-th separation unit, the frequency domain modeling subunit in the i-th separation unit is used to perform feature segmentation on the input feature data along the frequency dimension, and perform frequency domain modeling on the first feature after segmentation based on the first LSTM network to obtain frequency domain modeling features; the time domain modeling subunit in the i-th separation unit is used to perform feature segmentation on the frequency domain modeling features along the time domain dimension, and The network feature corresponding to the i-1th separation unit stored in the cache unit is obtained, and the segmented second feature and the network feature are processed based on the second LSTM network to obtain the time domain modeling feature and the updated network feature; the updated network feature is processed and stored by the cache unit to obtain the network feature corresponding to the i-th separation unit for use by the time domain modeling subunit in the i+1th separation unit when performing time domain modeling; the causal attention refinement subunit in the i-th separation unit is used to extract the time domain modeling feature based on the attention weight and output it to the i+1th separation unit for processing to enhance the time-frequency feature; wherein, i is a positive integer, and the network feature includes the hidden state feature and the cell state feature of the second LSTM network;

[0011] The feature sequences corresponding to different output channels are decoded by a decoder to obtain a time domain speech signal corresponding to each output channel; wherein each output channel corresponds to a speech source.

[0012] In a possible implementation manner, the updated network feature is processed by the cache unit, including:

[0013] Performing temporal coding on the updated network features through a temporal coding network to obtain encoded network features;

[0014] The encoded network features are realigned to obtain the network features corresponding to the i-th separation unit.

[0015] In a possible implementation, the temporal encoding network includes a third LSTM network that temporally encodes hidden states and a fourth LSTM network that temporally encodes cell states.

[0016] In a possible implementation, the encoded network features are realigned to obtain the network features corresponding to the i-th separation unit, which is expressed by the following formula:

[0017]

[0018] in, represents the lth hidden state feature in the network feature corresponding to the i-th separation unit, represents the lth cell state feature in the network feature corresponding to the i-th separation unit, h i ' ,l-1 represents the l-1th encoded hidden state feature corresponding to the ith separation unit, c i ' ,l-1 represents the l-1th encoded cell state feature corresponding to the i-th separation unit, l is a positive integer less than or equal to L, L is the number of the second LSTM networks, and the value of L is determined based on the segmentation parameters when performing feature segmentation along the time domain dimension.

[0019] In a possible implementation, the causal attention refinement subunit includes the following network structure:

[0020] Multi-head attention network, used to extract multi-head high-dimensional features of temporal modeling features;

[0021] A first feature extraction path and a second feature extraction path are connected to and parallel to the multi-head attention network, the first feature extraction path is used to extract features from the multi-head high-dimensional features, and the second feature extraction path is used to perform gated selection on the features extracted by the first feature extraction path to obtain features after causal attention refinement.

[0022] In one possible implementation, the multi-head attention network has a diagonal mask structure to mask future frame information.

[0023] In a possible implementation, the first LSTM network and the second LSTM network are unidirectional LSTM networks to conceal future frame information.

[0024] In a possible implementation, the separation model further includes:

[0025] a first feature mapping unit connected to the encoder, used to map the time-frequency feature representation into a high-dimensional representation, and input the high-dimensional representation into a first separation unit; and

[0026] A second feature mapping unit connected to the last separation unit is used to convert the causal attention refined features output by the last separation unit into real number representations and imaginary number representations corresponding to different output channels, respectively, to obtain feature sequences corresponding to different output channels, for decoding by the decoder;

[0027] The first feature mapping unit and the second feature mapping unit have a shielding layer to cover up future frame information.

[0028] According to another aspect of the present disclosure, a speech separation device is provided, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0029] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0030] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0031] In the present application, a mixed signal to be separated is obtained; the mixed signal is converted into a time-frequency feature representation through an encoder; the time-frequency feature representation is feature separated by a separation model to obtain feature sequences corresponding to different output channels; wherein the separation model includes a cache unit and at least one separation unit connected in sequence; each separation unit includes a frequency domain modeling subunit, a time domain modeling subunit and a causal attention refinement subunit connected in sequence, and the time domain modeling subunits in each separation unit are connected to the cache unit; the feature sequences corresponding to different output channels are decoded by a decoder to obtain a time domain speech signal corresponding to each output channel; wherein each output channel corresponds to a speech source; since the additional cache unit can store historical information during global modeling, the separation model can perform feature processing based on historical information, which can solve the problem that the causal model cannot effectively retain historical information when processing time series, resulting in poor speech separation performance, thereby improving the overall speech separation performance of the model.

[0032] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0034] Figure 1 A flow chart showing a method for speech separation according to an embodiment of the present disclosure is shown;

[0035] Figure 2 A schematic diagram showing a model structure of a frequency domain modeling subunit according to an embodiment of the present disclosure;

[0036] Figure 3 A schematic diagram showing a model structure of a time domain modeling subunit according to an embodiment of the present disclosure;

[0037] Figure 4 A schematic diagram showing a model structure of a causal attention refinement subunit according to an embodiment of the present disclosure;

[0038] Figure 5 A schematic diagram showing a speech separation process according to an embodiment of the present disclosure;

[0039] Figure 6 A block diagram of a speech separation device according to an embodiment of the present disclosure is shown;

[0040] Figure 7 A block diagram of a speech separation device according to another embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0041] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0042] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0043] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0044] First, several terms involved in this application are introduced.

[0045] Short-time Fourier transform (STFT) is used to convert signals from time domain to frequency domain. The basic idea is to divide the signal into multiple small time windows, assume that the signal is stationary in each window, and then perform Fourier transform on the signal in each window.

[0046] Inverse Short-Time Fourier Transform (iSTFT): used to convert the signal from the frequency domain back to the time domain.

[0047] Long Short-Term Memory Network (LSTM Network): is a special type of Recurrent Neural Network (RNN) that can learn long-term dependencies. LSTM networks solve the gradient vanishing or exploding problems encountered by traditional RNNs when processing long sequence data by introducing a gating mechanism. The gating mechanism includes a forget gate, an input gate, and an output gate, which together control the flow of information in the network, allowing LSTM to retain or discard information when necessary. Therefore, LSTM networks are suitable for processing and predicting important events in time series data, especially when there are long-term dependencies.

[0048] The network parameters of the LSTM network include hidden state and cell state, where the hidden state can be represented by the feature vector H and the cell state can be represented by the feature vector C. The hidden state H is the output of the LSTM network at each time step, and H can be regarded as the memory or representation of the LSTM network at that time step. The cell state C is used to store and transfer important information in the LSTM network and can remain unchanged between time steps, which enables LSTM to learn long-term dependencies.

[0049] Time-Frequency Grid-like Neural Network (TF-GridNet): A non-causal model that usually includes an encoder, a separation model, and a decoder. The encoder is based on STFT, and the separation model includes multiple sequentially connected intra-frame spectral modules, sub-band temporal modules, and full-band self-attention modules.

[0050] Assume that for the speaker signals of C speakers acquired under echo-free conditions and noise signal The mixed signal obtained by mixing the speaker signal and the noise signal can be expressed as: in, L0 represents the length of the original signal.

[0051] Encoder: used to convert the mixed signal Y into a time-frequency feature representation E∈R through STFT 2×F×T , where "2" represents the feature dimension. Since the feature is mapped to a complex spectrum in the encoder, the real part and the imaginary part of the complex number are superimposed to obtain the time-frequency feature representation. Therefore, the feature dimension is 2. F and T represent the number of frequency bands and the number of time domain steps of the time-frequency feature representation, respectively.

[0052] The encoder is usually connected to the separation model through a two-dimensional convolutional layer (Conv2D) and global normalization (Global LayerNormalization, gLN) to represent the time-frequency features E∈R 2×F×T Convert to high-dimensional feature representation.

[0053] Intra-frame spectrum module: It is used to process the spectrum information within each time frame and uses a bidirectional long short-term memory network (BiLSTM) to model the local spectrum information within the frame. In this way, the module can capture the local features in the spectrum domain.

[0054] Subband time domain module: It is used to treat the input tensor obtained from the intra-frame spectrum module as multiple subbands, each of which contains time series data. BiLSTM is used to model the time domain information in each subband. In this way, the module can capture local features in the time domain.

[0055] Full-band self-attention module: It is used to capture long-range global information in the entire frequency band using the self-attention mechanism. The self-attention mechanism enables the model to establish connections between different time frames, thereby capturing global time-frequency features.

[0056] The global time-frequency features output by the full-band self-attention module in the last separation model are output to the decoder for decoding after passing through the two-dimensional deconvolution layer (Deconv2D). The decoder reconstructs the processed features into time domain signals corresponding to each output channel through iSTFT. The number of output channels is the same as the number of speakers C.

[0057] The above-mentioned TF-GridNet cannot retain historical information, so it performs poorly when processing real-time speech separation tasks. In this application, TF-GridNet is creatively improved to obtain a time-frequency domain attention cache memory model (Time-Frequency-Based Attention Cache Memory model, TF-ACM). TF-ACM can effectively retain historical information, solve the problem of information loss in long-term temporal processing, and improve the speech separation performance of the model. In this application, TF-ACM includes an encoder, a separation model and a decoder, wherein the separation model of TF-ACM is different from the separation model of TF-GridNet. Below, the specific implementation method of TF-ACM is introduced.

[0058] Figure 1 The flowchart of the speech separation method according to an embodiment of the present disclosure is shown. In this embodiment, the method is described by taking the method used in an electronic device with computing capabilities such as a user terminal or a server as an example, wherein the user terminal includes but is not limited to: a mobile phone, a computer, a tablet computer, etc., and this embodiment does not limit the device types of the user terminal and the electronic device. Figure 1 As shown, the method includes:

[0059] Step 101: Obtain a mixed signal to be separated.

[0060] The mixed signal includes speech signals emitted by multiple speech sources. In some embodiments, the mixed signal also includes a noise signal. in, represents the speech signal emitted by C speech sources; represents a noise signal, L0 represents the length of the original mixed signal, and C represents the number of speech sources. The speech source may be a real person or an audio playback device, and this embodiment does not limit the type of speech source.

[0061] Optionally, the mixed signal may be collected in real time by the electronic device, or may be collected and sent in real time by other devices connected to the electronic device for communication. In this embodiment, the mixed signal obtained in real time is used to perform speech separation, that is, to perform a real-time speech separation task.

[0062] Step 102: Convert the mixed signal into a time-frequency feature representation through an encoder.

[0063] In one example, the encoder converts the mixed signal Y into a time-frequency feature representation through STFT, E∈R 2×F×T , where "2" represents the feature dimension, F and T represent the number of frequency bands and the number of time domain steps represented by the time-frequency feature, respectively.

[0064] Step 103, feature separation is performed on the time-frequency feature representation through a separation model to obtain feature sequences corresponding to different output channels; wherein the separation model includes a cache unit and at least one separation unit connected in sequence; each separation unit includes a frequency domain modeling subunit, a time domain modeling subunit and a causal attention refinement subunit connected in sequence, and the time domain modeling subunits in each separation unit are connected to the cache unit; for the i-th separation unit, the frequency domain modeling subunit in the i-th separation unit is used to perform feature segmentation on the input feature data along the frequency dimension, and perform frequency domain modeling on the first feature after segmentation based on the first LSTM network to obtain frequency domain modeling features; the time domain modeling subunit in the i-th separation unit The subunit is used to perform feature segmentation on the frequency domain modeling features along the time domain dimension, and obtain the network features corresponding to the i-1th separation unit stored in the cache unit, and process the segmented second features and network features based on the second LSTM network to obtain the time domain modeling features and the updated network features; the updated network features are processed and stored by the cache unit to obtain the network features corresponding to the i-th separation unit for use by the time domain modeling subunit in the i+1th separation unit when performing time domain modeling; the causal attention refinement subunit in the i-th separation unit is used to extract the time domain modeling features based on the attention weights and output them to the i+1th separation unit for processing to enhance the time-frequency features.

[0065] Wherein, i is a positive integer, and the network features include the hidden state features and cell state features of the second LSTM network.

[0066] In this embodiment, the features output by the encoder are input into the separation model for processing. Each separation unit in the separation model processes the input features G in sequence. Each separation unit includes a frequency domain modeling subunit, a time domain modeling subunit and a causal attention refinement subunit. Specifically, G i Input into the i-th separation unit to obtain the causal attention refined feature G i+1 , G i+1 Input to the i+1th separation unit to obtain the causal attention refined feature G i+2 , ..., and so on, until the last separation unit is processed. These sub-units aggregate local and global time-frequency information, enhance the time-frequency representation, and improve the separation ability of the model.

[0067] Next, the frequency domain modeling subunit (F-Local), the time domain modeling subunit (T-Local) and the causal attention refinement subunit (CAR) in the same separation unit are introduced respectively.

[0068] 1. Reference Figure 2 The frequency domain modeling subunit in each separation unit shown in the figure assumes that the characteristic data input by the frequency domain modeling subunit is G i ∈R N×F×T , indicating that the feature data input to the frequency domain modeling subunit in the i-th separation unit is a three-dimensional real number matrix of N×F×T, and the sizes of each dimension are N, F and T, N represents the number of feature channels, F and T represent the number of frequency bands and the number of time domain steps represented by the time-frequency feature, respectively.

[0069] For the frequency domain modeling subunit in the first separation unit, its input feature data is obtained by processing the time-frequency feature representation output by the encoder. The specific processing method may be high-dimensional mapping through the first feature mapping unit described below. In other embodiments, the processing method may also be other methods, which are not listed one by one in this embodiment. For the frequency domain modeling subunit in each separation unit after the first separation unit, its input feature data is the feature output by the causal attention refinement subunit in the previous separation unit, that is, the causal attention refinement subunit in each separation unit is connected to the frequency domain modeling subunit in the next separation unit, so that each separation unit is connected in sequence.

[0070] Accordingly, in the frequency domain modeling subunit, the input feature data is segmented along the frequency dimension, including: segmenting the feature data G according to the frequency domain dimension using a fixed width W1 and a stride S1 through an expansion operation i ∈R N×F×T , combined with the folding process, the feature data G i Converted to sub-band features (i.e. the first feature after segmentation) in,

[0071] Accordingly, frequency domain modeling is performed on the first feature after segmentation based on the first LSTM network to obtain frequency domain modeling features, including: performing layer normalization (LN) processing on the first feature after segmentation, inputting the normalized features into the first LSTM network (i.e. Figure 2 LSTM-F) to process the historical information interaction between different frequency sub-bands and obtain the interactive features Through the feature integration network, each interactive feature is integrated across sub-bands to obtain the frequency domain modeling features

[0072] Among them, H represents the hidden layer dimension of the first LSTM network, or the number of units in the layer of the LSTM network, that is, the number of LSTM units in the LSTM network that can process one time step in the sequence data. At this time, in the first LSTM network, the feature dimension of the feature data is W1×N, and the sequence length is L F .

[0073] Schematically, the feature integration network can be implemented by a one-dimensional causal deconvolution (CDeconv).

[0074] Schematically, layer normalization can be achieved through a two-dimensional causal convolution (CConv) layer and a normalization layer.

[0075] In this embodiment, the first LSTM network is a unidirectional LSTM network to mask future frame information, thereby ensuring that the model does not utilize future frame information after the current time frame at all, so as to implement the model as a causal model.

[0076] 2. Reference Figure 3 The time domain modeling subunit in each separation unit shown, assuming that the feature data input by the time domain modeling subunit is the frequency domain modeling feature output by the frequency domain modeling subunit N represents the number of feature channels, F and T represent the number of frequency bands and the number of time domain steps represented by the time-frequency feature, respectively.

[0077] Accordingly, in the time domain modeling subunit, the frequency domain modeling features are segmented along the time domain dimension, including: splitting the frequency domain modeling features according to the time domain dimension using a fixed width W2 and a stride S2 through an expansion operation Combined with the folding process, the frequency domain modeling feature G i Converted to sub-band features (i.e. the second feature after segmentation) in,

[0078] Accordingly, the segmented second feature and the network feature are processed based on the second LSTM network to obtain the time domain modeling feature and the updated network feature, including: and network characteristics Input the second LSTM network (i.e. Figure 3 LSTM-T in the time domain modeling feature and the updated network features (H i ,C i ).

[0079] Network characteristics Model the network characteristics corresponding to the subunits in the time domain in the previous separation unit.

[0080] In the second LSTM network, the feature dimension of each feature data is N, the sequence length is W2, and there are F features at each time step. At this time, the actual composition of the second LSTM network is equivalent to L LSTMs with shared parameters. Different LSTMs have the same parameters and process a sequence segment with a sequence length of W2.

[0081] In this embodiment, the second LSTM network is a unidirectional LSTM network to mask future frame information, thereby ensuring that the model does not utilize future frame information after the current time frame at all, so that the model can be implemented as a causal model.

[0082] Accordingly, the cache unit obtains the updated network features (H i ,C i ), the updated network features are processed to obtain the network features corresponding to the i-th separation unit.

[0083] In this embodiment, in order to solve the problem of information loss in the long time dimension T, a cache unit is proposed. The cache unit is a shared module independent of multiple separation units. In one example, the cache unit processes the updated network features, including: temporally encoding the updated network features through a temporal coding network to obtain the encoded network features; realigning the encoded network features to obtain the network features corresponding to the i-th separation unit.

[0084] Optionally, refer to Figure 3 As shown, the temporal encoding network includes a third LSTM network (i.e. Figure 3 LSTM-H in the figure) and the fourth LSTM network that encodes the cell state in time sequence (i.e. Figure 3 LSTM-C in , in order to use the updated network features to aggregate and process the historical information interaction across modules. At this time, for each updated network feature of the i-th separation unit, the temporal encoding process can be expressed by the following formula:

[0085] (H i ′,C i ′)=CacheMemory(H i ,C i )

[0086] Among them, CacheMemory represents the encoding of H using the third LSTM network (i.e. LSTM-H) i , and encode C with the fourth LSTM network (ie LSTM-C) i . By reprocessing the updated network features, it helps to preserve and enhance the temporal information.

[0087] In other embodiments, the timing coding network may also be a timing network such as a Transformer network. This embodiment does not limit the implementation method of the timing coding network.

[0088] Since the hidden layer parameters of each second LSTM network encapsulate the complete network feature information of its corresponding time period, in order to ensure (H i ′,C i ′) When the information is transmitted to the (i+1)th separation module, no information leakage occurs. In this embodiment, the encoded network features need to be realigned. Specifically, the encoded network features are realigned, which can be expressed by the following formula:

[0089]

[0090] in, represents the lth hidden state feature in the network feature corresponding to the i-th separation unit, represents the lth cell state feature in the network feature corresponding to the i-th separation unit, h i ' ,l-1 represents the l-1th encoded hidden state feature corresponding to the ith separation unit, c i ' ,l-1 represents the l-1th encoded cell state feature corresponding to the i-th separation unit, l is a positive integer less than or equal to L, L is the number of the second LSTM network, and the value of L is determined based on the segmentation parameters when performing feature segmentation along the time domain dimension.

[0091] Afterwards, the network features corresponding to the ith separation unit composed of each hidden state feature and cell state feature are The summary of the global information is returned to the second LSTM network in the i+1th separation module so that the second LSTM network in the i+1th separation module can perform the same time domain modeling processing as the second LSTM network in the i-th separation module. The specific process is referred to above and will not be repeated in this embodiment.

[0092] 3. Reference Figure 4 The causal attention refinement subunit in each separation unit shown. The causal attention refinement subunit includes the following network structure: a multi-head attention network for extracting multi-head high-dimensional features of time domain modeling features; a first feature extraction path and a second feature extraction path connected to the multi-head attention network and in parallel, the first feature extraction path is used to extract features from the multi-head high-dimensional features, and the second feature extraction path is used to perform gated selection on the features extracted by the first feature extraction path to obtain features after causal attention refinement.

[0093] In one example, a multi-head attention network has a diagonal mask structure to mask out future frame information.

[0094] Specifically, the causal attention refinement subunit extracts features from the temporal modeling features based on the attention weights, including:

[0095] The time domain modeling features are input into the feature extraction network in the multi-head attention network to obtain the high-dimensional features Q∈R of the single-head attention network N×F×T , K∈R N×F×T and V∈R N×F×T ; The attention output value is determined by combining the attention mechanism in the multi-head attention network with the diagonal mask; the cross-head information is aggregated through two-dimensional CConv, parametric rectified linear unit (PReLU) and LN in turn to obtain multi-head high-dimensional features.

[0096] Schematically, the feature extraction network is implemented by 2D CConv and LN.

[0097] Optionally, the first feature extraction path and the second feature extraction path respectively include sequentially connected one-dimensional point-by-point dot products and one-dimensional depth dot products, so that fine-grained and broader contextual features can be effectively captured, thereby updating the overall representation of the feature vector before entering the next processing stage and improving the feature extraction effect.

[0098] Gating selection refers to the process of generating a gating signal through an activation function to select some features from the features extracted by the first feature extraction path.

[0099] In one example, there is one second feature extraction path, and gated selection is enabled through an activation function. The gated selected features are obtained by multiplying the features obtained by the first feature extraction path. After that, the causal attention refined features are obtained through one-dimensional point-by-point convolution for subsequent processing, such as outputting to the next separation unit for processing.

[0100] In other embodiments, the second feature extraction path may also be at least one, that is, one or at least two, and different second feature extraction paths enable gated selection through activation functions, and the gated selected features are obtained by adding the features obtained by the first feature extraction path, and then the causal attention refined features are obtained through one-dimensional point-by-point convolution for subsequent processing. This embodiment does not limit the implementation method of the second feature extraction path.

[0101] After the above-mentioned separation units complete the feature processing in sequence, the causal attention refined features output by the last causal attention refinement subunit are processed to obtain feature sequences corresponding to different output channels of the final output of the separation model. Schematically, the processing method of the causal attention refined features output by the last causal attention refinement subunit can be: the causal attention refined features are converted into real number representations and imaginary number representations corresponding to different output channels by the second feature mapping unit described below. In other embodiments, the processing method can also be other methods, which are not listed one by one in this embodiment. Schematically, the separation model also includes: a first feature mapping unit connected to the encoder, the first feature mapping unit is used to map the time-frequency feature representation to a high-dimensional representation, and input the high-dimensional representation to the first separation unit; and a second feature mapping unit connected to the last separation unit, the second feature mapping unit is used to convert the causal attention refined features output by the last separation unit into real number representations and imaginary number representations corresponding to different output channels, and obtain feature sequences corresponding to different output channels for decoding by the decoder.

[0102] The first feature mapping unit and the second feature mapping unit have a shielding layer to cover up future frame information.

[0103] Schematically, the first feature mapping unit is realized by a combination of two-dimensional CConv layers and LN connected in sequence. At this time, the time-frequency feature representation E∈R 2×F×T Mapping to a high-dimensional representation G∈R N×F×T At this time, the high-dimensional representation is directly processed as the input of the first separation unit as the initial condition G0. Where N represents the number of feature channels, and F and T represent the number of frequency bands and the number of time domain steps of the time-frequency feature representation, respectively.

[0104] The second feature mapping unit is implemented by a two-dimensional CDeconv layer with 2×C output channels and 3×3 convolution kernels, and then by a rectified linear unit (ReLU) activation function. At this time, the causal attention refined features output by the last separation unit can be converted into predicted real and imaginary representations S∈R C×2×F×T . Where C represents the number of speech sources.

[0105] Step 104, decoding the feature sequences corresponding to different output channels respectively through a decoder to obtain a time domain speech signal corresponding to each output channel; wherein each output channel corresponds to a speech source.

[0106] In one example, the decoder decodes the feature sequences corresponding to different output channels through iSTFT to obtain the time domain speech signal corresponding to each output channel. Among them, c belongs to [1, C], C represents the number of speech sources, and L0 represents the length of the original mixed signal. At this time, the length of the time domain speech signal obtained by speech separation is consistent with the length of the original mixed signal.

[0107] In summary, the speech separation method provided in this embodiment obtains a mixed signal to be separated; converts the mixed signal into a time-frequency feature representation through an encoder; performs feature separation on the time-frequency feature representation through a separation model to obtain feature sequences corresponding to different output channels; wherein the separation model includes a cache unit and at least one separation unit connected in sequence; each separation unit includes a frequency domain modeling subunit, a time domain modeling subunit and a causal attention refinement subunit connected in sequence, and the time domain modeling subunits in each separation unit are connected to the cache unit; the feature sequences corresponding to different output channels are decoded by a decoder to obtain a time domain speech signal corresponding to each output channel; wherein each output channel corresponds to a speech source; since the additionally set cache unit can store historical information during global modeling, the separation model can perform feature processing based on historical information, which can solve the problem that the causal model cannot effectively retain historical information when processing time series, resulting in poor speech separation performance, thereby improving the overall speech separation performance of the model.

[0108] In addition, by realigning the encoded network features in the buffer unit, the problem of information leakage caused by sharing network parameters across separation units can be avoided, ensuring that no information leakage occurs during the transmission of network parameters.

[0109] In addition, by temporally encoding the updated network features through the temporal coding network, it is possible to aggregate and process historical information interactions across modules and improve the global representation performance of the features.

[0110] In addition, by connecting the first feature extraction path and the second feature extraction path in parallel in the causal attention refinement subunit, the second feature extraction path can perform gated selection on the features extracted by the first feature extraction path, which can effectively capture fine-grained and broader contextual features and improve the feature extraction effect.

[0111] In addition, by setting a shielding layer in the first feature mapping unit and the second feature mapping unit, setting a diagonal mask structure in the multi-head attention network, and setting the first LSTM network and the second LSTM network as a unidirectional LSTM network, the separation model can mask future frame information during the processing process and implement the model as a causal model.

[0112] In order to more clearly understand the speech separation process provided by the present application, an example is given below to illustrate the process. Figure 5 , the process includes the following steps:

[0113] Step 51: input the mixed signal Y to be separated into an encoder to obtain a time-frequency feature representation E.

[0114] Step 52, the time-frequency feature representation is input into the separation model, and the separation model obtains a high-dimensional representation G through a two-dimensional CConv layer and LN (i.e., the first feature mapping unit) connected in sequence; the high-dimensional representation G is processed by B separation units, and then processed by a two-dimensional CDeconv layer and a ReLU activation function (i.e., the second feature mapping unit) to obtain feature sequences S corresponding to different output channels;

[0115] Wherein, each time domain modeling sub-unit in the B separation units is connected to the cache unit, and B is a positive integer.

[0116] Step 53: The feature sequences S corresponding to different output channels are processed by the decoder corresponding to each output channel to obtain the time domain speech signal corresponding to the output channel. Figure 5 China-Israel That is, C=2 is used as an example for explanation.

[0117] The speech separation method proposed in this application exceeds most of the existing methods in terms of evaluation indicators. Below, the speech separation performance of the speech separation method proposed in this application is described by taking the following settings and using the signal-to-interference ratio improvement (SI-SNRi) and the signal-to-noise ratio improvement (SNRi) as the evaluation criteria for speech separation performance as an example.

[0118] Among them, SI-SNRi is calculated by calculating the difference between the signal-to-interference ratio (SIR) of the processed signal and the SIR of the original signal. SI-SNRi can be used to evaluate the system's interference removal capability. SNRi is calculated by calculating the difference between the signal-to-noise ratio (SNR) of the processed signal and the SNR of the original signal. SNRi can be used to evaluate the system's noise reduction capability. The larger the values ​​of SI-SNRi and SNRi, the better the performance of the model.

[0119] Assume that STFT uses a Hann window, and the window size and hop size are set to 64 and 8 respectively; the number of feature channels N is set to 128, B = 3; the number of hidden units H in each LSTM network layer in the separation unit is set to 64; in the CAR module, the number of attention heads is set to 2. Table 1 shows the speech separation results of this application on different data sets, specifically the causal separation performance on the WHAM! , WHAMR! and Libri2Mix data sets, and uses one second of 8kHz resolution audio to calculate the average inference time on the CPU. According to Table 1, the performance of this application is quantitatively compared with the existing models Dual-Path Recurrent Neural Network (DPRNN), SKiM, TF GridNet and Reversible Separator Feedforward Network (ReSepformer). According to Table 1, TFACM outperforms existing models in all indicators, with an improvement of about 1.8dB. In addition, TFACM has fewer parameters than all existing causal speech separation models, which can reduce the computing resources occupied by electronic devices.

[0120] Table 1:

[0121]

[0122] Furthermore, the present application also evaluates the effectiveness of the cache unit (CM) and the causal attention refinement subunit (CAR) on the WHAM! dataset. For the cache unit, the LSTM-H and LSTM-T modules therein are replaced with misaligned congruent mappings for comparison with the present application, effectively passing the state of the current separation unit directly to the next separation unit. For the causal attention refinement subunit, the gated convolution therein is completely removed for comparison with the present application. The results of these ablation studies are shown in Table 2 below. We found that deleting the cache unit will lead to a decrease in overall performance, which may be due to the model losing the ability to process internal information, greatly weakening its processing of time information. In addition, removing the causal attention refinement subunit will also lead to a significant decrease in performance, mainly because the causal attention refinement subunit provides coarse-grained global time-frequency perception by integrating separately processed frequency and time features. Therefore, it can be seen that incorporating CM and CAR modules into the separation model can effectively improve model performance and improve speech separation effects.

[0123] Table 2:

[0124] CM CAR SDRi SI-SNRi exist exist 13.3 13.0 Does not exist exist 11.8 11.4 exist Does not exist 11.8 11.5 Does not exist Does not exist 11.2 10.7

[0125] Figure 6 A block diagram of a speech separation device according to an embodiment of the present disclosure is shown. Figure 6 It can be seen that the device includes: a signal acquisition module 610, a signal encoding module 620, a feature separation module 630 and a signal decoding module 640.

[0126] A signal acquisition module 610, used to acquire a mixed signal to be separated;

[0127] A signal encoding module 620, configured to convert the mixed signal into a time-frequency feature representation through an encoder;

[0128] The feature separation module 630 is used to perform feature separation on the time-frequency feature representation through a separation model to obtain feature sequences corresponding to different output channels; wherein the separation model includes a cache unit and at least one separation unit connected in sequence; each separation unit includes a frequency domain modeling subunit, a time domain modeling subunit and a causal attention refinement subunit connected in sequence, and the time domain modeling subunits in each separation unit are connected to the cache unit; for the i-th separation unit, the frequency domain modeling subunit in the i-th separation unit is used to perform feature segmentation on the input feature data along the frequency dimension, and perform frequency domain modeling on the first feature after segmentation based on the first LSTM network to obtain frequency domain modeling features; the time domain modeling subunit in the i-th separation unit is used to perform frequency domain modeling on the frequency domain modeling features along the time domain dimension Feature segmentation, and obtaining the network features corresponding to the i-1th separation unit stored in the cache unit, processing the segmented second features and the network features based on the second LSTM network to obtain time domain modeling features and updated network features; the updated network features are processed and stored by the cache unit to obtain the network features corresponding to the i-th separation unit for use by the time domain modeling subunit in the i+1th separation unit when performing time domain modeling; the causal attention refinement subunit in the i-th separation unit is used to extract the time domain modeling features based on the attention weight and output them to the i+1th separation unit for processing to enhance the time-frequency features; wherein, i is a positive integer, and the network features include hidden state features and cell state features of the second LSTM network;

[0129] The signal decoding module 640 is used to decode the feature sequences corresponding to different output channels through a decoder to obtain a time domain speech signal corresponding to each output channel; wherein each output channel corresponds to a speech source.

[0130] For relevant details, please refer to the above method embodiment.

[0131] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0132] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0133] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0134] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0135] Figure 7 1 is a block diagram of a device 1900 for speech separation according to an exemplary embodiment. For example, the device 1900 may be provided as a server or a terminal device. Figure 7 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0136] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0137] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0138] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A speech separation method, characterized in that: The method comprises: Acquire a mixed signal to be separated; Converting the mixed signal into a time-frequency feature representation through an encoder; The time-frequency feature representation is feature separated by a separation model to obtain feature sequences corresponding to different output channels; wherein the separation model includes a cache unit and at least one separation unit connected in sequence; each separation unit includes a frequency domain modeling subunit, a time domain modeling subunit and a causal attention refinement subunit connected in sequence, and the time domain modeling subunits in each separation unit are connected to the cache unit; for the i-th separation unit, the frequency domain modeling subunit in the i-th separation unit is used to perform feature segmentation on the input feature data along the frequency dimension, and perform frequency domain modeling on the first feature after segmentation based on the first LSTM network to obtain frequency domain modeling features; the time domain modeling subunit in the i-th separation unit is used to perform feature segmentation on the frequency domain modeling features along the time domain dimension, and The network feature corresponding to the i-1th separation unit stored in the cache unit is obtained, and the segmented second feature and the network feature are processed based on the second LSTM network to obtain the time domain modeling feature and the updated network feature; the updated network feature is processed and stored by the cache unit to obtain the network feature corresponding to the i-th separation unit for use by the time domain modeling subunit in the i+1th separation unit when performing time domain modeling; the causal attention refinement subunit in the i-th separation unit is used to extract the time domain modeling feature based on the attention weight and output it to the i+1th separation unit for processing to enhance the time-frequency feature; wherein, i is a positive integer, and the network feature includes the hidden state feature and the cell state feature of the second LSTM network; The feature sequences corresponding to different output channels are decoded by a decoder to obtain a time domain speech signal corresponding to each output channel; wherein each output channel corresponds to a speech source.

2. The method according to claim 1, characterized in that The updated network features are processed by the cache unit, including: Performing temporal coding on the updated network features through a temporal coding network to obtain encoded network features; The encoded network features are realigned to obtain the network features corresponding to the i-th separation unit.

3. The method according to claim 2, characterized in that The temporal coding network includes a third LSTM network for temporally coding hidden states and a fourth LSTM network for temporally coding cell states.

4. The method according to claim 2, characterized in that: The encoded network features are realigned to obtain the network features corresponding to the i-th separation unit, which is expressed by the following formula: in, represents the lth hidden state feature in the network feature corresponding to the i-th separation unit, represents the lth cell state feature in the network feature corresponding to the i-th separation unit, h i ′ ,l-1 represents the l-1th encoded hidden state feature corresponding to the ith separation unit, c i ′ ,l-1 represents the l-1th encoded cell state feature corresponding to the i-th separation unit, l is a positive integer less than or equal to L, L is the number of the second LSTM networks, and the value of L is determined based on the segmentation parameters when performing feature segmentation along the time domain dimension.

5. The method according to claim 1, characterized in that The causal attention refinement subunit includes the following network structure: Multi-head attention network, used to extract multi-head high-dimensional features of temporal modeling features; A first feature extraction path and a second feature extraction path are connected to and parallel to the multi-head attention network, the first feature extraction path is used to extract features from the multi-head high-dimensional features, and the second feature extraction path is used to perform gated selection on the features extracted by the first feature extraction path to obtain features after causal attention refinement.

6. The method according to claim 5, characterized in that The multi-head attention network has a diagonal mask structure to mask future frame information.

7. The method according to any one of claims 1 to 6, characterized in that: The first LSTM network and the second LSTM network are unidirectional LSTM networks to conceal future frame information.

8. The method according to any one of claims 1 to 6, characterized in that: The separation model also includes: a first feature mapping unit connected to the encoder, used to map the time-frequency feature representation into a high-dimensional representation, and input the high-dimensional representation into a first separation unit; and A second feature mapping unit connected to the last separation unit is used to convert the causal attention refined features output by the last separation unit into real number representations and imaginary number representations corresponding to different output channels, respectively, to obtain feature sequences corresponding to different output channels, for decoding by the decoder; The first feature mapping unit and the second feature mapping unit have a shielding layer to cover up future frame information.

9. A speech separation device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 8 when executing the instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images