Speech separation method, device and storage medium based on fine-grained coding

Abstract

The present invention discloses a speech separation method, device and storage medium based on fine-grained coding. The method includes a coarse separation stage and a fine separation stage. The coarse separation stage first preliminarily separates mixed speech in a coarse-grained coding space defined by a single-order latent domain. The fine separation stage utilizes a fine-grained coding mechanism to decompose the existing single-order latent domain into a high-order latent domain, and construct a fine-grained coding space, thereby achieving a more fine-grained feature separation for the result of the preliminary separation. The present invention designs a gradually refined speech separation framework, which can provide more refined speech separation.

Description

Technical Field

[0001] The present invention relates to the field of speech signal processing and analysis, and in particular to a speech separation method, device and storage medium based on fine-grained coding. Background Art

[0002] Speech separation, also known as the "cocktail party" problem in the field of speech processing, aims to separate the clean speech belonging to each speaker from the mixed speech of multiple speakers speaking at the same time, also known as speaker separation. Speech separation technology, as a key speech preprocessing technology, is usually combined with speech recognition technology and voiceprint recognition technology to meet the needs of voice interaction in noisy environments in daily life. It is widely used in scenarios such as conference log translation, target voice retrieval, and home smart device interaction. The present invention is aimed at speaker-independent speech separation under single-channel conditions, that is, under a single sound channel condition, speech separation is achieved without relying on prior information about the speaker's identity.

[0003] There are two main types of existing speech separation methods, namely frequency domain-based methods and learnable latent domain-based methods. Frequency domain-based methods use short-time Fourier transform to convert speech signals into frequency domain, obtain spectral features, and then achieve speaker separation in the frequency domain. This type of method has two main disadvantages: on the one hand, most methods only reconstruct the amplitude spectrum and ignore the phase information, which is also an important attribute of speech; on the other hand, the frequency domain is not necessarily an optimal speaker-separable coding space and may not provide sufficient speaker separability. Learnable latent domain-based methods use deep neural networks to learn a latent domain that can be used for speaker separation. Compared with frequency domain-based methods, the advantage of learnable latent domain-based methods is that the entire separation network model, including encoding, separation, and decoding, is learnable. The time domain speech separation network TASNET is a classic example of this type of method. The network uses a 1-D convolutional neural network to learn a latent domain, and multiple convolution kernels of the convolution layer constitute the basis function of this latent domain. The speech signal is encoded into the latent domain, then speaker separation is achieved in the latent domain, and finally decoded to obtain the speech signal in the time domain. Most current research on speech separation focuses on designing an efficient feature separator to achieve more accurate feature separation. Both of the above methods achieve speaker separation in a single-order domain, which is the frequency domain or a learnable latent domain. However, single-order cannot provide sufficient speaker separability, especially for mixed speech containing speakers with highly similar speech features. Summary of the invention

[0004] In view of the above problems, the present invention provides a speech separation method, device and storage medium based on fine-grained coding, and realizes fine separation of mixed speech by designing a gradually refined speech separation method based on fine-grained coding.

[0005] A first aspect of the present invention provides a speech separation method based on fine-grained coding, the method comprising the following steps:

[0006] Rough separation stage: With mixed speech as input, a convolutional neural network is used to learn a single-order latent field to obtain a coarse-grained coding space, in which the coding features of the mixed speech are obtained. The feature separator is used to learn the time domain dependency of the coding feature sequence and estimate the feature mask to obtain the coarse separation features in the coarse-grained coding space;

[0007] Fine separation stage: taking the coarse separation features as input, according to the fine-grained coding mechanism, the same new latent domain is learned along each basis function of the existing latent domain. The existing latent domain and the newly learned latent domain together constitute a high-order latent domain, and a fine-grained coding space is obtained. In the fine-grained coding space, a multi-branch network structure with shared parameters is used. Each branch network structure separates in parallel each coarse separation feature obtained from the coarse separation stage, and the parallel separation results of each branch network structure are merged to obtain the fine separation features in the fine-grained coding space. After decoding, the fine separation features are the speech separation results.

[0008] Furthermore, the coarse separation stage includes a coarse encoder and a coarse separator. The coarse encoder uses a coarse separator including N c A 1-D convolutional layer with filters is combined with a nonlinear activation function to learn a single-order latent field to obtain the coding features of the mixed speech in the coarse-grained coding space; the coarse separator takes the coding features as input, learns the long-term dependency of the coding feature sequence, and estimates Q feature masks corresponding to Q speakers.

[0009] Furthermore, the coarse separator uses the DPTNET separator to learn the long-term dependency of the encoding feature sequence according to the dual-path time domain modeling strategy. The specific steps include:

[0010] Use a layer normalization and a linear layer to adjust the number of channels of the encoded features;

[0011] The encoded features are encoded with a step size of The length of is divided into a plurality of continuous feature segments of length L;

[0012] R DPTNET blocks are used to implement time domain modeling. Each DPTNET block includes intra-segment paths and inter-segment paths, which respectively learn the local time domain information within a segment and the global time domain information between segments. In DPTNET, both the intra-segment path and the inter-segment path are composed of improved Transformer layers, in which the first linear layer of the feedforward network is replaced by a bidirectional long short-term memory neural network to learn the sequential information in the time domain.

[0013] After time domain modeling, a nonlinear transformation PReLU layer and a linear layer are used to expand the number of channels by Q times;

[0014] The feature segments are converted into feature sequences using an overlap-sum operation, and the final feature mask is estimated using a linear layer and a nonlinear activation function.

[0015] Furthermore, during the training process, the coarse separation stage also includes a coarse decoder, which uses a 1-D transposed convolutional layer to decode the separated coarse separation features into the time domain to obtain the speech signal belonging to each speaker.

[0016] Furthermore, the fine separation stage includes a fine encoder, a fine separator, and a fine decoder. Before the fine separation stage is implemented, the existing latent domain Upper N c The basis functions are divided into P groups, and the fine encoder is used in parallel to act on each group of coarse separation features to learn the same new latent domain. Hidden Domain and hidden domain Together they form a high-level hidden domain Get a fine-grained encoding space, and each group gets the decomposed encoding features; the fine separator follows the latent domain N c The fine separator uses P basis functions to separate P coded features in parallel, and the characteristics of each speaker are obtained by merging the separation results of different branches. The network structure of the fine separator is consistent with that of the coarse separator, which is R DPTNET blocks; the fine decoder decodes the characteristics of each speaker into a time domain signal to obtain the speech signal belonging to each speaker.

[0017] A second aspect of the present invention provides a speech separation device based on fine-grained coding, the device comprising:

[0018] A coarse separation module is used to take the mixed speech as input, use the convolutional neural network to learn a single-order latent field, obtain a coarse-grained coding space, obtain the coding features of the mixed speech in the coarse-grained coding space, use the feature separator to learn the time domain dependency of the coding feature sequence, estimate the feature mask, and obtain the coarse separation features in the coarse-grained coding space;

[0019] The fine separation module is used to take the coarse separation features as input, and learn the same new latent domain along each basis function of the existing latent domain according to the fine-grained coding mechanism. The existing latent domain and the newly learned latent domain together constitute a high-order latent domain to obtain a fine-grained coding space. In the fine-grained coding space, a multi-branch network structure with shared parameters is used. Each branch network structure separates in parallel each coarse separation feature obtained from the coarse separation stage, and the parallel separation results of each branch network structure are merged to obtain the fine separation features in the fine-grained coding space. The fine separation features are decoded to obtain the speech separation results.

[0020] Further, the coarse separation module includes a coarse encoder and a coarse separator, and the coarse encoder is used to utilize the N c A 1-D convolutional layer with filters is combined with a nonlinear activation function to learn a single-order latent field to obtain the coding features of the mixed speech in the coarse-grained coding space; the coarse separator is used to take the coding features as input, learn the long-term dependency of the coding feature sequence, and estimate the Q feature masks corresponding to the Q speakers.

[0021] Furthermore, the fine separation module includes a fine encoder, a fine separator, and a fine decoder. Before the fine separation module acts, the existing latent domain Upper N c The basis functions are divided into P groups, and the fine encoder is used to act in parallel on each group of coarse separation features to learn the same new latent domain. Hidden Domain and hidden domain Together they form a high-level hidden domain Get a fine-grained coding space, and each group gets the decomposed coding features; the fine separator is used to separate the latent domain N c The basis functions are used to separate P coded features in parallel, and the features of each speaker are obtained by merging the separation results of different branches. The fine decoder is used to decode the features of each speaker into a time domain signal to obtain the speech signal belonging to each speaker.

[0022] The third aspect of the present invention provides a speech separation device based on fine-grained coding, comprising: a processor; and a memory, wherein a computer executable program is stored in the memory, and when the computer executable program is executed by the processor, the above-mentioned speech separation method based on fine-grained coding is executed.

[0023] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which instructions are stored. When the instructions are executed by a processor, the processor executes the above-mentioned speech separation method based on fine-grained coding.

[0024] The present invention provides a speech separation method, device and storage medium based on fine-grained coding, which first preliminarily separates mixed speech in a coarse-grained coding space defined by a single-order latent domain, and then uses a fine-grained coding mechanism to decompose the existing single-order latent domain into a high-order latent domain to construct a fine-grained coding space, thereby achieving a more fine-grained feature separation for the preliminary separation result. The beneficial effects finally achieved are:

[0025] 1. A fine-grained encoding mechanism is designed to decompose the single-order latent domain into high-order latent domains, thereby decomposing a coarse-grained encoding space into a fine-grained encoding space with more refined representation capabilities and providing stronger speaker separability.

[0026] 2. A gradually refined speech separation framework is designed. In the coarse separation stage, the mixed speech is preliminarily separated in the coarse-grained coding space. In the fine separation stage, the fine-grained coding space is constructed using the designed fine-grained coding mechanism, and the separation results of the coarse separation stage are more accurately separated with a multi-branch network structure, and the network parameters are shared between different branches.

[0027] 3. The method of the present invention is generalizable and can be applied to any existing separation network TASNET based on the encoding-separation-decoding structure, including but not limited to DPRNN-TASNET and DPTNET-TASNET. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 1 is a flow chart of a speech separation method based on fine-grained coding in an embodiment of the present invention;

[0029] Figure 2 is a schematic diagram of a fine-grained encoding mechanism method in an embodiment of the present invention;

[0030] Figure 3 Schematic diagram of the separation method in the fine separation stage in an embodiment of the present invention;

[0031] Figure 4 is a schematic diagram of a decoding method in a fine separation stage in an embodiment of the present invention;

[0032] Figure 5 is a structural schematic diagram of a speech separation device based on fine-grained coding in an embodiment of the present invention;

[0033] Figure 6 It is the architecture of the computer device in the embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to further explain the technical solution of the present invention in detail, this embodiment is implemented on the premise of the technical solution of the present invention, and provides a detailed implementation method and specific steps.

[0035] refer to Figure 1 , a speech separation method based on fine-grained coding, the method comprising the following steps:

[0036] Rough separation stage: With mixed speech as input, a convolutional neural network is used to learn a single-order latent field to obtain a coarse-grained coding space, in which the coding features of the mixed speech are obtained. The feature separator is used to learn the time domain dependency of the coding feature sequence, estimate the feature mask, and obtain the coarse separation features in the coarse-grained coding space;

[0037] Fine separation stage: taking the coarse separation features as input, according to the fine-grained coding mechanism, the same new latent domain is learned along each basis function of the existing latent domain. The existing latent domain and the newly learned latent domain together constitute a high-order latent domain, and a fine-grained coding space is obtained. In the fine-grained coding space, a multi-branch network structure with shared parameters is used. Each branch network structure separates in parallel each coarse separation feature obtained from the coarse separation stage, and the parallel separation results of each branch network structure are merged to obtain the fine separation features in the fine-grained coding space. After decoding, the fine separation features are the speech separation results.

[0038] In an optional embodiment, the coarse separation stage includes a coarse encoder and a coarse separator. The coarse encoder uses a coarse separator including N c A 1-D convolutional layer with filters is combined with a nonlinear activation function to learn a single-order latent field to obtain the coding features of the mixed speech in the coarse-grained coding space; the coarse separator takes the coding features as input, learns the long-term dependency of the coding feature sequence, and estimates Q feature masks corresponding to Q speakers.

[0039] Specifically, the coarse encoder ε c Take mixed speech x∈R 1×T As input, use N c The 1-D convolutional layer with filters combined with the nonlinear activation function ReLU learns a single-order latent field, thereby defining a coarse-grained coding space and obtaining the coding features of the mixed speech in the coarse-grained coding space.

[0040] Specifically, the coarse separator With encoding feature F c As input, learn the long-term dependency of feature sequences and estimate Q feature masks They correspond to Q speakers respectively. The feature mask is a non-negative value, indicating the weights corresponding to different basis functions. For speaker i, the separated features are Where ⊙ represents element-wise dot product.

[0041] In an alternative embodiment, the coarse separator It can be any of the existing TASNET separators.

[0042] In a specific embodiment, the coarse separator The DPTNET separator in DPTNET-TASNET is used to learn the long-term dependency of the encoding feature sequence according to the dual-path time domain modeling strategy. The specific steps include:

[0043] Step 1, use a layer normalization and a linear layer to adjust the feature F c The number of channels;

[0044] Step 2: Encode the feature F c According to the step length The length of is divided into a plurality of continuous feature segments of length L;

[0045] Step 3: Use R DPTNET blocks to implement time domain modeling. Each DPTNET block includes intra-segment paths and inter-segment paths, which respectively learn the local time domain information within a segment and the global time domain information between segments. In DPTNET, both the intra-segment path and the inter-segment path are composed of improved Transformer layers, in which the first linear layer of the feedforward network is replaced by a bidirectional long short-term memory neural network to learn the sequential information in the time domain.

[0046] Step 4: After time domain modeling, a nonlinear transformation PReLU layer and a linear layer are used to expand the number of channels by Q times;

[0047] Step 5: Use the overlap sum operation to convert the feature segment into a feature sequence, and use a linear layer and a nonlinear activation function ReLU to estimate the final feature mask.

[0048] In an optional embodiment, during the training process, the coarse separation stage also includes a coarse decoder, which uses a 1-D transposed convolution layer to decode the separated coarse separation features into the time domain to obtain a speech signal belonging to each speaker.

[0049] Specifically, the coarse decoder The separated features are decoded into the time domain using a 1-D transposed convolutional layer to obtain the speech signal belonging to each speaker.

[0050] The decoding process of the rough separation stage is only performed during the training process. In the inference process, only the separated features need to be decoded. Input to the fine separation stage.

[0051] In an optional embodiment, the fine separation stage includes a fine encoder, a fine separator, and a fine decoder. Before the fine separation stage is implemented, the existing latent domain Upper N c The basis functions are divided into P groups, and the fine encoder is used in parallel to act on each group of coarse separation features to learn the same new latent domain. Hidden Domain and hidden domain Together they form a high-level hidden domain Get a fine-grained encoding space, and each group gets the decomposed encoding features; the fine separator follows the latent domain N c The basis functions are used to separate P coded features in parallel, and the features of each speaker are obtained by merging the separation results of different branches. The fine decoder decodes the features of each speaker into a time domain signal to obtain the speech signal belonging to each speaker.

[0052] Specifically, in each branch, the precision encoder ε r Characteristics after separation at the rough separation stage As input, the fine-grained encoding mechanism is used to encode the features in a fine-grained manner:

[0053] refer to Figure 2 , in the execution process of the fine-grained encoding mechanism, the existing latent domain Upper N c The basis functions are divided into P groups, and then the precision encoder ε r Apply to each group in parallel Learning a new latent domain Hidden Domain and hidden domain Together they form a higher-order (second-order) hidden domain Get a fine-grained encoding space, and each group gets the decomposed features refer to Figure 2 , in the execution process of the fine-grained encoding mechanism, the existing latent domain Upper N c The basis functions are divided into P groups, and then the precision encoder ε r Apply to each group in parallel Learning a new latent domain Hidden Domain and hidden domain Together they form a higher-order (second-order) hidden domain Get a fine-grained encoding space, and each group gets the decomposed features

[0054]

[0055] After fine-grained encoding, the features in the fine-grained encoding space are obtained Size is N r ×P×T ′ .

[0056] refer to Figure 3 , which corresponds to the encoder sharing process in the fine-grained encoding mechanism, using the fine separator Along the hidden domain N c basis functions to separate P feature blocks in parallel

[0057]

[0058]

[0059] Among them, the fine separator and the rough separator in the rough separation stage The model results are consistent, consisting of R DPTNET blocks. By merging the separation results of different branches, the feature belonging to the jth speaker is the sum of the jth components after separation of each branch:

[0060] refer to Figure 4 , decoder Includes two sub-decoders implemented based on 1-D transposed convolutional layers and Along the hidden domain N c The basis functions are processed in parallel, that is, each feature block is processed in parallel. Decode to coarse separation space; Decode the features into time domain signals to obtain the speech signals belonging to each speaker

[0061]

[0062]

[0063] Furthermore, during the training process, the entire method model is jointly optimized end-to-end, and the optimization goal is to maximize the difference between the estimated speech and the real speech s in the coarse separation stage and the fine separation stage. i The scale-invariant source-to-noise ratio (SI-SNR) compared with (i=1,…,Q) uses the sentence-level permutation-invariant training method uPIT to align multiple output values ​​and true values. The training loss function of the entire network is

[0064]

[0065]

[0066]

[0067] in, and Represent the loss functions of the coarse separation stage and the fine separation stage respectively, π1 and π2 are the sets corresponding to the permutations in all possible Q! of Q speakers. The elements in .

[0068] Below, refer to Figure 5 To describe the embodiment of the present disclosure Figures 1 to 4 The device corresponding to the method shown is a speech separation device based on fine-grained coding, wherein the device 100 includes: a coarse separation module 101, which is used to take mixed speech as input, use a convolutional neural network to learn a single-order latent field, obtain a coarse-grained coding space, obtain coding features of the mixed speech in the coarse-grained coding space, use a feature separator to learn the time domain dependency of the coding feature sequence, estimate the feature mask, and obtain coarse separation features in the coarse-grained coding space; a fine separation module 102, which is used to take the coarse separation features as input, learn the same new latent field along each basis function of the existing latent field according to the fine-grained coding mechanism, and the existing latent field and the newly learned latent field together constitute a high-order latent field to obtain a fine-grained coding space, in which a multi-branch network structure with shared parameters is used, each branch network structure separates each coarse separation feature obtained from the coarse separation stage in parallel, and merges the parallel separation results of each branch network structure to obtain fine separation features in the fine-grained coding space, and the fine separation features are decoded to obtain the speech separation result. In addition to these two modules, the device 100 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present disclosure, their illustration and description are omitted here.

[0069] Furthermore, the coarse separation module 101 includes a coarse encoder 1011 and a coarse separator 1012. The coarse encoder 1011 is used to utilize the N c The 1-D convolution layer of filters is combined with a nonlinear activation function to learn a single-order latent field to obtain the coding features of the mixed speech in the coarse-grained coding space; the coarse separator 1012 is used to take the coding features as input, learn the long-term dependency of the coding feature sequence, and estimate Q feature masks corresponding to Q speakers.

[0070] Optionally, the coarse separation module 101 further includes a coarse decoder 1013, which is used to decode the separated coarse separation features into the time domain using a 1-D transposed convolution layer during the training process to obtain a speech signal belonging to each speaker.

[0071] Furthermore, the fine separation module 102 includes a fine encoder 1021, a fine separator 1022, and a fine decoder 1023. Before the fine separation module 102 operates, the existing latent domain Upper N c The basis functions are divided into P groups, and the fine encoder 1021 is used to act in parallel on each group of coarse separation features to learn the same new latent domain Hidden Domain and hidden domain Together they form a high-level hidden domain A fine-grained coding space is obtained, and each group obtains the decomposed coding features; the fine separator 1022 is used to separate the latent domain N c The basis functions are used to separate P coding features in parallel, and the features of each speaker are obtained by merging the separation results of different branches; the fine decoder 1023 is used to decode the features of each speaker into a time domain signal to obtain a speech signal belonging to each speaker.

[0072] The specific working process of the speech separation device 100 based on fine-grained coding is referred to the description of the speech separation method based on fine-grained coding, which will not be repeated here.

[0073] In addition, the device of the embodiment of the present invention can also be used by Figure 6 The architecture of the computing device shown is implemented. Figure 6 The architecture of the computing device is shown. Figure 6 As shown, a computer system 201, a system bus 203, one or more CPUs 204, an input / output component 202, a memory 205, etc. The memory 205 can store various data or files used for computer processing and / or communication and program instructions including a speech separation method based on fine-grained coding executed by the CPU. Figure 6 The architecture shown is only exemplary and can be adjusted according to actual needs when implementing different devices. Figure 6 One or more components in .

[0074] The embodiment of the present invention may also be implemented as a computer-readable storage medium. The computer-readable storage medium according to the embodiment stores computer-readable instructions. When the computer-readable instructions are executed by a processor, the speech separation method based on fine-grained coding according to the embodiment of the present invention described with reference to the above drawings may be executed.

[0075] In summary, the speech separation method, device and storage medium based on fine-grained coding provided in the above embodiments are firstly separated in the coarse-grained coding space defined by the single-order latent field, and then the fine-grained coding mechanism is used to decompose the existing single-order latent field into high-order latent fields to construct a fine-grained coding space, thereby achieving more fine-grained feature separation for the results of the preliminary separation. A fine-grained coding mechanism is designed to decompose a single-order latent domain into a high-order latent domain, thereby decomposing a coarse-grained coding space into a fine-grained coding space, which has a more refined representation ability and provides stronger speaker separability; a gradually refined speech separation framework is designed, and in the coarse separation stage, the mixed speech is preliminarily separated in the coarse-grained coding space; in the fine separation stage, the fine-grained coding space is constructed using the designed fine-grained coding mechanism, and the separation results of the coarse separation stage are more accurately separated with a multi-branch network structure, and network parameters are shared between different branches; the method of the present invention is generalizable and can be applied to any existing separation network TASNET based on the encoding-separation-decoding structure, including but not limited to DPRNN-TASNET and DPTNET-TASNET.

[0076] In this document, the terms "comprises," "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such step or method.

[0077] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A speech separation method based on fine-grained coding, characterized in that: The method comprises the following steps: Rough separation stage: With mixed speech as input, a convolutional neural network is used to learn a single-order latent field to obtain a coarse-grained coding space, in which the coding features of the mixed speech are obtained. The feature separator is used to learn the time domain dependency of the coding feature sequence and estimate the feature mask to obtain the coarse separation features in the coarse-grained coding space; Fine separation stage: taking the coarse separation features as input, according to the fine-grained coding mechanism, the same new latent domain is learned along each basis function of the existing latent domain. The existing latent domain and the newly learned latent domain together constitute a high-order latent domain, and a fine-grained coding space is obtained. In the fine-grained coding space, a multi-branch network structure with shared parameters is used. Each branch network structure separates in parallel each coarse separation feature obtained from the coarse separation stage, and the parallel separation results of each branch network structure are merged to obtain the fine separation features in the fine-grained coding space. After decoding, the fine separation features are the speech separation results.

2. The speech separation method according to claim 1, characterized in that: The rough separation stage includes a rough encoder and a rough separator. The rough encoder uses a coarse encoder containing N c A 1-D convolutional layer with filters is combined with a nonlinear activation function to learn a single-order latent field to obtain the coding features of the mixed speech in the coarse-grained coding space; the coarse separator takes the coding features as input, learns the long-term dependency of the coding feature sequence, and estimates Q feature masks corresponding to Q speakers.

3. The speech separation method according to claim 2, characterized in that: The coarse separator uses the DPTNET separator to learn the long-term dependency of the encoding feature sequence based on the dual-path time domain modeling strategy. The specific steps include: Use a layer normalization and a linear layer to adjust the number of channels of the encoded features; The encoded features are encoded with a step size of The length of is divided into a plurality of continuous feature segments of length L; R DPTNET blocks are used to implement time domain modeling. Each DPTNET block includes intra-segment paths and inter-segment paths, which respectively learn the local time domain information within a segment and the global time domain information between segments. In DPTNET, both the intra-segment path and the inter-segment path are composed of improved Transformer layers, in which the first linear layer of the feedforward network is replaced by a bidirectional long short-term memory neural network to learn the sequential information in the time domain. After time domain modeling, a nonlinear transformation PReLU layer and a linear layer are used to expand the number of channels by Q times; The feature segments are converted into feature sequences using an overlap-sum operation, and the final feature mask is estimated using a linear layer and a nonlinear activation function.

4. The speech separation method according to claim 1, characterized in that: During the training process, the coarse separation stage also includes a coarse decoder, which uses a 1-D transposed convolutional layer to decode the separated coarse separation features into the time domain to obtain the speech signal belonging to each speaker.

5. The speech separation method according to claim 1, characterized in that: The fine separation stage includes a fine encoder, a fine separator, and a fine decoder. Before the fine separation stage is implemented, the existing latent domain Upper N c The basis functions are divided into P groups, and the fine encoder is used in parallel to act on each group of coarse separation features to learn the same new latent domain. Hidden Domain and hidden domain Together they form a high-level hidden domain Get a fine-grained encoding space, and each group gets the decomposed encoding features; the fine separator follows the latent domain N c The basis functions are used to separate P coded features in parallel, and the features of each speaker are obtained by merging the separation results of different branches. The fine decoder decodes the features of each speaker into a time domain signal to obtain the speech signal belonging to each speaker.

6. A speech separation device based on fine-grained coding, characterized in that: The device comprises: A coarse separation module is used to take mixed speech as input, use a convolutional neural network to learn a single-order latent field, obtain a coarse-grained coding space, obtain coding features of the mixed speech in the coarse-grained coding space, and use a feature separator to learn the time domain dependency of the coding feature sequence and estimate the feature mask to obtain coarse separation features in the coarse-grained coding space; The fine separation module is used to take the coarse separation features as input, and learn the same new latent domain along each basis function of the existing latent domain according to the fine-grained coding mechanism. The existing latent domain and the newly learned latent domain together constitute a high-order latent domain to obtain a fine-grained coding space. In the fine-grained coding space, a multi-branch network structure with shared parameters is used. Each branch network structure separates in parallel each coarse separation feature obtained from the coarse separation stage, and the parallel separation results of each branch network structure are merged to obtain the fine separation features in the fine-grained coding space. The fine separation features are decoded to obtain the speech separation results.

7. The speech separation device according to claim 6, characterized in that: The coarse separation module includes a coarse encoder and a coarse separator. The coarse encoder is used to utilize the N c A 1-D convolutional layer with filters is combined with a nonlinear activation function to learn a single-order latent field to obtain the coding features of the mixed speech in the coarse-grained coding space; the coarse separator is used to take the coding features as input, learn the long-term dependency of the coding feature sequence, and estimate the Q feature masks corresponding to the Q speakers.

8. The speech separation device according to claim 6, characterized in that: The fine separation module includes a fine encoder, a fine separator, and a fine decoder. Before the fine separation module works, the existing latent domain Upper N c The basis functions are divided into P groups, and the fine encoder is used to act in parallel on each group of coarse separation features to learn the same new latent domain. Hidden Domain and hidden domain Together they form a high-level hidden domain Get a fine-grained coding space, and each group gets the decomposed coding features; the fine separator is used to separate the latent domain N c The basis functions are used to separate P coded features in parallel, and the features of each speaker are obtained by merging the separation results of different branches. The fine decoder is used to decode the features of each speaker into a time domain signal to obtain the speech signal belonging to each speaker.

9. A speech separation device based on fine-grained coding, characterized in that: include: processor; and a memory, wherein a computer executable program is stored in the memory, and when the computer executable program is executed by the processor, the speech separation method based on fine-grained coding described in any one of claims 1 to 5 is executed.

10. A computer-readable storage medium having instructions stored thereon, wherein when the instructions are executed by a processor, the processor executes the speech separation method based on fine-grained coding as described in any one of claims 1 to 5.

Images