Speech separation method and device, computer device and storage medium

Abstract

The embodiment of the application belongs to the field of artificial intelligence, and relates to a speech separation method, comprising obtaining a trained speech separation model and a speech time domain mixed signal, performing convolution calculation on the speech time domain mixed signal based on an encoder of the speech separation model to obtain encoded features; inputting the encoded features into a mask network of the speech separation model, performing feature extraction on the encoded features according to the mask network to obtain a mask matrix corresponding to each different human voice in the speech time domain mixed signal; and calculating the product of the mask matrix and the encoded features according to a decoder of the speech separation model to obtain a separated speech signal corresponding to each human voice. The application also provides a speech separation device, a computer device and a storage medium. In addition, the application also relates to blockchain technology, and the separated speech signal can be stored in the blockchain. The application realizes accurate separation of speech and improves the efficiency of speech separation.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to speech separation methods, apparatus, computer equipment, and storage media. Background Technology

[0002] Traditional speech separation methods typically transform the mixed speech signal to the frequency domain and then separate the spectrograms of each speaker from the mixed spectrogram. In recent years, time-domain-based speech separation methods have received more attention. While directly modeling the audio signal in the time domain to achieve speech separation has yielded good results, the direct processing of the time-domain signal requires significant computational resources, ultimately leading to low speech separation efficiency. Summary of the Invention

[0003] The purpose of this application is to provide a speech separation method, apparatus, computer device, and storage medium to solve the technical problem of low speech separation efficiency.

[0004] To address the aforementioned technical problems, this application provides a speech separation method, employing the following technical solution:

[0005] The trained speech separation model and the mixed temporal signal of the speech to be separated are obtained. The speech separation model includes an encoder, a masking network, and a decoder.

[0006] The encoder based on the speech separation model performs convolution calculation on the speech temporal mixed signal to obtain the encoded features;

[0007] The encoded features are input into the masking network of the speech separation model, and feature extraction is performed on the encoded features according to the masking network to obtain the masking matrix corresponding to each different human voice in the speech temporal mixed signal;

[0008] The decoder of the speech separation model calculates the product of the mask matrix and the encoded features to obtain the separated speech signal corresponding to each human voice.

[0009] Furthermore, the masking network includes a preprocessing layer, a convolutional attention layer, and a stacking layer. The step of extracting features from the encoded features based on the masking network to obtain the mask matrix corresponding to each different human voice in the speech temporal mixed signal includes:

[0010] The encoded features are linearly extracted and segmented according to the preprocessing layer of the mask network to obtain a block feature sequence;

[0011] The block feature sequence is input into the convolutional attention layer of the mask network to calculate the separated convolutional features;

[0012] The separated convolutional features are superimposed based on the superposition layer of the mask network to obtain the mask matrix.

[0013] Furthermore, the preprocessing layer includes a normalization layer and a linear layer. The step of performing linear feature extraction and block segmentation on the encoded features based on the preprocessing layer of the mask network to obtain a block feature sequence includes:

[0014] The encoded features are normalized according to the normalization layer to obtain normalized features;

[0015] Linear features are obtained by extracting linear features from the normalized features based on the linear layer;

[0016] Obtain a preset overlap rate, and divide the linear feature into the block feature sequence from the time domain dimension according to the preset overlap rate.

[0017] Furthermore, the convolutional attention layer includes a one-dimensional convolutional layer and a multi-head attention mechanism layer. The step of inputting the block feature sequence to the convolutional attention layer of the mask network and calculating the separated convolutional features includes:

[0018] Based on the one-dimensional convolutional layer, the block feature sequence is convolved in the temporal dimension to obtain temporal convolutional features;

[0019] The multi-head attention mechanism layer performs multi-head attention calculation on the block feature sequence to obtain attention features;

[0020] The temporal convolutional features and the attention features are fused to obtain the separated convolutional features.

[0021] Furthermore, the step of fusing the temporal convolutional features and the attention features to obtain the separated convolutional features includes:

[0022] Feedforward features are obtained by calculating the temporal convolutional features and the attention features using a feedforward neural network;

[0023] The feedforward features, the temporal convolutional features, and the attention features are normalized to obtain the separated convolutional features.

[0024] Furthermore, the step of obtaining the trained speech separation model includes:

[0025] Construct a basic separation model and obtain multiple sets of training speech samples;

[0026] The basic separation model is trained based on the training speech samples to obtain the signal-to-noise ratio evaluation value of the training speech samples;

[0027] The basic separation model is iteratively trained based on the signal-to-noise ratio (SNR) evaluation value. When the SNR evaluation value reaches its maximum value, the training of the basic separation model is considered complete, and the trained speech separation model is obtained.

[0028] Furthermore, the step of iteratively training the basic separation model based on the signal-to-noise evaluation value includes:

[0029] The iterative parameters are calculated based on the signal-to-noise ratio evaluation value. These iterative parameters are then used as shared parameters of the basic masking network in the basic separation model. The basic masking network is then adjusted based on these shared parameters to obtain the adjusted speech separation model.

[0030] To address the aforementioned technical problems, this application also provides a speech separation device, which employs the following technical solution:

[0031] The acquisition module is used to acquire the trained speech separation model and the speech temporal mixed signal to be separated. The speech separation model includes an encoder, a masking network, and a decoder.

[0032] The encoding module is used to perform convolution calculation on the speech temporal mixed signal based on the encoder of the speech separation model to obtain encoded features;

[0033] The masking module is used to input the encoded features into the masking network of the speech separation model, and to extract features from the encoded features according to the masking network to obtain the masking matrix corresponding to each different human voice in the speech temporal mixed signal.

[0034] The decoding module is used to calculate the product of the mask matrix and the encoded features based on the decoder of the speech separation model to obtain the separated speech signal corresponding to each human voice.

[0035] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0036] The trained speech separation model and the mixed temporal signal of the speech to be separated are obtained. The speech separation model includes an encoder, a masking network, and a decoder.

[0037] The encoder based on the speech separation model performs convolution calculation on the speech temporal mixed signal to obtain the encoded features;

[0038] The encoded features are input into the masking network of the speech separation model, and feature extraction is performed on the encoded features according to the masking network to obtain the masking matrix corresponding to each different human voice in the speech temporal mixed signal;

[0039] The decoder of the speech separation model calculates the product of the mask matrix and the encoded features to obtain the separated speech signal corresponding to each human voice.

[0040] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0041] The trained speech separation model and the mixed temporal signal of the speech to be separated are obtained. The speech separation model includes an encoder, a masking network, and a decoder.

[0042] The encoder based on the speech separation model performs convolution calculation on the speech temporal mixed signal to obtain the encoded features;

[0043] The encoded features are input into the masking network of the speech separation model, and feature extraction is performed on the encoded features according to the masking network to obtain the masking matrix corresponding to each different human voice in the speech temporal mixed signal;

[0044] The decoder of the speech separation model calculates the product of the mask matrix and the encoded features to obtain the separated speech signal corresponding to each human voice.

[0045] The proposed speech separation method acquires a trained speech separation model and a mixed temporal signal of speech to be separated. The speech separation model includes an encoder, a masking network, and a decoder. The encoder, based on the speech separation model, performs convolution calculations on the mixed temporal signal to obtain encoded features, achieving preliminary feature extraction. Then, the encoded features are input into the masking network of the speech separation model, and feature extraction is performed based on the masking network to obtain a mask matrix corresponding to each different human voice in the mixed temporal signal. The masking network allows for precise extraction of the mask matrix corresponding to each different human voice, and its small parameter requirement significantly improves speech separation efficiency. Finally, the decoder calculates the product of the mask matrix and the encoded features to obtain the separated speech signal corresponding to each human voice. This achieves precise speech separation, improves efficiency, and significantly reduces the number of model parameters, thereby increasing training efficiency and reducing the CPU and GPU memory requirements for speech separation. This allows the model to be applied in resource-constrained scenarios, further improving the speed of speech separation. Attached Figure Description

[0046] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0048] Figure 2 A flowchart of an embodiment of the speech separation method according to this application;

[0049] Figure 3 This is a schematic diagram of a masking network in a speech separation model;

[0050] Figure 4 This is a schematic diagram of the convolutional attention layer in a masked network of a speech separation model.

[0051] Figure 5 This is a schematic diagram of one embodiment of the speech separation device according to this application;

[0052] Figure 6 This is a schematic diagram of the structure of one embodiment of the computer device according to this application.

[0053] Reference numerals: speech separation device 500, acquisition module 501, encoding module 502, masking module 503, and decoding module 504. Detailed Implementation

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0055] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0056] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0057] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0058] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0059] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0060] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0061] It should be noted that the speech separation method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the speech separation device is generally set in the server / terminal device.

[0062] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0063] Continue to refer to Figure 2 A flowchart of an embodiment of the speech separation method according to this application is shown. The speech separation method includes the following steps:

[0064] Step S201: Obtain the trained speech separation model and the speech temporal mixed signal to be separated. The speech separation model includes an encoder, a masking network, and a decoder.

[0065] In this embodiment, the speech temporal mixture signal is a temporal mixture signal including multiple different human voices. A trained speech separation model and the speech temporal mixture signal to be separated are obtained. Based on the speech separation model, the speech separation signal of each different speaker can be separated from the speech temporal mixture signal. The speech separation model includes an encoder, a masking network, and a decoder; the encoder is used to encode the input speech temporal mixture signal, the masking network is used to further extract features from the input, and the decoder is used to decode to obtain the separated speech signal corresponding to each different human voice.

[0066] Step S202: The encoder based on the speech separation model performs convolution calculation on the speech temporal mixed signal to obtain the encoded features.

[0067] Upon receiving a mixed speech signal in the temporal domain, the encoder based on this speech separation model performs convolution calculations on the mixed speech signal to obtain encoded features. Specifically, the encoder θ enc It consists of a Conv1D convolutional layer followed by an activation function (ReLU); the resulting speech temporal mixed signal X = (x1, x2, ..., x...) is obtained. T When the speech temporal mixed signal is convolved by the Conv1D convolutional layer, the convolution result is obtained; then, the encoding feature H = (h1, h2, ..., h...) is calculated using the ReLU activation function. T ), where T is the length of the speech temporal mixed signal, and the specific calculation formula for this coding feature is as follows:

[0068] H = θ enc (X)=ReLU(Conv1D(X))

[0069] Where, θ enc Let X be the encoder, and let X be the speech time-domain mixed signal.

[0070] Step S203: Input the encoded features into the masking network of the speech separation model, and extract features from the encoded features according to the masking network to obtain the masking matrix corresponding to each different human voice in the speech temporal mixed signal.

[0071] In this embodiment, the masking network includes a preprocessing layer, a convolutional attention layer, and a stacking layer. The preprocessing layer can segment the temporal signal of the speech into small signal blocks. The convolutional attention layer can perform convolution operations within and between signal blocks on the signal blocks. The stacking layer can stack the features calculated by the convolutional attention layer to finally obtain a mask matrix for each different human voice in the temporal signal of the speech. One mask matrix corresponds to one human voice (i.e., one speaker).

[0072] Specifically, as follows: Figure 3 As shown, Figure 3 This diagram illustrates a masking network in a speech separation model. The masking network includes a preprocessing layer (Chunk), an intra-block convolutional attention layer (Intra Conv-Attn), a permute layer, an inter-block convolutional attention layer (InterConv-Attn), and an overlap-add layer. When obtaining the encoded features H of the temporal mixed speech signal, these features are passed through the preprocessing layer to obtain the block feature sequence H. s The feature sequence of a feature block is calculated using an intra-block convolutional attention layer (Intra Conv-Attn) to obtain local features. These local features are then transformed to a feature dimension P using a transformation layer, yielding transformed features. Inter-block convolutional attention layers (Inter Conv-Attn) are then used to model the long-term dependencies between these transformed features, resulting in inter-block features. These inter-block features are then input into a stacking layer, which performs feature transformations to obtain the temporal features H. o The mask matrix is ​​finally obtained based on this time-domain feature. Figure 3 N in mask This represents the number of times the loop will repeat.

[0073] Step S204: Calculate the product of the mask matrix and the encoded features according to the decoder of the speech separation model to obtain the separated speech signal corresponding to each human voice.

[0074] In this embodiment, after obtaining the multiple separated mask matrices, the decoder of the speech separation model calculates the product of each mask matrix and the encoded feature, thus obtaining the separated speech signal corresponding to each different human voice (speaker). Specifically, the decoder consists of transposed convolution (Conv1D). T The mask matrix and the encoded feature are composed of H. The separated speech signal is obtained by calculating the mask matrix and the encoded feature using the transpose matrix. The calculation formula for this separated speech signal is as follows:

[0075]

[0076] in, To separate the speech signal, M 1:K Given K mask matrices, M 1:K ={M1,M2,...,M K}, H represents the encoding feature, and ⊙ represents the element-wise matrix multiplication operation.

[0077] It should be emphasized that, in order to further ensure the privacy and security of the aforementioned separated voice signals, the separated voice signals can also be stored in a blockchain node.

[0078] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0079] This embodiment achieves accurate speech separation, improves the efficiency of speech separation, and greatly reduces the number of model parameters, thereby improving the training efficiency of the model and the CPU memory required for speech separation. This allows the model to be applied in resource-constrained scenarios, further improving the speed of speech separation.

[0080] In some optional implementations of this embodiment, the masking network described above includes a preprocessing layer, a convolutional attention layer, and a stacking layer. The step of extracting features from the encoded features based on the masking network to obtain the mask matrix corresponding to each different human voice in the speech temporal mixed signal includes:

[0081] The encoded features are linearly extracted and segmented according to the preprocessing layer of the mask network to obtain a block feature sequence;

[0082] The block feature sequence is input into the convolutional attention layer of the mask network to calculate the separated convolutional features;

[0083] The separated convolutional features are superimposed based on the superposition layer of the mask network to obtain the mask matrix.

[0084] In this embodiment, the masking network includes a preprocessing layer, a convolutional attention layer, and a stacking layer. When the encoded features are obtained, the preprocessing layer of the masking network performs linear feature extraction and block segmentation on the encoded features to obtain the block feature sequence corresponding to the encoded features. Then, the block feature sequence is input into the convolutional attention layer of the masking network, and the convolutional attention layer performs decoupling convolution on the block feature sequence to obtain decoupling convolution features. Finally, the decoupling convolution features are stacked based on the stacking layer of the masking network to obtain the mask matrix.

[0085] Specifically, when obtaining the encoded features, the preprocessing layer of the mask network performs linear feature extraction and block segmentation on the encoded features to obtain the block feature sequence H. s The feature sequence of this block is passed through a convolutional attention layer θ. ca Repeated loop calculations are performed to output the sub-feature H. ca ,in, Then, the output sub-features are processed by a linear function and an activation function (PReLU) to obtain the separable convolutional features H. dk H dk =PReLU(Linear(H ca Finally, the separated convolutional features are processed through a masked network stacking layer to obtain the temporal feature H. o This temporal feature is also used to calculate the mask matrix corresponding to different human voices using linear functions and activation functions (such as ReLU). This mask matrix M... 1:K , of which M 1:K =ReLU(Linear(H o ) ×2 ).

[0086] This embodiment uses a masking network to extract features from the encoded features, which improves the accuracy of speech separation while reducing the parameters required for the speech separation model, thus achieving accurate and effective speech separation.

[0087] In some optional implementations of this embodiment, the preprocessing layer includes a normalization layer and a linear layer. The step of performing linear feature extraction and block segmentation on the encoded features based on the preprocessing layer of the mask network to obtain a block feature sequence includes:

[0088] The encoded features are normalized according to the normalization layer to obtain normalized features;

[0089] Linear features are obtained by extracting linear features from the normalized features based on the linear layer;

[0090] Obtain a preset overlap rate, and divide the linear feature into the block feature sequence from the time domain dimension according to the preset overlap rate.

[0091] In this embodiment, the preprocessing layer includes a normalization layer and a linear layer. When preprocessing the encoded features according to the preprocessing layer, the encoded features are normalized according to the normalization layer to obtain normalized features. Then, the linear layer extracts linear features from the normalized features to obtain linear features. Finally, the linear features are segmented into block feature sequences with corresponding overlap rates in the temporal dimension through chunking. Specifically, chunking involves dividing the two-dimensional features into three-dimensional features. Therefore, after obtaining the linear features H... d When [TxD] is reached, a preset overlap rate is obtained, and the linear feature is divided into block feature sequences H according to the preset overlap rate. s :[T s [xSxD], where S is the block length and T is the value of the block length. s The number of blocks. The preset overlap rate can be selected from values ​​such as 50%, 25%, and 33.3%. When the preset overlap rate is 50%, it can ensure that the area of ​​the block boundary can be fully modeled by the model, and skip a block with no overlapping area, thereby further ensuring the efficient processing of the model.

[0092] This embodiment normalizes the encoded features, extracts linear features, and performs block processing through a preprocessing layer. This preprocessing layer enables the extraction of three-dimensional features from two-dimensional temporal feature sequences, further improving the model processing efficiency and speech separation efficiency.

[0093] In some optional implementations of this embodiment, the convolutional attention layer includes a one-dimensional convolutional layer and a multi-head attention mechanism layer. The step of inputting the block feature sequence to the convolutional attention layer of the mask network and calculating the separated convolutional features includes:

[0094] Based on the one-dimensional convolutional layer, the block feature sequence is convolved in the temporal dimension to obtain temporal convolutional features;

[0095] The multi-head attention mechanism layer performs multi-head attention calculation on the block feature sequence to obtain attention features;

[0096] The temporal convolutional features and the attention features are fused to obtain the separated convolutional features.

[0097] In this embodiment, the convolutional attention layer includes a one-dimensional convolutional layer and a multi-head attention mechanism layer. The one-dimensional convolutional layer and the multi-head attention mechanism layer are two parallel network layers. The one-dimensional convolutional layer performs a 1D depth convolution operation on the block feature sequence, while the multi-head attention mechanism layer performs a multi-head attention operation on the block feature sequence according to the perception.

[0098] Specifically, with Figure 4 For example, Figure 4This is a schematic diagram of a convolutional attention layer in a masked network for a speech separation model. It shows the process of obtaining block feature sequences. At that time, based on the one-dimensional convolutional layer, a 1D depthwise convolution operation is performed on the block feature sequence in the temporal dimension to obtain the first convolutional feature. in, in, The first convolutional feature is a block feature sequence; then, a full convolution operation is performed on the pointwise convolutional layer (PointwiseConv1D) of the feature domain dimension to obtain the second convolutional feature. Finally, the first and second convolutional features are normalized using a normalization layer (Add&Norm) to obtain the temporal convolutional features. For the multi-head attention mechanism layer, after obtaining the block feature sequence At that time, multi-head attention is calculated on the block feature sequence according to the multi-head attention mechanism layer to obtain multi-head attention features; then, the multi-head attention features and the block feature sequence are normalized (Add & Norm) through the normalization layer to obtain the final attention feature H. a After obtaining the attention feature H a Temporal convolution features H c At that time, the temporal convolutional features and attention features are fused to obtain the separated convolutional features.

[0099]

[0100] Among them, MultiHeadAttention is a multi-head attention layer, and LayerNorm is a normalization layer. The input is a block feature sequence.

[0101] This embodiment processes the block feature sequence through a one-dimensional convolutional layer and a multi-head attention mechanism layer, and then fuses the obtained features to obtain the final separated convolutional features, thereby achieving accurate processing of speech features and further improving the purity of speech separation.

[0102] In some optional implementations of this embodiment, the step of fusing the temporal convolutional features and the attention features to obtain the separated convolutional features includes:

[0103] Feedforward features are obtained by calculating the temporal convolutional features and the attention features using a feedforward neural network;

[0104] The feedforward features, the temporal convolutional features, and the attention features are normalized to obtain the separated convolutional features.

[0105] In this embodiment, upon obtaining the temporal convolutional features and attention features, feature fusion is performed on these features to obtain the separable convolutional features. Specifically, the temporal convolutional features and attention features are calculated using a feedforward neural network to obtain feedforward features. In this feedforward neural network, each neuron is only connected to the neurons of the previous layer, receiving the output of the previous layer and outputting it to the next layer; there is no direct feedback between layers. Upon obtaining the temporal convolutional features and attention features, element-wise matrix addition is performed on them. The result of this addition is then input into the feedforward neural network, which calculates the feedforward features. Finally, upon obtaining the feedforward features, the temporal convolutional features, and attention features are fused and normalized to obtain the separable convolutional features H. ca .

[0106]

[0107]

[0108] Among them, H a For attention features, H c For temporal convolution features, It is a feedforward feature.

[0109] This embodiment calculates temporal convolutional features and attention features using a feedforward neural network. Then, the temporal convolutional features, attention features, and feedforward features are fused to obtain the separated convolutional features, thereby further realizing the accurate extraction of speech features.

[0110] In some optional implementations of this embodiment, the steps for obtaining the trained speech separation model include:

[0111] Construct a basic separation model and obtain multiple sets of training speech samples;

[0112] The basic separation model is trained based on the training speech samples to obtain the signal-to-noise ratio evaluation value of the training speech samples;

[0113] The basic separation model is iteratively trained based on the signal-to-noise ratio (SNR) evaluation value. When the SNR evaluation value reaches its maximum value, the training of the basic separation model is considered complete, and the trained speech separation model is obtained.

[0114] In this embodiment, the basic separation model has the same structure as the trained speech separation model, but its parameters differ from those of the trained speech separation model. By training the basic separation model and adjusting its parameters, a trained speech separation model is finally obtained. Specifically, default parameters are obtained, a basic separation model is constructed based on these default parameters, and multiple sets of training speech samples are acquired. The basic separation model is then trained using these training speech samples to obtain the signal-to-noise ratio (SI-SNR) of the training speech samples. The SI-SNR is the projection vector of the separation vector calculated by the basic separation model onto the vertical direction of the corresponding real vector in the training speech sample. The closer the separation vector is to the real vector and the more parallel it is, and the greater the deviation in vector direction, the larger the value of the projection vector, meaning the model's result is more in line with expectations. Therefore, the basic separation model is iteratively trained based on this SI-SNR. When the SI-SNR calculated by the basic separation model after a certain iteration reaches its maximum value, the training of the basic separation model is considered complete, thus obtaining the trained speech separation model.

[0115] This embodiment uses signal-to-noise ratio evaluation values ​​to iteratively train the basic separation model, which improves the training efficiency of the model, saves the training time, and reduces the CPU resources required.

[0116] In some optional implementations of this embodiment, the step of iteratively training the basic separation model based on the signal-to-noise evaluation value includes:

[0117] The iterative parameters are calculated based on the signal-to-noise ratio evaluation value. These iterative parameters are then used as shared parameters of the basic masking network in the basic separation model. The basic masking network is then adjusted based on these shared parameters to obtain the adjusted speech separation model.

[0118] In this embodiment, to reduce the number of model training parameters and improve the training speed, parameters of each network in the basic separation model can be shared during training. Specifically, when calculating the signal-to-noise ratio (SNR) evaluation value based on the basic separation model, iterative parameters are calculated based on this SNR evaluation value. These iterative parameters are used as shared parameters of the basic mask network in the basic separation model. This basic mask network has the same structure as the mask network of the trained speech separation model. These shared parameters indicate that the parameters of different network layers in the basic mask network are the same. For example, the network parameters of the one-dimensional convolutional layer and the multi-head attention layer in the basic mask network are shared, meaning that the network parameters of the one-dimensional convolutional layer and the multi-head attention layer in the basic mask network are the same. Alternatively, only the parameters of specified network layers can be shared, such as the parameters of the feedforward layer. Upon obtaining the shared parameters, the parameters of the basic mask network are adjusted based on these shared parameters to obtain the adjusted speech separation model. Taking the sharing of all parameters of the basic mask network in the basic separation model as an example, after sharing the parameters of the basic mask network, the convolutional attention layer in the basic mask network can be represented as follows:

[0119]

[0120] in, This indicates that the iteration module is performed N times, and the network has the same parameters in each iteration.

[0121] This embodiment reduces the number of model parameters to 1 / N of the original network by sharing parameters within the model. This significantly reduces the number of parameters in the entire model, thereby improving both training efficiency and speech separation efficiency while reducing the number of model parameters, thus ensuring the effectiveness of speech separation.

[0122] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0123] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0124] Further reference Figure 5 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a speech separation device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0125] like Figure 5 As shown, the speech separation device 500 described in this embodiment includes: an acquisition module 501, an encoding module 502, a masking module 503, and a decoding module 504. Wherein:

[0126] The acquisition module 501 is used to acquire the trained speech separation model and the speech temporal mixed signal to be separated. The speech separation model includes an encoder, a masking network and a decoder.

[0127] In this embodiment, the speech temporal mixture signal is a temporal mixture signal including multiple different human voices. A trained speech separation model and the speech temporal mixture signal to be separated are obtained. Based on the speech separation model, the speech separation signal of each different speaker can be separated from the speech temporal mixture signal. The speech separation model includes an encoder, a masking network, and a decoder; the encoder is used to encode the input speech temporal mixture signal, the masking network is used to further extract features from the input, and the decoder is used to decode to obtain the separated speech signal corresponding to each different human voice.

[0128] Encoding module 502 is used to perform convolution calculation on the speech temporal mixed signal based on the encoder of the speech separation model to obtain encoded features;

[0129] In this embodiment, the speech separation model is a pre-trained separation model, which can separate the speech signals of each different speaker from the speech of multiple people. The speech separation model includes an encoder, a masking network, and a decoder. When a mixed speech signal in the temporal domain is received, the encoder based on the speech separation model performs convolution calculations on the mixed speech signal to obtain encoded features. Specifically, the encoder θ... enc It consists of a Conv1D convolutional layer followed by an activation function (ReLU); the resulting speech temporal mixed signal X = (x1, x2, ..., x...) is obtained. T When the speech temporal mixed signal is convolved by the Conv1D convolutional layer, the convolution result is obtained; then, the encoding feature H = (h1, h2, ..., h...) is calculated using the ReLU activation function. T ), where T is the length of the speech temporal mixed signal, and the specific calculation formula for this coding feature is as follows:

[0130] H = θ enc (X)=ReLU(Conv1D(X))

[0131] Where, θ enc Let X be the encoder, and let X be the speech time-domain mixed signal.

[0132] In some optional implementations of this embodiment, the encoding module 502 includes:

[0133] The building unit is used to build the basic separation model and obtain multiple sets of training speech samples;

[0134] The first training unit is used to train the basic separation model based on the training speech samples to obtain the signal-to-noise ratio evaluation value of the training speech samples.

[0135] The second training unit is used to iteratively train the basic separation model based on the signal-to-noise ratio evaluation value. When the signal-to-noise ratio evaluation value reaches its maximum value, the training of the basic separation model is determined to be complete, and the trained speech separation model is obtained.

[0136] In this embodiment, the basic separation model has the same structure as the trained speech separation model, but its parameters differ from those of the trained speech separation model. By training the basic separation model and adjusting its parameters, a trained speech separation model is finally obtained. Specifically, default parameters are obtained, a basic separation model is constructed based on these default parameters, and multiple sets of training speech samples are acquired. The basic separation model is then trained using these training speech samples to obtain the signal-to-noise ratio (SI-SNR) of the training speech samples. The SI-SNR is the projection vector of the separation vector calculated by the basic separation model onto the vertical direction of the corresponding real vector in the training speech sample. The closer the separation vector is to the real vector and the more parallel it is, and the greater the deviation in vector direction, the larger the value of the projection vector, meaning the model's result is more in line with expectations. Therefore, the basic separation model is iteratively trained based on this SI-SNR. When the SI-SNR calculated by the basic separation model after a certain iteration reaches its maximum value, the training of the basic separation model is considered complete, thus obtaining the trained speech separation model.

[0137] In some optional implementations of this embodiment, the second training unit includes:

[0138] A shared subunit is used to calculate iterative parameters based on the signal-to-noise ratio evaluation value, use the iterative parameters as shared parameters of the basic masking network in the basic separation model, and adjust the basic masking network based on the shared parameters to obtain the adjusted speech separation model.

[0139] In this embodiment, to reduce the number of model training parameters and improve the training speed, parameters of each network in the basic separation model can be shared during training. Specifically, when calculating the signal-to-noise ratio (SNR) evaluation value based on the basic separation model, iterative parameters are calculated based on this SNR evaluation value. These iterative parameters are used as shared parameters of the basic mask network in the basic separation model. This basic mask network has the same structure as the mask network of the trained speech separation model. These shared parameters indicate that the parameters of different network layers in the basic mask network are the same. For example, the network parameters of the one-dimensional convolutional layer and the multi-head attention layer in the basic mask network are shared, meaning that the network parameters of the one-dimensional convolutional layer and the multi-head attention layer in the basic mask network are the same. Alternatively, only the parameters of specified network layers can be shared, such as the parameters of the feedforward layer. Upon obtaining the shared parameters, the parameters of the basic mask network are adjusted based on these shared parameters to obtain the adjusted speech separation model. Taking the sharing of all parameters of the basic mask network in the basic separation model as an example, after sharing the parameters of the basic mask network, the convolutional attention layer in the basic mask network can be represented as follows:

[0140]

[0141] in, This indicates that the iteration module is performed N times, and the network has the same parameters in each iteration.

[0142] Masking module 503 is used to input the encoded features into the masking network of the speech separation model, and to extract features from the encoded features according to the masking network to obtain the masking matrix corresponding to each different human voice in the speech temporal mixed signal;

[0143] In some optional implementations of this embodiment, the mask module 503 includes:

[0144] The feature extraction unit is used to perform linear feature extraction and block processing on the encoded features according to the preprocessing layer of the mask network to obtain a block feature sequence;

[0145] The computation unit is used to input the block feature sequence into the convolutional attention layer of the mask network and calculate the separated convolutional features;

[0146] The stacking unit is used to stack the separated convolutional features based on the stacking layers of the mask network to obtain the mask matrix.

[0147] In some optional implementations of this embodiment, the feature extraction unit includes:

[0148] A normalization subunit is used to normalize the encoded features according to the normalization layer to obtain normalized features;

[0149] The feature extraction subunit is used to perform linear feature extraction on the normalized features based on the linear layer to obtain linear features;

[0150] A molecular unit is used to obtain a preset overlap rate, and the linear feature is divided into the block feature sequence from the time domain dimension according to the preset overlap rate.

[0151] In some optional implementations of this embodiment, the computing unit includes:

[0152] The first computational subunit is used to perform one-dimensional convolution calculation on the block feature sequence in the temporal dimension according to the one-dimensional convolutional layer to obtain temporal convolutional features;

[0153] The second computational subunit is used to perform multi-head attention computation on the block feature sequence according to the multi-head attention mechanism layer to obtain attention features;

[0154] The feature fusion subunit is used to fuse the temporal convolutional features and the attention features to obtain the separated convolutional features.

[0155] In some optional implementations of this embodiment, the feature fusion subunit includes:

[0156] The third computational subunit is used to calculate the feedforward features based on the temporal convolutional features and the attention features of the feedforward neural network;

[0157] The fourth computational subunit is used to normalize the feedforward features, the temporal convolutional features, and the attention features to obtain the separated convolutional features.

[0158] In this embodiment, the masking network includes a preprocessing layer, a convolutional attention layer, and a stacking layer. The preprocessing layer can segment the temporal signal of the speech into small signal blocks. The convolutional attention layer can perform convolution operations within and between signal blocks on the signal blocks. The stacking layer can stack the features calculated by the convolutional attention layer to finally obtain a mask matrix for each different human voice in the temporal signal of the speech. One mask matrix corresponds to one human voice (i.e., one speaker).

[0159] Specifically, as follows: Figure 3 As shown, Figure 3 This diagram illustrates a masking network in a speech separation model. The masking network includes a preprocessing layer (Chunk), an intra-block convolutional attention layer (Intra Conv-Attn), a permute layer, an inter-block convolutional attention layer (InterConv-Attn), and an overlap-add layer. When obtaining the encoded features H of the temporal mixed speech signal, these features are passed through the preprocessing layer to obtain the block feature sequence H. s The feature sequence of a feature block is calculated using an intra-block convolutional attention layer (Intra Conv-Attn) to obtain local features. These local features are then transformed to a feature dimension P using a transformation layer, yielding transformed features. Inter-block convolutional attention layers (Inter Conv-Attn) are then used to model the long-term dependencies between these transformed features, resulting in inter-block features. These inter-block features are then input into a stacking layer, which performs feature transformations to obtain the temporal features H. o The mask matrix is ​​finally obtained based on this time-domain feature. Figure 3 N in mask This represents the number of times the loop will repeat.

[0160] The decoding module 504 is used to calculate the product of the mask matrix and the coding features according to the decoder of the speech separation model to obtain the separated speech signal corresponding to each human voice.

[0161] In this embodiment, after obtaining the multiple separated mask matrices, the decoder of the speech separation model calculates the product of each mask matrix and the encoded feature, thus obtaining the separated speech signal corresponding to each different human voice (speaker). Specifically, the decoder consists of transposed convolution (Conv1D). T The mask matrix and the encoded feature are composed of H. The separated speech signal is obtained by calculating the mask matrix and the encoded feature using the transpose matrix. The calculation formula for this separated speech signal is as follows:

[0162]

[0163] in, To separate the speech signal, M 1:K Given K mask matrices, M 1:K ={M1,M2,...,M K}, H represents the encoding feature, and ⊙ represents the element-wise matrix multiplication operation.

[0164] It should be emphasized that, in order to further ensure the privacy and security of the aforementioned separated voice signals, the separated voice signals can also be stored in a blockchain node.

[0165] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0166] The speech separation device proposed in this embodiment achieves accurate speech separation, improves the efficiency of speech separation, and greatly reduces the number of model parameters, thereby improving the training efficiency of the model and the CPU memory required for speech separation. This allows the model to be applied in resource-constrained scenarios, further improving the speed of speech separation.

[0167] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed] for details. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0168] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected via a system bus. It should be noted that only the computer device 6 with components 61-63 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0169] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0170] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 6. Of course, the memory 61 may include both the internal storage unit and its external storage device of the computer device 6. In this embodiment, the memory 61 is typically used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions for speech separation methods. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or will be output.

[0171] In some embodiments, the processor 62 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 62 is typically used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to execute computer-readable instructions stored in the memory 61 or to process data, for example, to execute computer-readable instructions for the speech separation method.

[0172] The network interface 63 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 6 and other electronic devices.

[0173] The computer device proposed in this embodiment achieves accurate speech separation, improves the efficiency of speech separation, and greatly reduces the number of model parameters, thereby improving the training efficiency of the model and the CPU and video memory required for speech separation. This allows the model to be applied in resource-constrained scenarios, further improving the speed of speech separation.

[0174] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the speech separation method described above.

[0175] The computer-readable storage medium proposed in this embodiment achieves accurate speech separation, improves the efficiency of speech separation, and greatly reduces the number of model parameters, thereby improving the training efficiency of the model and the CPU memory required for speech separation. This allows the model to be applied in resource-constrained scenarios, further improving the speed of speech separation.

[0176] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0177] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A speech separation method, characterized in that, Includes the following steps: The trained speech separation model and the mixed temporal signal of the speech to be separated are obtained. The speech separation model includes an encoder, a masking network, and a decoder. The encoder based on the speech separation model performs convolution calculation on the speech temporal mixed signal to obtain the encoded features; The encoded features are input into the masking network of the speech separation model, and feature extraction is performed on the encoded features according to the masking network to obtain the masking matrix corresponding to each different human voice in the speech temporal mixed signal; The decoder of the speech separation model calculates the product of the mask matrix and the coding features to obtain the separated speech signal corresponding to each human voice. The masking network includes a preprocessing layer, a convolutional attention layer, and a stacking layer. The step of extracting features from the encoded features based on the masking network to obtain the mask matrix corresponding to each different human voice in the speech temporal mixed signal includes: The encoded features are linearly extracted and segmented according to the preprocessing layer of the mask network to obtain a block feature sequence; The block feature sequence is input into the convolutional attention layer of the mask network to calculate the separated convolutional features; The separated convolutional features are superimposed based on the superposition layer of the mask network to obtain the mask matrix; The preprocessing layer includes a normalization layer and a linear layer. The step of performing linear feature extraction and block segmentation on the encoded features based on the preprocessing layer of the mask network to obtain a block feature sequence includes: The encoded features are normalized according to the normalization layer to obtain normalized features; Linear features are obtained by extracting linear features from the normalized features based on the linear layer. A preset overlap rate is obtained, and the linear feature is divided into the block feature sequence from the time domain dimension according to the preset overlap rate, wherein the preset overlap rate is 50%.

2. The speech separation method according to claim 1, characterized in that, The convolutional attention layer includes a one-dimensional convolutional layer and a multi-head attention mechanism layer. The step of inputting the block feature sequence into the convolutional attention layer of the mask network and calculating the separated convolutional features includes: Based on the one-dimensional convolutional layer, the block feature sequence is convolved in the temporal dimension to obtain temporal convolutional features; The multi-head attention mechanism layer performs multi-head attention calculation on the block feature sequence to obtain attention features; The temporal convolutional features and the attention features are fused to obtain the separated convolutional features.

3. The speech separation method according to claim 2, characterized in that, The step of fusing the temporal convolutional features and the attention features to obtain the separated convolutional features includes: Feedforward features are obtained by calculating the temporal convolutional features and the attention features using a feedforward neural network; The feedforward features, the temporal convolutional features, and the attention features are normalized to obtain the separated convolutional features.

4. The speech separation method according to claim 1, characterized in that, The steps for obtaining the trained speech separation model include: Construct a basic separation model and obtain multiple sets of training speech samples; The basic separation model is trained based on the training speech samples to obtain the signal-to-noise ratio evaluation value of the training speech samples; The basic separation model is iteratively trained based on the signal-to-noise ratio (SNR) evaluation value. When the SNR evaluation value reaches its maximum value, the training of the basic separation model is considered complete, and the trained speech separation model is obtained.

5. The speech separation method according to claim 4, characterized in that, The step of iteratively training the basic separation model based on the signal-to-noise evaluation value includes: The iterative parameters are calculated based on the signal-to-noise ratio evaluation value. These iterative parameters are then used as shared parameters of the basic masking network in the basic separation model. The basic masking network is then adjusted based on these shared parameters to obtain the adjusted speech separation model.

6. A speech separation device, characterized in that, include: The acquisition module is used to acquire the trained speech separation model and the speech temporal mixed signal to be separated. The speech separation model includes an encoder, a masking network, and a decoder. The encoding module is used to perform convolution calculation on the speech temporal mixed signal based on the encoder of the speech separation model to obtain encoded features; The masking module is used to input the encoded features into the masking network of the speech separation model, and to extract features from the encoded features according to the masking network to obtain the masking matrix corresponding to each different human voice in the speech temporal mixed signal. The decoding module is used to calculate the product of the mask matrix and the coding features based on the decoder of the speech separation model to obtain the separated speech signal corresponding to each human voice; The mask module includes: The sequence acquisition unit is used to perform linear feature extraction and block processing on the encoded features according to the preprocessing layer of the mask network to obtain a block feature sequence; A convolutional computation unit is used to input the block feature sequence into the convolutional attention layer of the mask network and calculate the separated convolutional features; The feature stacking unit is used to stack the separated convolutional features based on the stacking layer of the mask network to obtain the mask matrix; The sequence acquisition unit includes: A normalization subunit is used to normalize the encoded features according to the normalization layer to obtain normalized features; The feature extraction subunit is used to perform linear feature extraction on the normalized features based on the linear layer to obtain linear features; The feature segmentation unit is used to obtain a preset overlap rate, and the linear feature is segmented into the block feature sequence from the time domain dimension according to the preset overlap rate, wherein the preset overlap rate is 50%.

7. A computer device comprising a memory and a processor, the memory storing computer-readable instructions, wherein the processor, when executing the computer-readable instructions, implements the steps of the speech separation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech separation method as described in any one of claims 1 to 5.

Images