Voice separation method and model, computer equipment and storage medium

Abstract

The invention relates to a voice separation method and model, computer equipment and a storage medium. The method comprises the following steps: acquiring multi-channel audio sampling data; inputting the multi-channel audio sampling data into an encoder for encoding to obtain two-dimensional time-frequency expression data corresponding to the multi-channel audio sampling data; inputting the two-dimensional time-frequency expression data into a graph convolutional neural network for spatial feature extraction to obtain spatial feature data corresponding to the two-dimensional time-frequency expression data; inputting the spatial feature data into a time-frequency mask network for time-frequency mask processing to obtain time-frequency mask data corresponding to each speaker sound source; and inputting the time-frequency mask data and the spatial feature data respectively corresponding to each speaker sound source into a decoder for decoding processing to obtain single-sound-source audio data respectively corresponding to each speaker sound source output by the decoder. By adopting the method, the voice separation effect can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of speech data processing, and in particular to a speech separation method, model, computer device and storage medium. Background Art

[0002] With the development of speech data processing technology, speech separation technology has emerged. Speech separation technology refers to separating the speech signals corresponding to each speaker from the mixed speech signals of multiple speakers. This problem is a challenging task in the field of speech data processing because the speech signals of multiple speakers overlap in time and frequency, and they may have similar spectral characteristics.

[0003] The traditional method of speech separation is to predict the mask data corresponding to the sound source of a single speaker through deep learning, and multiply the mask data by the frequency point value to achieve speech separation. Although this method is easy to implement, it does not have an ideal effect on phase processing, resulting in poor quality of speech separation. Summary of the invention

[0004] Based on this, it is necessary to provide a speech separation method, model, computer device and storage medium that can improve the quality of speech separation in response to the above technical problems.

[0005] A speech separation method is implemented based on a speech separation model, wherein the speech separation model includes an encoder, a graph convolutional neural network, a time-frequency mask network, and a decoder. The method includes:

[0006] Acquire multi-channel audio sampling data, where the multi-channel audio sampling data is audio sampling data of a mixed sound source from multiple speaker sound sources collected through multiple audio sampling channels;

[0007] Input the multi-channel audio sampling data into the encoder for encoding processing to obtain two-dimensional time-frequency expression data corresponding to the multi-channel audio sampling data;

[0008] Input the two-dimensional time-frequency expression data into the graph convolutional neural network to extract spatial features, and obtain spatial feature data corresponding to the two-dimensional time-frequency expression data;

[0009] Input the spatial feature data into the time-frequency mask network for time-frequency mask processing to obtain the time-frequency mask data corresponding to each speaker's voice source;

[0010] The time-frequency mask data and spatial feature data corresponding to the voice sources of each speaker are input into the decoder for decoding processing, and the single sound source audio data corresponding to the voice sources of each speaker are output by the decoder.

[0011] In some embodiments, the speech separation model further includes a beamforming network, and the method further includes: inputting the single sound source audio data into the beamforming network for speech enhancement processing, to obtain enhanced single sound source audio data output by the beamforming network.

[0012] In some embodiments, a method for performing speech enhancement processing on single-source audio data using a beamforming network includes: determining a reference channel from multiple audio sampling channels; calculating a speech signal covariance matrix and an interference signal covariance matrix based on the single-source audio data and the multi-channel audio sampling data; based on a minimum mean square distortion-free beamforming algorithm, using the speech signal covariance matrix, the interference signal covariance matrix and the reference channel as variables, performing enhancement processing on the single-source audio data to obtain enhanced single-source audio data.

[0013] In some embodiments, a speech signal covariance matrix and an interference signal covariance matrix are calculated based on single-source audio data and multi-channel audio sampling data, including: performing a short-time Fourier transform on the multi-channel audio sampling data to obtain a first change matrix; performing a short-time Fourier transform on the single-source audio data to obtain a second change matrix; calculating the speech signal covariance matrix based on the second change matrix and a transposed matrix of the second change matrix; and calculating the interference signal covariance matrix based on a difference matrix between the first change matrix and the second change matrix and a transposed matrix of the difference matrix.

[0014] In some embodiments, the method further comprises: iteratively transmitting the enhanced single-source audio data to an encoder, so that the encoder adjusts the two-dimensional time-frequency representation data according to the enhanced single-source audio data and the multi-channel audio sampling data.

[0015] In some embodiments, the method further includes: obtaining the output of the decoder after the first iteration as the first output; obtaining the output of the decoder after the second iteration as the second output; calculating the loss function based on the first output and the second output, and performing parameter optimization training on the speech separation model based on the loss function.

[0016] In some embodiments, the time-frequency mask data and spatial feature data corresponding to each speaker's voice source are input into a decoder for decoding processing to obtain the single sound source audio data corresponding to each speaker's voice source output by the decoder, including: performing Hadamard product operation on the time-frequency mask data corresponding to each speaker's voice source and the spatial feature data respectively; inputting the product operation data after the Hadamard product operation into the decoder, and using the decoder to perform a transposed convolution operation on the product operation data to obtain the single sound source audio data corresponding to each speaker's voice source.

[0017] In some embodiments, the graph convolutional neural network includes multiple graph convolutional layers, and the two-dimensional time-frequency expression data is input into the graph convolutional neural network to extract spatial features. The formula for obtaining spatial feature data corresponding to the two-dimensional time-frequency expression data is:

[0018]

[0019] in, represents the spatial feature data output by the graph convolution layer of layer l, g represents the graph convolution function, represents the diagonal node degree matrix, is the adjacency matrix, is the weight matrix of the graph convolution of the l-1th layer, Represents two-dimensional time-frequency expression data.

[0020] A speech separation model, the model comprising:

[0021] An encoder is used to obtain multi-channel audio sampling data, where the multi-channel audio sampling data is audio sampling data of a mixed sound source from multiple speakers collected through multiple audio sampling channels, and encode the multi-channel audio sampling data to obtain two-dimensional time-frequency expression data corresponding to the multi-channel audio sampling data;

[0022] A graph convolutional neural network is used to perform graph convolution operations on the two-dimensional time-frequency expression data to extract spatial features and obtain spatial feature data corresponding to the two-dimensional time-frequency expression data;

[0023] A time-frequency masking network is used to perform time-frequency masking processing on the spatial feature data to obtain time-frequency masking data corresponding to the voice sources of each speaker;

[0024] The decoder is used to decode the time-frequency mask data and spatial feature data corresponding to each speaker's voice source to obtain the single sound source audio data corresponding to each speaker's voice source.

[0025] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any one of the above-mentioned speech separation methods is implemented.

[0026] A computer-readable storage medium stores a computer program, which implements any one of the above-mentioned speech separation methods when executed by a processor.

[0027] The above-mentioned speech separation method, model, computer device and storage medium encode the multi-channel audio sampling data of the mixed sound source to obtain the corresponding two-dimensional time-frequency expression data, then input the two-dimensional time-frequency expression data into the graph convolutional neural network for spatial feature extraction, and perform time-frequency masking on the extracted spatial feature data to obtain the time-frequency masking data corresponding to each speaker's sound source, and then obtain the single sound source audio data corresponding to each speaker's sound source after separation according to the time-frequency masking data and spatial feature data corresponding to each speaker's sound source. By adopting this scheme, spatial feature extraction can be performed based on the graph convolutional neural network to construct the correlation between multiple channels, so that spatial features containing the correlation between multiple channels can be extracted from the multi-channel audio sampling data of the mixed sound source, and the spatial feature data is combined with the time-frequency masking process to retain the construction of the long-term context dependency of the time-frequency masking process, that is, while giving full play to the advantages of speech separation based on time-frequency masking, the spatial dependency of data between multiple channels is combined, so that the effect and quality of speech separation can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of the structure of a speech separation model in some embodiments;

[0029] Figure 2 A flowchart of a speech separation method in some embodiments;

[0030] Figure 3 A schematic diagram of the structure of a speech separation model in some other embodiments;

[0031] Figure 4 A schematic diagram of the structure of a speech separation model in some other embodiments;

[0032] Figure 5 1 is a diagram of the internal structure of a computer device in some embodiments. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0034] The speech separation method provided in the present application can be implemented based on a computer device, wherein the computer device can be a terminal or a server. The server can be implemented with an independent server or a server cluster composed of multiple servers. The terminal can be but is not limited to various vehicle-mounted terminals, smart terminals, personal computers, laptops, smart phones, tablet computers and portable wearable devices, etc.

[0035] In some embodiments, a speech separation model is provided, see Figure 1 As shown, Figure 1 A schematic diagram of the structure of a speech separation model in some embodiments is shown, and the speech separation model 100 may include an encoder 102, a graph convolutional neural network 104, a time-frequency mask network 106, and a decoder 108. Among them, multi-channel audio sampling data from multiple mixed sound sources is input to the input end of the encoder 102, the output end of the encoder 102 is connected to the input end of the graph convolutional neural network 104, the output end of the graph convolutional neural network 104 is connected to the input end of the time-frequency mask network 106, the output end of the time-frequency mask network 106 is connected to the input end of the decoder 108, and the output end of the decoder 108 outputs the separated single sound source audio data.

[0036] In one embodiment, Figure 2 As shown, a speech separation method is provided, which is applied to Figure 1 The speech separation model in is taken as an example to illustrate, including the following steps:

[0037] Step S202: Acquire multi-channel audio sampling data.

[0038] The multi-channel audio sampling data is audio sampling data of a mixed sound source from multiple speaker sound sources collected through multiple audio sampling channels.

[0039] The sound source refers to the source of the sound. For example, if there are two speakers, it can be considered that there are two speaker sound sources. Multi-channel refers to multiple audio sampling channels. For example, a multi-channel microphone can be used to collect multi-channel audio signals to obtain multi-channel audio sampling data.

[0040] Specifically, a multi-channel microphone may be used to collect audio data. For example, the multi-channel microphone may have C audio sampling channels, where C is a positive integer greater than 1. The audio data may come from multiple sources, for example, multiple speakers, i.e., multiple speaker sound sources. Thus, multi-channel audio sampling data including a mixed sound source from multiple speaker sound sources is obtained.

[0041] Step S204: input the multi-channel audio sampling data into the encoder for encoding processing to obtain two-dimensional time-frequency expression data corresponding to the multi-channel audio sampling data.

[0042] The encoder is used to convert the audio data into data expressed in two-dimensional time and frequency to achieve encoding processing of the audio data. Exemplarily, a parallel encoder can be used to improve the encoding efficiency of multi-channel audio sampling data.

[0043] Exemplarily, the encoder may include at least one 1D (Dimensional) convolution layer, and the specific formula for encoding through the 1D convolution layer may refer to the following:

[0044] R c =ParEnc({y c} c ,c)

[0045] in, is the microphone signal of the cth channel of a mixed sound source including the sound sources from multiple speakers, x s,c is the microphone signal of the cth channel from the sound source s, where c is the channel, S=N ​​represents N speaker sound sources (e.g., S=2 represents 2 speaker sound sources), and R c is the two-dimensional time-frequency expression data of the c-th channel after encoding. c} c The data represents the collection of microphone signals of C channels, that is, multi-channel audio sampling data.

[0046] Step S206: Input the two-dimensional time-frequency expression data into the graph convolutional neural network to extract spatial features, and obtain spatial feature data corresponding to the two-dimensional time-frequency expression data.

[0047] Specifically, the graph convolutional neural network can be trained based on a deep learning algorithm, and the graph convolutional neural network can include multiple graph convolutional layers, which can be used to perform graph convolution operations on input data to extract spatial feature data of the input data. Among them, the spatial feature data extracted by the graph convolutional neural network can reflect the channel characteristics between different audio sampling channels, and thus can reflect the spatial characteristics of the audio sampling data.

[0048] Exemplarily, a graph convolutional neural network may include multiple graph convolutional layers, each of which is connected in series, that is, the output of one graph convolutional layer can be used as the input of the next graph convolutional layer, thereby improving the accuracy of spatial feature extraction. Furthermore, in order to improve the accuracy, a dropout module, a ReLU module, etc. may be added between graph convolutional layers.

[0049] In some embodiments, the graph convolutional neural network includes multiple graph convolutional layers, and the two-dimensional time-frequency expression data is input into the graph convolutional neural network to extract spatial features. The specific formula for obtaining the spatial feature data corresponding to the two-dimensional time-frequency expression data can be:

[0050]

[0051] in, represents the spatial feature data output by the graph convolution layer of layer l, g represents the graph convolution operation function, represents the diagonal node degree matrix, is the adjacency matrix, is the weight matrix of the graph convolution of the l-1th layer, Represents the input two-dimensional time-frequency expression data.

[0052] More specifically, the diagonal node degree matrix can be constructed on the graph convolution layer based on the graph convolution operation principle: Adjacency Matrix Weight Matrix Among them, the adjacency matrix and the weight matrix The parameters of are learnable parameters. The graph convolutional neural network of the speech separation model can be trained in advance with sample data to obtain the adjacency matrix after parameter learning. and the weight matrix Adjacency matrix after parameter learning and the weight matrix Perform spatial feature extraction on the input two-dimensional time-frequency expression data.

[0053] Exemplarily, the sample data for training can be obtained by the following method: by applying room impulse response and WSJ speech to construct an audio data set of mixed multi-speaker sound sources used for model training, the signal-to-noise ratio of the audio is between -5db and 5db.

[0054] Step S208: inputting the spatial feature data into a time-frequency masking network for time-frequency masking processing to obtain time-frequency masking data corresponding to the voice sources of each speaker.

[0055] Specifically, the spatial feature data can be subjected to time-frequency masking processing, which can be implemented specifically through a time-frequency masking network. Through the time-frequency masking processing, the audio sampling data of the mixed sound source from different speaker sound sources can be separated, and the time-frequency masking data corresponding to each speaker sound source can be obtained. Among them, the time-frequency masking network can include a time-domain convolution layer (for example, a 1D convolution layer) and a masking network (for example, a Masks layer). The input data is subjected to time-domain convolution processing through the time-domain convolution layer, and the time-frequency masking processing is performed through the masking network.

[0056] Exemplarily, the formula corresponding to the time-frequency mask processing can be expressed as:

[0057]

[0058] in, is the time-frequency mask data of the multi-channel s-th speaker sound source; Seperator is a string processing function. is the time-frequency mask data of the s-th source speaker of the c-th channel. Represents the spatial feature data output by the graph convolutional layer at layer l.

[0059] Exemplarily, the time-frequency mask network may include multiple 1D time-domain convolutional layers, Layer Norm layers, PReLU activation function layers, Sigmoid activation function layers, and a masking network. Figure 4 As shown, Figure 4 FIG. 1 shows a schematic diagram of the structure of the speech separation model in some other embodiments, wherein the time-frequency mask network can be used as a separator, and its internal structure is shown in FIG. The connection relationship of each unit and the data flow can be referred to in FIG. Figure 4 shown.

[0060] Step S210: input the time-frequency mask data and spatial feature data corresponding to the voice sources of each speaker into the decoder for decoding processing, and obtain the single sound source audio data corresponding to the voice sources of each speaker output by the decoder.

[0061] Specifically, the time-frequency mask data corresponding to each speaker's voice source and the spatial feature data can be multiplied, where the product operation can be an ordinary product operation or a Hadamard product operation, and then the data output after the product operation is decoded. The two-dimensional time-frequency expressed data can be restored to normal audio data through the decoding process, thereby obtaining the single sound source audio data corresponding to each speaker's voice source separated and restored from the multi-channel audio sampling data of the mixed sound source.

[0062] In some embodiments, the time-frequency mask data and spatial feature data corresponding to each speaker's voice source are input into a decoder for decoding processing to obtain the single sound source audio data corresponding to each speaker's voice source output by the decoder, including: performing Hadamard product operation on the time-frequency mask data corresponding to each speaker's voice source and the spatial feature data respectively; inputting the product operation data after the Hadamard product operation into the decoder, and using the decoder to perform a transposed convolution operation on the product operation data to obtain the single sound source audio data corresponding to each speaker's voice source.

[0063] In this embodiment, the single sound source audio data corresponding to each speaker's voice source output by the decoder and the spatial feature data output by the graph convolutional neural network can be processed by Hadamard product operation, and the data processed by Hadamard product operation can be input into the decoder for 1D transposed convolution operation, so as to output the restored single sound source audio data.

[0064] By performing Hadamard product operations on the single sound source audio data corresponding to each speaker's sound source output by the decoder and the spatial feature data output by the graph convolutional neural network, the spatial features of the data can be further combined to improve the accuracy of speech separation.

[0065] More specifically, the decoder of the speech separation model may include at least one 1D transposed convolution layer. The specific corresponding formula can be referred to as follows:

[0066]

[0067] Where ⊙ represents the Hadamard product operation, represents the time-frequency mask data of the sound source of the sth speaker, The spatial feature data output by the graph convolution layer of the lth layer, that is, the output of the graph convolutional neural network, Dec represents the transposed convolution operation.

[0068] The above-mentioned speech separation method encodes the multi-channel audio sampling data of the mixed sound source to obtain the corresponding two-dimensional time-frequency expression data, then inputs the two-dimensional time-frequency expression data into the graph convolutional neural network for spatial feature extraction, and performs time-frequency masking processing on the extracted spatial feature data to obtain the time-frequency masking data corresponding to each speaker's sound source, and then obtains the single sound source audio data corresponding to each speaker's sound source after separation according to the time-frequency masking data and spatial feature data corresponding to each speaker's sound source. By adopting this scheme, spatial feature extraction can be performed based on the graph convolutional neural network to construct the correlation between multiple channels, so that spatial features containing the correlation between multiple channels can be extracted from the multi-channel audio sampling data of the mixed sound source, and the spatial feature data is combined with the time-frequency masking processing, which can retain the construction of the long-term context dependency of the time-frequency masking processing, that is, while giving full play to the advantages of speech separation based on time-frequency masking, the spatial dependency of data between multiple channels is combined, so that the effect and quality of speech separation can be improved.

[0069] In some embodiments, reference Figure 3 As shown, Figure 3 A structural schematic diagram of a speech separation model in some other embodiments is shown, wherein the speech separation model also includes a beamforming network 110, and the method also includes: inputting single sound source audio data into the beamforming network 110 for speech enhancement processing, and obtaining enhanced single sound source audio data output by the beamforming network 110.

[0070] In some embodiments, the specific steps of the beamforming network 110 performing speech enhancement processing on single-source audio data include: determining a reference channel from multiple audio sampling channels, and calculating the speech signal covariance matrix and the interference signal covariance matrix based on the single-source audio data and the multi-channel audio sampling data, and based on the minimum mean square distortion-free beamforming algorithm, using the speech signal covariance matrix, the interference signal covariance matrix and the reference channel as variables, performing enhancement processing on the single-source audio data to obtain enhanced single-source audio data.

[0071] Specifically, the beamforming network 110 may include an MVDR (Minimum Variance Distortionless Response) beam former, and the separated single sound source audio data output by the decoder 108 is further input into the MVDR beam former, and the speech signal covariance matrix corresponding to the single sound source audio data is calculated by the MVDR beam former, and the interference signal covariance matrix is ​​calculated according to the single sound source audio data and the multi-channel audio sampling data, and the single sound source audio data is subjected to speech enhancement processing by the speech signal covariance matrix and the interference signal covariance matrix. By adding the beamforming network 110, the separated single sound source speech data can be further enhanced, thereby improving the quality of speech separation.

[0072] More specifically, based on the minimum mean square distortion-free wave velocity formation algorithm, the speech signal covariance matrix, the interference signal covariance matrix and the reference channel are used as variables to enhance the single sound source audio data. The specific formula for obtaining the enhanced single sound source audio data can be referred to as follows:

[0073]

[0074] in, represents the single source speech data of the sth speaker’s voice source, and c represents the reference channel, which can be used as a hotspot vector.

[0075] In some embodiments, the specific steps of calculating the speech signal covariance matrix based on the single sound source audio data include: performing a short-time Fourier transform on the single sound source audio data to obtain a second change matrix; and calculating the speech signal covariance matrix based on the second change matrix and the transposed matrix of the second change matrix. Exemplarily, the specific formula can be referred to as follows:

[0076]

[0077] in, represents the speech signal covariance matrix corresponding to the single sound source audio data, represents the single source audio data of the s-th speaker source of C channels, express The matrix obtained after short-time Fourier transform is the second change matrix. express The transposed matrix of , s represents the sound source of the sth speaker, t represents time, and f represents frequency.

[0078] In some embodiments, the specific steps of calculating the interference signal covariance matrix based on the single sound source audio data and the multi-channel audio sampling data include: performing a short-time Fourier transform on the multi-channel audio sampling data to obtain a first change matrix; performing a short-time Fourier transform on the single sound source audio data to obtain a second change matrix; and calculating the interference signal covariance matrix based on the difference matrix between the first change matrix and the second change matrix and the transposed matrix of the difference matrix. Exemplarily, the specific formula can be referred to as follows:

[0079]

[0080] in, represents the interference signal covariance matrix, Y represents {y c} c The matrix obtained after short-time Fourier transform is the first transformation matrix. c}c 表 shows multi-channel audio sampling data of a mixed sound source of C channels, represents the second change matrix, s, t, f are variables, represents the difference matrix between the first change matrix and the second change matrix, Represents the transposed matrix of the difference matrix.

[0081] In some embodiments, reference Figure 3 and Figure 4 As shown, the method further includes: iteratively transmitting the enhanced single-source audio data to an encoder, so that the encoder adjusts the two-dimensional time-frequency expression data according to the enhanced single-source audio data and the multi-channel audio sampling data.

[0082] In this embodiment, the enhanced single sound source audio data output by the beamforming network 110 can be further fed back to the encoder 102, thereby forming a cyclic iterative network structure. By iteratively forming a cyclic iterative network structure through the structure of a single speech separation model, the quality of speech separation can be further improved while reducing the number of parameters of the network model.

[0083] More specifically, in order to obtain better enhancement and separation effects, an iterative signal extraction process can be performed. For details, please refer to the following expression:

[0084] If the single sound source audio data is obtained from the multi-channel audio sampling data by using the speech separation model, that is, from {y c} c estimate The process can be expressed as:

[0085]

[0086] Among them, (1) represents the first iteration.

[0087] Then, the process after n iterations can be expressed as:

[0088]

[0089] Where n is the number of iterations.

[0090] In some embodiments, the method further includes: obtaining the output of the decoder after the first iteration as the first output; obtaining the output of the decoder after the second iteration as the second output; calculating the loss function based on the first output and the second output, and performing parameter optimization training on the speech separation model based on the loss function.

[0091] refer to Figure 3 and Figure 4 As shown in FIG. 1 , if the current output of the speech separation model is taken as the first output, then the output of the speech separation model next to the current output can be taken as the second output, and the loss function is calculated based on the first output and the second output, and the parameters of each network structure of the speech separation model are optimized using the loss function. More specifically, the loss function It can be expressed as:

[0092]

[0093] in, is the output of the decoder of the speech separation model after the first iteration, is the output of the decoder of the speech separation model after the second iteration.

[0094] By optimizing the outputs after two iterations and minimizing the loss function of the permutation-invariant signal-to-noise ratio, the parameter accuracy of the speech separation model can be improved, thereby further improving the quality and effect of speech separation.

[0095] By adopting the scheme of the above-mentioned embodiment, at least the following effects can be achieved: the dependency between channels can be modeled based on the graph convolution operation mechanism, and the long-term dependency between contexts can be modeled based on the time-frequency mask network. Moreover, the permutation-invariant signal-to-noise ratio loss function technology is combined to achieve the model training results that are more in line with expectations. In addition, a single model structure is adopted, and enhancement and refinement are performed in an iterative manner, which reduces the number of parameters of the speech separation model while improving the speech separation effect.

[0096] It should be understood that although Figure 2The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0097] In one embodiment, Figure 1 As shown, a speech separation model is provided, including: an encoder 102, a graph convolutional neural network 104, a time-frequency mask network 106 and a decoder 108. Among them:

[0098] The encoder 102 is used to obtain multi-channel audio sampling data, where the multi-channel audio sampling data is audio sampling data of a mixed sound source from multiple speakers collected through multiple audio sampling channels, and encode the multi-channel audio sampling data to obtain two-dimensional time-frequency expression data corresponding to the multi-channel audio sampling data;

[0099] A graph convolution neural network 104 is used to perform a graph convolution operation on the two-dimensional time-frequency expression data to extract spatial features, and obtain spatial feature data corresponding to the two-dimensional time-frequency expression data;

[0100] A time-frequency mask network 106, configured to perform time-frequency mask processing on the spatial feature data to obtain time-frequency mask data corresponding to the voice sources of each speaker;

[0101] The decoder 108 is used to decode the time-frequency mask data and the spatial feature data corresponding to each speaker's voice source to obtain the single sound source audio data corresponding to each speaker's voice source.

[0102] For the specific definition of the speech separation model, please refer to the definition of the speech separation method above, which will not be repeated here. Each module in the above speech separation model can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0103] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a speech separation method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0104] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0105] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: acquiring multi-channel audio sampling data, wherein the multi-channel audio sampling data is audio sampling data of a mixed sound source from multiple speaker sound sources collected through multiple audio sampling channels; inputting the multi-channel audio sampling data into an encoder for encoding processing to obtain two-dimensional time-frequency expression data corresponding to the multi-channel audio sampling data; inputting the two-dimensional time-frequency expression data into a graph convolutional neural network for spatial feature extraction to obtain spatial feature data corresponding to the two-dimensional time-frequency expression data; inputting the spatial feature data into a time-frequency mask network for time-frequency mask processing to obtain time-frequency mask data corresponding to each speaker sound source; inputting the time-frequency mask data and the spatial feature data corresponding to each speaker sound source into a decoder for decoding processing to obtain single sound source audio data corresponding to each speaker sound source output by the decoder.

[0106] In one embodiment, when the processor executes the computer program, the following steps are further implemented: inputting the single sound source audio data into the beamforming network for speech enhancement processing, and obtaining enhanced single sound source audio data output by the beamforming network.

[0107] In one embodiment, when the processor executes the computer program, the following steps are also implemented: determining a reference channel from multiple audio sampling channels; calculating a speech signal covariance matrix and an interference signal covariance matrix based on the single sound source audio data and the multi-channel audio sampling data; based on a minimum mean square distortion-free wave velocity formation algorithm, using the speech signal covariance matrix, the interference signal covariance matrix and the reference channel as variables, performing enhancement processing on the single sound source audio data to obtain enhanced single sound source audio data.

[0108] In one embodiment, when the processor executes the computer program, the following steps are also implemented: performing a short-time Fourier transform on the multi-channel audio sampling data to obtain a first change matrix; performing a short-time Fourier transform on the single sound source audio data to obtain a second change matrix; calculating the speech signal covariance matrix based on the second change matrix and the transposed matrix of the second change matrix; calculating the interference signal covariance matrix based on the difference matrix between the first change matrix and the second change matrix and the transposed matrix of the difference matrix.

[0109] In one embodiment, when the processor executes the computer program, the following steps are further implemented: iterating the enhanced single sound source audio data to the encoder, so that the encoder adjusts the two-dimensional time-frequency expression data according to the enhanced single sound source audio data and the multi-channel audio sampling data.

[0110] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining the output of the decoder after the first iteration as the first output; obtaining the output of the decoder after the second iteration as the second output; calculating the loss function based on the first output and the second output, and performing parameter optimization training on the speech separation model based on the loss function.

[0111] In one embodiment, when the processor executes the computer program, the following steps are also implemented: performing Hadamard product operations on the time-frequency mask data corresponding to each speaker's voice source and the spatial feature data respectively; inputting the product operation data after the Hadamard product operation into a decoder, and using the decoder to perform a transposed convolution operation on the product operation data to obtain the single sound source audio data corresponding to each speaker's voice source respectively.

[0112] In one embodiment, when the processor executes the computer program, the following steps are further implemented: the two-dimensional time-frequency expression data is input into the following formula to extract the spatial feature data:

[0113]

[0114] in, represents the spatial feature data output by the graph convolution layer of layer l, g represents the graph convolution function, represents the diagonal node degree matrix, is the adjacency matrix, is the weight matrix of the graph convolution of the l-1th layer, Represents two-dimensional time-frequency expression data.

[0115] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining multi-channel audio sampling data, where the multi-channel audio sampling data is audio sampling data of a mixed sound source from multiple speaker sound sources collected through multiple audio sampling channels; inputting the multi-channel audio sampling data into an encoder for encoding processing to obtain two-dimensional time-frequency expression data corresponding to the multi-channel audio sampling data; inputting the two-dimensional time-frequency expression data into a graph convolutional neural network for spatial feature extraction to obtain spatial feature data corresponding to the two-dimensional time-frequency expression data; inputting the spatial feature data into a time-frequency mask network for time-frequency mask processing to obtain time-frequency mask data corresponding to each speaker sound source; inputting the time-frequency mask data and the spatial feature data corresponding to each speaker sound source into a decoder for decoding processing to obtain single sound source audio data corresponding to each speaker sound source output by the decoder.

[0116] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: inputting the single sound source audio data into the beamforming network for speech enhancement processing, and obtaining enhanced single sound source audio data output by the beamforming network.

[0117] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: determining a reference channel from multiple audio sampling channels; calculating a speech signal covariance matrix and an interference signal covariance matrix based on the single sound source audio data and the multi-channel audio sampling data; based on a minimum mean square distortion-free wave velocity formation algorithm, using the speech signal covariance matrix, the interference signal covariance matrix and the reference channel as variables, performing enhancement processing on the single sound source audio data to obtain enhanced single sound source audio data.

[0118] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: performing a short-time Fourier transform on the multi-channel audio sampling data to obtain a first change matrix; performing a short-time Fourier transform on the single sound source audio data to obtain a second change matrix; calculating the speech signal covariance matrix based on the second change matrix and the transposed matrix of the second change matrix; calculating the interference signal covariance matrix based on the difference matrix between the first change matrix and the second change matrix and the transposed matrix of the difference matrix.

[0119] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: iterating the enhanced single sound source audio data to the encoder, so that the encoder adjusts the two-dimensional time-frequency expression data according to the enhanced single sound source audio data and the multi-channel audio sampling data.

[0120] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: obtaining the output of the decoder after the first iteration as the first output; obtaining the output of the decoder after the second iteration as the second output; calculating the loss function based on the first output and the second output, and performing parameter optimization training on the speech separation model based on the loss function.

[0121] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: performing Hadamard product operations on the time-frequency mask data corresponding to each speaker's voice source and the spatial feature data respectively; inputting the product operation data after the Hadamard product operation into a decoder, and using the decoder to perform a transposed convolution operation on the product operation data to obtain the single sound source audio data corresponding to each speaker's voice source.

[0122] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: the two-dimensional time-frequency expression data is input into the following formula to extract the spatial feature data:

[0123]

[0124] in, represents the spatial feature data output by the graph convolution layer of layer l, g represents the graph convolution function, represents the diagonal node degree matrix, is the adjacency matrix, is the weight matrix of the graph convolution of the l-1th layer, Represents two-dimensional time-frequency expression data.

[0125] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0126] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A speech separation method is implemented based on a speech separation model, wherein the speech separation model includes an encoder, a graph convolutional neural network, a time-frequency mask network, and a decoder, and the method includes: Acquire multi-channel audio sampling data, where the multi-channel audio sampling data is audio sampling data of a mixed sound source from multiple speaker sound sources collected through multiple audio sampling channels; Inputting the multi-channel audio sampling data into the encoder for encoding processing to obtain two-dimensional time-frequency expression data corresponding to the multi-channel audio sampling data; Inputting the two-dimensional time-frequency expression data into the graph convolutional neural network to extract spatial features, thereby obtaining spatial feature data corresponding to the two-dimensional time-frequency expression data; Inputting the spatial feature data into the time-frequency masking network for time-frequency masking processing to obtain time-frequency masking data corresponding to the voice sources of each speaker; The time-frequency mask data and the spatial feature data corresponding to the sound sources of each speaker are input into the decoder for decoding processing, so as to obtain the single sound source audio data corresponding to the sound sources of each speaker output by the decoder.

2. The method according to claim 1, characterized in that The speech separation model also includes a beamforming network, and the method further includes: Inputting the single sound source audio data into the beamforming network for speech enhancement processing, and obtaining enhanced single sound source audio data output by the beamforming network; The method for performing speech enhancement processing on the single sound source audio data by the beamforming network includes: Determine a reference channel from the plurality of audio sampling channels; Calculating a speech signal covariance matrix and an interference signal covariance matrix according to the single sound source audio data and the multi-channel audio sampling data; Based on the minimum mean square distortion-free wave velocity forming algorithm, the speech signal covariance matrix, the interference signal covariance matrix and the reference channel are used as variables to perform enhancement processing on the single sound source audio data to obtain the enhanced single sound source audio data.

3. The method according to claim 2, characterized in that The calculating of the speech signal covariance matrix and the interference signal covariance matrix according to the single sound source audio data and the multi-channel audio sampling data comprises: Performing short-time Fourier transform on the multi-channel audio sampling data to obtain a first change matrix; Performing a short-time Fourier transform on the single sound source audio data to obtain a second change matrix; Calculate the speech signal covariance matrix according to the second change matrix and the transposed matrix of the second change matrix; The interference signal covariance matrix is ​​calculated according to a difference matrix between the first change matrix and the second change matrix and a transposed matrix of the difference matrix.

4. The method according to claim 2, characterized in that: The method further comprises: The enhanced single-source audio data is iteratively sent to the encoder, so that the encoder adjusts the two-dimensional time-frequency representation data according to the enhanced single-source audio data and the multi-channel audio sampling data.

5. The method according to claim 4, characterized in that The method further comprises: Obtaining the output of the decoder after the first iteration as a first output; Obtaining the output of the decoder after the second iteration as the second output; A loss function is calculated according to the first output and the second output, and parameter optimization training is performed on the speech separation model according to the loss function.

6. The method according to any one of claims 1 to 5, characterized in that: The step of inputting the time-frequency mask data and the spatial feature data corresponding to the sound sources of each speaker into the decoder for decoding processing to obtain the single sound source audio data corresponding to the sound sources of each speaker output by the decoder, including: Performing Hadamard product operations on the time-frequency mask data corresponding to the sound sources of each speaker and the spatial feature data respectively; The product operation data after the Hadamard product operation is input into the decoder, and the decoder is used to perform a transposed convolution operation on the product operation data to obtain the single sound source audio data corresponding to each of the speaker's sound sources.

7. The method according to any one of claims 1 to 5, characterized in that: The graph convolutional neural network includes a plurality of graph convolutional layers. The two-dimensional time-frequency expression data is input into the graph convolutional neural network to extract spatial features. The formula for obtaining spatial feature data corresponding to the two-dimensional time-frequency expression data is: in, represents the spatial feature data output by the graph convolution layer of layer l, g represents the graph convolution function, represents the diagonal node degree matrix, is the adjacency matrix, is the weight matrix of the graph convolution of the l-1th layer, Represents the input two-dimensional time-frequency expression data.

8. A speech separation model, characterized in that: The model includes: An encoder is used to obtain multi-channel audio sampling data, where the multi-channel audio sampling data is audio sampling data of a mixed sound source from multiple speakers collected through multiple audio sampling channels, and encode the multi-channel audio sampling data to obtain two-dimensional time-frequency expression data corresponding to the multi-channel audio sampling data; A graph convolutional neural network is used to perform a graph convolution operation on the two-dimensional time-frequency expression data to extract spatial features, and obtain spatial feature data corresponding to the two-dimensional time-frequency expression data; A time-frequency masking network, used for performing time-frequency masking processing on the spatial feature data to obtain time-frequency masking data corresponding to the voice sources of each speaker; The decoder is used to decode the time-frequency mask data and the spatial feature data corresponding to each speaker's voice source to obtain the single sound source audio data corresponding to each speaker's voice source.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images