Voice extraction method, system, device and storage medium

Abstract

The embodiment of the present invention discloses a speech extraction method, system, device and storage medium, including: obtaining a feature vector of a target speaker and a multi-channel feature vector of a multi-channel mixed speech; inputting the feature vector and the multi-channel feature vector into a preset model to extract a target feature vector; determining a target speech signal according to the target feature vector and the multi-channel feature vector; the training method of the preset model includes: obtaining sample training data; inputting feature vector samples and multi-channel feature vector samples into a feature extraction module to obtain a first feature vector; inputting the first feature vector and the multi-channel feature vector samples into a separation module to obtain a second feature vector; updating the feature extraction module and the separation module according to the target value calculated by the objective function of the feature vector sample and the second feature vector. The technical solution of the present invention can improve the effect of speech extraction and separation without finding an intermediate access layer, and can be widely used in the field of artificial intelligence.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a speech extraction method, system, device and storage medium. Background Art

[0002] The purpose of multi-channel speech extraction is to extract the individual speech of each speaker from a mixed speech containing noise and reverberation at the same time. Most speech extraction methods use the features of each speaker as input, or concatenate the speaker features and the features of the mixed speech as input, and input the input to a certain middle layer of the speech extraction module, but it is difficult to find the best middle layer as the access layer of the input; when the access layer of the selected input is inaccurate, the effect of extracting single-channel speech from multi-channel speech is poor. Summary of the invention

[0003] The embodiments of the present invention provide a speech extraction method, system, device and storage medium, which can improve the effect of speech extraction and separation without searching for an intermediate access layer.

[0004] To achieve the above object, an embodiment of the present invention provides a speech extraction method, comprising:

[0005] Obtaining a pre-registered target speaker's feature vector and a multi-channel feature vector of a multi-channel mixed speech;

[0006] Inputting the pre-registered feature vector of the target speaker and the multi-channel feature vector into a preset model to extract a target feature vector of the target speaker;

[0007] Determining a target speech signal of a target speaker according to the target feature vector of the target speaker and the multi-channel feature vector;

[0008] Wherein, the preset model includes a feature extraction module and a separation module, and the training method of the preset model includes:

[0009] Acquire sample training data, wherein the sample training data includes feature vector samples of single-channel speech, pre-registered feature vector samples, and multi-channel feature vector samples of multi-channel mixed speech;

[0010] Inputting the pre-registered feature vector sample and the multi-channel feature vector sample into the feature extraction module to obtain a first feature vector;

[0011] Inputting the first feature vector and the multi-channel feature vector sample into the separation module to obtain a second feature vector;

[0012] An optimization target value is calculated according to an optimization target function between the feature vector sample of the single-channel speech and the second feature vector, and the feature extraction module and the separation module are updated according to the optimization target value.

[0013] To achieve the above object, an embodiment of the present invention provides a speech extraction system, comprising:

[0014] An acquisition module, used to acquire a feature vector of a pre-registered target speaker and a multi-channel feature vector of a multi-channel mixed speech;

[0015] An extraction module, used for inputting the pre-registered feature vector of the target speaker and the multi-channel feature vector into a preset model to extract a target feature vector of the target speaker;

[0016] A conversion module, configured to determine a target speech signal of a target speaker according to the target feature vector of the target speaker and the multi-channel feature vector;

[0017] Wherein, the preset model includes a feature extraction module and a separation module, and the training method of the preset model includes:

[0018] Acquire sample training data, wherein the sample training data includes feature vector samples of single-channel speech, pre-registered feature vector samples, and multi-channel feature vector samples of multi-channel mixed speech;

[0019] Inputting the pre-registered feature vector sample and the multi-channel feature vector sample into the feature extraction module to obtain a first feature vector;

[0020] Inputting the first feature vector and the multi-channel feature vector sample into the separation module to obtain a second feature vector;

[0021] An optimization target value is calculated according to an optimization target function between the feature vector sample of the single-channel speech and the second feature vector, and the feature extraction module and the separation module are updated according to the optimization target value.

[0022] To achieve the above-mentioned purpose, an embodiment of the present invention provides a speech extraction device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the speech extraction method described in the embodiment of the first aspect are implemented.

[0023] To achieve the above-mentioned purpose, an embodiment of the present invention provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the speech extraction method described in the first aspect embodiment.

[0024] The speech extraction method, system, device and storage medium provided by the embodiment of the present invention first input the pre-registered feature vector of the target speaker and the multi-channel feature vector of the multi-channel mixed speech into the preset model to extract the target feature vector of the target speaker, and then determine the target speech signal of the target speaker according to the target feature vector and the multi-channel feature vector of the target speaker; wherein the preset model includes a feature extraction module and a separation module, and the training method of the preset model includes: inputting the pre-registered feature vector sample and the multi-channel feature vector into the speaker feature extraction module to obtain a first feature vector, then inputting the feature vector and the multi-channel feature vector sample of the multi-channel mixed speech into the separation module to obtain a second feature vector, calculating the optimization target value according to the optimization target function between the feature vector sample of the single-channel speech and the second feature vector, and updating the feature extraction module and the separation module according to the optimization target value; that is, the input of the feature extraction module is the pre-registered feature vector sample and the multi-channel feature vector sample, and the output of the feature extraction module and the multi-channel feature vector sample are the input of the separation module, so that the feature extraction module and the separation module are connected and trained together, so as to solve the parameter sharing problem of different modules, and improve the effect of speech extraction and separation without finding an intermediate access layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flowchart of the steps of a speech extraction method provided by an embodiment of the present invention;

[0026] Figure 2 is a data flow diagram of a speech extraction method provided by an embodiment of the present invention;

[0027] Figure 3 is a flowchart of the steps of obtaining a multi-channel feature vector of a multi-channel mixed speech provided by an embodiment of the present invention;

[0028] Figure 4 A data flow diagram for obtaining a multi-channel feature vector of a multi-channel mixed speech provided by an embodiment of the present invention;

[0029] Figure 5 is a flowchart of the steps of a method for training a preset model provided by an embodiment of the present invention;

[0030] Figure 6 It is a data flow diagram of a training method for a preset model provided by an embodiment of the present invention;

[0031] Figure 7 is a flowchart of steps for obtaining a first feature vector provided by an embodiment of the present invention;

[0032] Figure 8 is a data flow diagram for obtaining a first eigenvector provided by an embodiment of the present invention;

[0033] Fig. 9 is a flowchart of steps for obtaining an N-dimensional fourth eigenvector provided by an embodiment of the present invention;

[0034] Fig.10 is a data flow diagram for obtaining an N-dimensional fourth eigenvector provided by an embodiment of the present invention;

[0035] Fig.11 is a flow chart of steps for obtaining a second feature vector provided by an embodiment of the present invention;

[0036] Fig.12 is a data flow diagram for obtaining a second eigenvector provided by an embodiment of the present invention;

[0037] Fig.13 is a schematic diagram of a speech extraction system provided by an embodiment of the present invention;

[0038] Fig.14 It is a schematic diagram of a speech extraction device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0040] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0042] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no special meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0043] First, some nouns involved in this application are analyzed:

[0044] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0045] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text comprehension. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information retrieval, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0046] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0047] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0048] The speech extraction method provided by the embodiment of the present invention first trains the preset model so that the parameters of each module of the preset model reach a better state; then the multi-channel mixed speech and the speech of the pre-registered target speaker are input into the trained preset model to extract the pure single-channel speech of the target speaker from the multi-channel mixed speech.

[0049] The speech extraction system provided in the embodiment of the present invention includes a terminal and a server, wherein the server is deployed with an AI model, and the terminal is used to obtain multi-channel mixed speech, single-channel mixed speech, and speech registered by a target speaker, etc. The terminal sends the obtained speech to the server; the server first trains the AI ​​model according to the received speech, and the trained AI model is then used to extract the mixed speech to obtain the target speech, and finally the server sends the target speech to the terminal for display. Among them, the terminal can be a sound pickup hole or a pickup, or a device with a sound pickup hole or a pickup, such as a mobile phone, an electronic watch or an electronic bracelet, etc., and the embodiment of the present invention does not make specific restrictions.

[0050] See also Figure 1 The embodiment of the present invention provides a speech extraction method, including step S100, step S200 and step S300. Figure 2 , Figure 2 is the data flow diagram of the speech extraction method. Figure 2 In the method, the pre-registered feature vector of the target speaker and the multi-channel feature vector of the multi-channel mixed speech are first input into the preset model to obtain the target feature vector of the target speaker; then the target feature vector of the target speaker and the multi-channel feature vector of the multi-channel mixed speech are input into the decoding module to obtain the target speech signal of the target speaker. The specific steps of the speech extraction method are described below.

[0051] S100, obtaining a feature vector of a pre-registered target speaker and a multi-channel feature vector of a multi-channel mixed speech.

[0052] Specifically, firstly, a speech segment registered by a target speaker is obtained, and then i-vector extraction is performed on the speech segment registered by the target speaker to obtain a feature vector of the pre-registered target speaker.

[0053] Specifically, the i-vector space uses a low-dimensional total variable space to represent the speaker subspace and channel subspace. By projecting the speaker's speech onto the space through dimensionality reduction, a vector representation of a fixed length, i.e., an i-vector, can be obtained. The i-vector extraction generally includes: preprocessing the speaker's training speech segment, such as pre-emphasis, framing, windowing, and endpoint detection; extracting the corresponding training speech features from the preprocessed training speech segment, and training the overall change subspace corresponding to the preset UBM model based on the preset UBM model; projecting the training speech features onto the overall change subspace to obtain the i-vector.

[0054] Optionally, see Figure 3 The multi-channel feature vector of the multi-channel mixed speech is obtained by the following method, including step S110, step S120, step S130, step S140 and step S150. Figure 4 , Figure 4 is the data flow diagram for obtaining the multi-channel feature vector. Figure 4 In the method, a single-channel mixed speech is first divided into multiple single-channel speech segments, and the multiple single-channel speech segments are sequentially input into a spectrum encoder; then a multi-channel mixed speech is divided into multiple multi-channel mixed speech segments, and the multiple multi-channel mixed speech segments are sequentially input into a space encoder; then the output of the spectrum encoder and the output of the space encoder are concatenated to form a multi-channel feature vector.

[0055] Specifically, the specific steps of obtaining the multi-channel feature vector of the multi-channel mixed speech include:

[0056] S110, acquiring a multi-channel mixed voice and dividing the multi-channel mixed voice into a plurality of mixed voice segments of preset lengths;

[0057] S120, sequentially inputting the mixed speech segments into the spatial encoder to obtain a plurality of mixed feature vectors;

[0058] S130, obtaining a single-channel mixed voice and dividing the single-channel mixed voice into a plurality of single-channel voice segments of preset lengths;

[0059] S140, sequentially inputting the single-channel speech segments into a spectrum encoder to obtain a plurality of single-channel feature vectors;

[0060] S150, concatenating a plurality of mixed feature vectors and a plurality of single-channel feature vectors to generate a multi-channel feature vector.

[0061] It should be noted that the preset lengths of the mixed speech segment and the single-channel speech segment are the same; however, if the input signal dimension of the spectrum encoder is L, the input signal dimension of the spatial encoder is 2L, where L is a positive number.

[0062] Specifically, the spatial encoder consists of a 2-dimensional convolutional layer, and a mixed speech segment is input into the spatial encoder to obtain an S-dimensional mixed feature vector; the spectrum encoder consists of a 1-dimensional convolutional layer, and a single-channel speech segment is input into the spectrum encoder to obtain an N1-dimensional single-channel feature vector; the dimension of the multi-channel feature vector formed by concatenating the S-dimensional mixed feature vector and the N1-dimensional single-channel feature vector is N1+S, where N1 and S are both positive integers. By concatenating the mixed feature vector and the single-channel feature vector, the spatial features and spectral features of the multi-channel mixed speech are combined.

[0063] Multi-channel mixed speech and single-channel mixed speech are stored in digital form, and the speech length can be understood as the duration of the speech. For example, a multi-channel mixed speech with a duration of 60 seconds is divided into 20 mixed speech segments with a duration of 3 seconds, and the 20 mixed speech segments with a duration of 3 seconds are input into a 2-dimensional spatial encoder in turn to obtain an S-dimensional mixed feature vector; a single-channel mixed speech with a duration of 30 seconds is divided into 10 single-channel speech segments with a duration of 3 seconds, and the 10 single-channel speech segments with a duration of 3 seconds are input into a 1-dimensional spectrum encoder in turn to obtain an N1-dimensional single-channel feature vector; the S-dimensional mixed feature vector and the N1-dimensional single-channel feature vector are concatenated to form an N1+S-dimensional multi-channel feature vector.

[0064] S200: Input the pre-registered feature vector and multi-channel feature vector of the target speaker into a preset model to extract the target feature vector of the target speaker.

[0065] Those skilled in the art can understand that the preset model is trained with sample training data, and each parameter in the preset model has reached an optimal state.

[0066] Among them, see Figure 5 The preset model includes a feature extraction module and a separation module, and the training method of the preset model includes steps S210, S220 and S230. Figure 6 , Figure 6 It is the data flow diagram of the training method of the preset model. Figure 6 In the method, firstly, the pre-registered feature vector samples and the multi-channel feature vector samples are input into the feature extraction module to obtain the first feature vector; then, the first feature vector and the multi-channel feature vector samples are input into the separation module to obtain the second feature vector; then, the optimization target value is determined according to the optimization target function between the second feature vector and the feature vector samples of the single-channel speech, and the feature extraction module and the separation module are updated according to the optimization target value.

[0067] Specifically, the specific steps of the training method of the preset model include:

[0068] S210, obtaining sample training data, where the sample training data includes feature vector samples of single-channel speech, pre-registered feature vector samples, and multi-channel feature vector samples of multi-channel mixed speech.

[0069] Specifically, the feature vector sample of single-channel speech is obtained by performing i-vector extraction on the single-channel speech sample, the pre-registered feature vector sample is obtained by performing i-vector extraction on the pre-registered speech sample, and the multi-channel feature vector sample of multi-channel mixed speech is obtained by applying the method of steps S110 to S150 to the multi-channel mixed speech sample.

[0070] S220 , inputting the pre-registered feature vector samples and the multi-channel feature vector samples into a feature extraction module to obtain a first feature vector.

[0071] Those skilled in the art can understand that the function of the feature extraction module is to enable the pre-registered feature vector samples to learn the similar parts of the original multi-channel feature vector samples.

[0072] Optionally, see Figure 7 The feature extraction module includes a first neural network, a second neural network and a third neural network. The pre-registered feature vector samples and multi-channel feature vector samples are input into the feature extraction module to obtain a first feature vector, including steps S221, S222 and S223. Figure 8 , Figure 8 is the data flow diagram for obtaining the first eigenvector. Figure 8 In the method, the pre-registered feature vector samples are first input into the first neural network, and then the multi-channel feature vector samples are input into the second neural network; then the output of the first neural network and the output of the second neural network are convolved and input into the third neural network to obtain the first feature vector.

[0073] Specifically, the specific steps of obtaining the first eigenvector include:

[0074] S221. Process the pre-registered feature vector samples through the first neural network to obtain an N-dimensional third feature vector, where N is a positive integer greater than or equal to 2.

[0075] Specifically, the first neural network can select a 1-dimensional convolutional layer, and the number of output channels of the 1-dimensional convolutional layer is N, where N is a positive integer greater than or equal to 2. Experiments have shown that when N is 2, the requirement can be met, and N can also be a positive integer greater than 2.

[0076] S222. Process the multi-channel feature vector samples through a second neural network to obtain an N-dimensional fourth feature vector.

[0077] It should be noted that the role of the second neural network is to extract N deeper feature sequences from the multi-channel feature vector samples.

[0078] Optionally, see Fig. 9 The second neural network includes a first instance regularization layer, a first convolutional layer, a TCN (Temporal Convolutional Network) layer and a second convolutional layer. The multi-channel feature vector samples are processed by the second neural network to obtain an N-dimensional fourth feature vector, including steps S2221, S2222, S2223 and S2224. Fig.10 , Fig.10 It is the data flow diagram for obtaining the fourth eigenvector of N dimensions. Fig.10 In the example, the multi-channel feature vector sample is first input into the first instance regularization layer, and the output vector of the first instance regularization layer is input into the first convolutional layer, and then the output vector of the first convolutional layer is input into the TCN layer, and then the output vector of the TCN layer is input into the second convolutional layer to obtain the N-dimensional fourth feature vector.

[0079] Specifically, the specific steps of obtaining the N-dimensional fourth eigenvector include:

[0080] S2221, inputting the multi-channel feature vector sample into the first instance regularization layer to obtain a first processing vector;

[0081] S2222, input the first processing vector to the first convolutional layer to obtain a second processing vector;

[0082] S2223, input the second processing vector to the TCN layer to obtain a third processing vector;

[0083] S2224. Input the third processed vector into the second convolutional layer to obtain an N-dimensional fourth eigenvector.

[0084] Specifically, during the training process of the neural network, overfitting may occur; for overfitting, L2 regularization or Dropout regularization can be used. For example, if L2 regularization is used in the neural network, the Frobenius norm term is added to the cost function, and the complex neural network becomes simpler by adjusting the parameters of the Frobenius norm term, thereby reducing overfitting. If Dropout regularization is used in the neural network, Dropout regularization can be understood as randomly deleting the nodes of each layer in the neural network during each training process; and the probability of retaining a hidden unit may be different for different layers of the neural network. For example, a neural network includes 7 layers, and the third and fourth layers have more neuron nodes and more parameters, so the third and fourth layers have a more significant impact on overfitting, so the probability of retaining a hidden unit in the third and fourth layers can be set higher; while the fifth and sixth layers have fewer neuron nodes, so the fifth and sixth layers have less impact on overfitting, so the probability of retaining a hidden unit in the fifth and sixth layers can be set lower.

[0085] TCN includes causal convolution, dilated convolution and residual link. In causal convolution, the value of the previous layer at time t only depends on the value of the next layer at time t and before; the difference from the traditional convolutional neural network is that causal convolution cannot see future data. It is a unidirectional structure, not a bidirectional one, that is, only the previous cause can have the next result. It is a strict time constraint model, so it is called causal convolution. Dilated convolution allows the input to be sampled at intervals during convolution, and the sampling rate is controlled by d. For example, d=1 in the bottom layer means that each point is sampled during input; d=2 in the middle layer means that one point is sampled every 2 points during input as input; generally speaking, the higher the level, the larger the size of d used; therefore, dilated convolution makes the size of the effective window grow exponentially with the number of layers; in this way, the convolutional network can obtain a large receptive field with relatively few layers. Residual link has been proven to be an effective method for training deep networks, which allows the network to transmit information in a cross-layer manner, such as building a residual block to replace a layer of convolution.

[0086] Those skilled in the art can understand that the role of the first instance regularization layer is to make the convolutional neural network easier to converge and reduce the sensitivity to network hyperparameters; the role of the first convolutional layer, TCN layer and the second convolutional layer is to extract deeper features. Among them, the first convolutional layer, TCN layer and the second convolutional layer are 1-dimensional convolutional layers.

[0087] The second neural network includes a TCN layer, which can also be replaced by an RNN (Recurrent Neural Network) layer. However, the TCN layer requires a small number of parameters and has a fast training speed. The RNN includes an input layer, a hidden layer, and an output layer. A first weight matrix is ​​provided from the input layer to the hidden layer, and a second weight matrix is ​​provided from the hidden layer to the output layer. The hidden layer contains multiple layers. The input vector is input to the first layer of the hidden layer after reacting with the first weight matrix. The output of the first layer of the hidden layer is input to the second layer of the hidden layer together with the input vector. By analogy, the output of the last layer of the hidden layer is reacted with the second weight matrix to obtain an output vector. The number of hidden layers is set according to actual needs.

[0088] S223, input the N-dimensional third eigenvector and the N-dimensional fourth eigenvector into the third neural network for processing to obtain a first eigenvector.

[0089] It should be noted that the vectors at corresponding positions of the N-dimensional third eigenvector and the N-dimensional fourth eigenvector are multiplied and then input into the third neural network.

[0090] Specifically, the third neural network can select a 1D convolutional layer, use ReLU (Rectified Linear Unit) as a nonlinear activation function, and finally generate an E-dimensional first eigenvector; the role of the nonlinear activation function here is to solve the disadvantage of sigmoid gradient disappearance when training deep network models, and it also has the advantage of reducing overfitting.

[0091] It should be noted that the function of the third neural network is to perform weight modulation and update on the third eigenvector and the fourth eigenvector. For example, N is 2, the first dimension of the 2-dimensional third eigenvector is [1, 2, 3], the first dimension of the 2-dimensional fourth eigenvector is [4, 5, 6], and the weight of the first dimension is [0.1, 0.3, 0.2]. Then the result of weight modulation of these two vectors is [1*0.1+4*0.9, 2*0.3+5*0.7, 3*0.2+6*0.8].

[0092] S230 , input the first feature vector and the multi-channel feature vector samples into a separation module to obtain a second feature vector.

[0093] Optionally, see Fig.11 The separation module includes a fourth neural network, the fourth neural network includes a second instance regularization layer, a third convolution layer, a TCN unit and a fourth convolution layer, the TCN unit includes a plurality of TCN layers, the first feature vector and the multi-channel feature vector sample are input into the separation module to obtain a second feature vector, including steps S231, S232, S233 and S234. Fig.12 , Fig.12 is the data flow diagram for obtaining the second eigenvector. Fig.12 In the embodiment, the first feature vector and the multi-channel feature vector samples are first input into the second instance regularization layer, then the output vector of the second instance regularization layer is input into the third convolutional layer, then the output vector of the third convolutional layer is input into the TCN unit, and then the output vector of the TCN unit is input into the fourth convolutional layer to obtain the second feature vector.

[0094] Specifically, the specific steps of obtaining the second eigenvector include:

[0095] S231, inputting the first feature vector and the multi-channel feature vector sample into the second instance regularization layer to obtain a fourth processing vector;

[0096] S232, input the fourth processing vector into the third convolutional layer to obtain a fifth processing vector;

[0097] S233, inputting the fifth processing vector into the TCN unit to obtain a sixth processing vector;

[0098] S234. Input the sixth processed vector into the fourth convolutional layer to obtain a second feature vector.

[0099] The separation module acts as a linear weight on the sample features of the multi-channel feature vector to produce individual features for each speaker.

[0100] Those skilled in the art can understand that the role of the second instance regularization layer is to make the convolutional neural network easier to converge and reduce the sensitivity to network hyperparameters; the role of the third convolutional layer, TCN layer unit and the fourth convolutional layer is to extract deeper features. Among them, the third convolutional layer, TCN unit and the fourth convolutional layer are 1-dimensional convolutional layers.

[0101] The number of TCN layers in the TCN unit is set according to actual needs and is not specifically limited in this embodiment.

[0102] The fourth neural network includes a TCN layer, which requires a small number of parameters and has a fast training speed.

[0103] Specifically, the PReLU (Parametric Rectified Linear Unit) activation function is used before the fourth convolutional layer, and the ReLU (Rectified Linear Unit) activation function is used after the fourth convolutional layer; both PReLU and ReLU can solve the disadvantage of sigmoid gradient disappearance when training deep network models, and have the advantage of reducing overfitting.

[0104] PReLU and ReLU are both non-saturated activation functions. The ReLU function stands for "rectified linear unit". The ReLU function sets all negative values ​​in the matrix x to zero, and the rest of the values ​​remain unchanged; the ReLU function is calculated after the convolution layer. The PReLU function does not fix the slope of the negative part of the matrix x, and needs to learn the slope in the back propagation, and the rest of the values ​​remain unchanged.

[0105] S240, calculating an optimization target value according to an optimization target function between the feature vector sample of the single-channel speech and the second feature vector, and updating the feature extraction module and the separation module according to the optimization target value.

[0106] Specifically, the number of training times of the preset model can be determined according to the size of the preset optimization target value or the preset number of training times, or the two methods can be combined; for example, when the number of training times reaches 10,000 times, the training of the preset model is completed, and the feature extraction module and classification module corresponding to the best results in the 10,000 training times are used as the preset model, and it is considered that the parameters in the preset model have reached a better state; for example, when the optimization target value of the preset model reaches the preset target value, the feature extraction module and classification module corresponding to the optimization target value are used as the preset model; for example, when the optimization target value of the preset model reaches the preset target value or the number of training times reaches 10,000 times, the feature extraction module and classification module corresponding to the best result are used as the preset model.

[0107] Optionally, the optimization objective function is as follows:

[0108]

[0109]

[0110]

[0111] Among them, SI_SNR represents the optimization objective function, represents the second feature vector, s represents the feature vector of the single channel of the target speaker, ||s|| 2 =<s,s> Represents the signal energy.

[0112] It should be noted that the larger the value of the above optimization objective function is, the better the training result of the preset model is.

[0113] Optionally, updating the feature extraction module and the separation module according to the optimization target value specifically includes:

[0114] The feature extraction module and the separation module are updated according to the maximum value of the optimization target value.

[0115] For example, the preset number of training times is 10,000 times; in the 5,000th training, the optimization target value obtained according to the optimization objective function is 40; in the training before the 5,000th training, the maximum optimization target value obtained according to the optimization objective function is 20. Since 40 is greater than 20, the parameters corresponding to the optimization target value of 40 are updated for the extraction module and the separation module; if in the remaining 5,000 training times, the optimization target values ​​obtained according to the optimization objective function are all less than 40, finally the extraction module and separation module corresponding to the optimization target value of 40 are determined as the preset models.

[0116] For another example, the preset optimization target value is 30; in a certain training, the optimization target value obtained according to the optimization objective function is 35. Since 35 is greater than 30, the parameters corresponding to the optimization target value of 35 are updated for the extraction module and the separation module, and the extraction module and the separation module corresponding to the optimization target value of 35 are determined as the preset model.

[0117] For another example, the preset number of training times is 10,000 times or the preset optimization target value is 30; in the 7,000th training, the optimization target value obtained according to the optimization objective function is 33. Since 33 is greater than 30, the parameters corresponding to the optimization target value of 33 are updated for the extraction module and the separation module, and the extraction module and the separation module corresponding to the optimization target value of 33 are determined as the preset models; or in the 10,000 times of training, the maximum optimization target value obtained according to the optimization objective function is 28, the parameters corresponding to the optimization target value of 28 are updated for the extraction module and the separation module, and the extraction module and the separation module corresponding to the optimization target value of 28 are determined as the preset models.

[0118] S300: Determine a target speech signal of the target speaker according to the target feature vector and the multi-channel feature vector of the target speaker.

[0119] Specifically, the target feature vector and multi-channel feature vector of the target speaker are input into the decoding module to obtain the target speech signal of the target speaker. The function of the decoding module is to convert the target feature vector and multi-channel feature vector of the target speaker into a time domain signal, thereby completing the extraction of the speech signal.

[0120] In an optional embodiment, the target voice signal may be further converted into text, for example, the voice signal of each participant is extracted from the multi-person conference audio, and the voice signal of each participant is converted into text as the conference record.

[0121] The speech extraction method, system, device and storage medium provided by the embodiment of the present invention first input the pre-registered feature vector of the target speaker and the multi-channel feature vector of the multi-channel mixed speech into the preset model to extract the target feature vector of the target speaker, and then determine the target speech signal of the target speaker according to the target feature vector and the multi-channel feature vector of the target speaker; wherein the preset model includes a feature extraction module and a separation module, and the training method of the preset model includes: inputting the pre-registered feature vector sample and the multi-channel feature vector into the speaker feature extraction module to obtain a first feature vector, then inputting the feature vector and the multi-channel feature vector sample of the multi-channel mixed speech into the separation module to obtain a second feature vector, calculating the optimization target value according to the optimization target function between the feature vector sample of the single-channel speech and the second feature vector, and updating the feature extraction module and the separation module according to the optimization target value; that is, the input of the feature extraction module is the pre-registered feature vector sample and the multi-channel feature vector sample, and the output of the feature extraction module and the multi-channel feature vector sample are the input of the separation module, so that the feature extraction module and the separation module are connected and trained together, so as to solve the parameter sharing problem of different modules, and improve the effect of speech extraction and separation without finding an intermediate access layer.

[0122] like Fig.13 As shown, an embodiment of the present invention provides a speech extraction system, including:

[0123] An acquisition module, used to acquire a feature vector of a pre-registered target speaker and a multi-channel feature vector of a multi-channel mixed speech;

[0124] An extraction module, used for inputting the pre-registered feature vector and multi-channel feature vector of the target speaker into a preset model to extract the target feature vector of the target speaker;

[0125] A conversion module, used to determine a target speech signal of a target speaker according to a target feature vector and a multi-channel feature vector of the target speaker;

[0126] The preset model includes a feature extraction module and a separation module, and the training method of the preset model includes:

[0127] Acquire sample training data, the sample training data including feature vector samples of single-channel speech, pre-registered feature vector samples, and multi-channel feature vector samples of multi-channel mixed speech;

[0128] Inputting the pre-registered feature vector samples and the multi-channel feature vector samples into the feature extraction module to obtain a first feature vector;

[0129] Inputting the first feature vector and the multi-channel feature vector samples into the separation module to obtain the second feature vector;

[0130] An optimization target value is calculated according to an optimization target function between the feature vector sample of the single-channel speech and the second feature vector, and a feature extraction module and a separation module are updated according to the optimization target value.

[0131] The specific functional implementation of the acquisition module, extraction module and conversion module can be found in the above Figure 1 Steps S100 to S300 in the corresponding embodiment are not described in detail here.

[0132] In an optional embodiment, the acquisition module specifically includes:

[0133] A first acquisition unit, configured to acquire a multi-channel mixed speech and divide the multi-channel mixed speech into a plurality of mixed speech segments of preset lengths;

[0134] A first vector generating unit, used for sequentially inputting the mixed speech segments into a spatial encoder to obtain a plurality of mixed feature vectors;

[0135] A second acquisition unit, configured to acquire a single-channel mixed speech and divide the single-channel mixed speech into a plurality of single-channel speech segments of preset lengths;

[0136] A second vector generating unit, used for sequentially inputting the single-channel speech segments into a spectrum encoder to obtain a plurality of single-channel feature vectors;

[0137] The third vector generating unit is used to concatenate the plurality of mixed feature vectors and the plurality of single-channel feature vectors to generate the multi-channel feature vector.

[0138] In an optional embodiment, the feature extraction module specifically includes:

[0139] A third feature vector generating unit, configured to process the pre-registered feature vector sample through the first neural network to obtain an N-dimensional third feature vector;

[0140] a fourth feature vector generating unit, configured to process the multi-channel feature vector samples through the second neural network to obtain an N-dimensional fourth feature vector;

[0141] The first feature vector generating unit is used to input the N-dimensional third feature vector and the N-dimensional fourth feature vector into the third neural network for processing to obtain a first feature vector; wherein N is a positive integer greater than or equal to 2.

[0142] In an optional embodiment, the fourth feature vector generating unit specifically includes:

[0143] A first processing vector generating subunit, configured to input the multi-channel feature vector sample into the first instance regularization layer to obtain a first processing vector;

[0144] a second processing vector generating subunit, configured to input the first processing vector into the first convolutional layer to obtain a second processing vector;

[0145] A third processing vector generating subunit, configured to input the second processing vector into the TCN layer to obtain the third processing vector;

[0146] The fourth eigenvector generating subunit is used to input the third processing vector into the second convolutional layer to obtain the N-dimensional fourth eigenvector.

[0147] In an optional embodiment, the separation module specifically includes:

[0148] a fourth processing vector generating unit, configured to input the first feature vector and the multi-channel feature vector sample into the second instance regularization layer to obtain a fourth processing vector;

[0149] a fifth processing vector generating unit, configured to input the fourth processing vector into the third convolutional layer to obtain a fifth processing vector;

[0150] a sixth processing vector generating unit, configured to input the fifth processing vector into the TCN unit to obtain a sixth processing vector;

[0151] The second feature vector generating unit is used to input the sixth processing vector into the fourth convolution layer to obtain a second feature vector.

[0152] See also Fig.14 , Fig.14 It is a schematic diagram of another speech extraction device provided by an embodiment of the invention. An embodiment of the invention further provides a data processing device 1400, which includes: a memory 1410, a processor 1420, and a computer program stored in the memory 1410 and executable on the processor 1420.

[0153] The processor 1420 and the memory 1410 may be connected via a bus or other means.

[0154] The non-transitory software program and instructions required to implement the data processing method of the above embodiment are stored in the memory 1410. When executed by the processor 1420, the data processing method of the data processing device in the above embodiment is executed, for example, the above described Figure 1 Method steps S100 to S300, Figure 3 Method steps S110 to S150, Figure 5 Method steps S210 to S240, Figure 7 Method steps S221 to S231, Fig. 9 Method steps S2221 to S2224, Fig.11 Method steps S231 to S234 in.

[0155] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by a processor 1420 or a controller, for example, by a processor 1420 in the above-mentioned data processing device 1400 embodiment, so that the above-mentioned processor 1420 can execute the data processing method applied to the data processing device 1400 in the above-mentioned embodiment, for example, execute the above-mentioned Figure 1 Method steps S100 to S300, Figure 3 Method steps S110 to S150, Figure 5 Method steps S210 to S240, Figure 7 Method steps S221 to S231, Fig. 9 Method steps S2221 to S2224, Fig.11Method step S231 to method step S234 in. It will be appreciated by those skilled in the art that all or some of the steps and systems in the method disclosed above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0157] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A speech extraction method, characterized in that: include: Obtaining a pre-registered target speaker's feature vector and a multi-channel feature vector of a multi-channel mixed speech; Inputting the pre-registered target speaker's feature vector and the multi-channel feature vector into a preset model to extract the target feature vector of the target speaker; Determining a target speech signal of a target speaker according to the target feature vector of the target speaker and the multi-channel feature vector; Wherein, the preset model includes a feature extraction module and a separation module, and the training method of the preset model includes: Acquire sample training data, wherein the sample training data includes feature vector samples of single-channel speech, pre-registered feature vector samples, and multi-channel feature vector samples of multi-channel mixed speech; Inputting the pre-registered feature vector sample and the multi-channel feature vector sample into the feature extraction module to obtain a first feature vector; Inputting the first feature vector and the multi-channel feature vector sample into the separation module to obtain a second feature vector; Calculate an optimization target value according to an optimization target function between the feature vector sample of the single-channel speech and the second feature vector, and update the feature extraction module and the separation module according to the optimization target value; The separation module includes a fourth neural network, the fourth neural network includes a second instance regularization layer, a third convolutional layer, a TCN unit and a fourth convolutional layer, the TCN unit includes a plurality of TCN layers, and the first feature vector and the multi-channel feature vector sample are input into the separation module to obtain a second feature vector, including: Inputting the first feature vector and the multi-channel feature vector sample into the second instance regularization layer to obtain a fourth processing vector; Inputting the fourth processing vector into the third convolutional layer to obtain a fifth processing vector; Inputting the fifth processing vector into the TCN unit to obtain a sixth processing vector; The sixth processing vector is input into the fourth convolutional layer to obtain a second feature vector.

2. The speech extraction method according to claim 1, characterized in that: The multi-channel feature vector of the multi-channel mixed speech is obtained by the following method: Acquire multi-channel mixed speech and divide the multi-channel mixed speech into a plurality of mixed speech segments of preset lengths; Inputting the mixed speech segments into a spatial encoder in sequence to obtain a plurality of mixed feature vectors; Acquire single-channel mixed speech and divide the single-channel mixed speech into a plurality of single-channel speech segments of preset lengths; Inputting the single-channel speech segments into a spectrum encoder in sequence to obtain a plurality of single-channel feature vectors; The plurality of mixed feature vectors and the plurality of single-channel feature vectors are concatenated to generate the multi-channel feature vector.

3. The speech extraction method according to claim 1, characterized in that: The feature extraction module includes a first neural network, a second neural network and a third neural network, and the pre-registered feature vector sample and the multi-channel feature vector sample are input into the feature extraction module to obtain a first feature vector, including: Processing the pre-registered feature vector sample by the first neural network to obtain an N-dimensional third feature vector; Processing the multi-channel feature vector samples by the second neural network to obtain an N-dimensional fourth feature vector; Inputting the N-dimensional third eigenvector and the N-dimensional fourth eigenvector into the third neural network for processing to obtain a first eigenvector; Wherein, N is a positive integer greater than or equal to 2.

4. The speech extraction method according to claim 3, characterized in that: The second neural network includes a first instance regularization layer, a first convolutional layer, a TCN layer and a second convolutional layer, and the multi-channel feature vector samples are processed by the second neural network to obtain an N-dimensional fourth feature vector, including: Inputting the multi-channel feature vector sample into the first instance regularization layer to obtain a first processing vector; Inputting the first processing vector into the first convolutional layer to obtain a second processing vector; Inputting the second processing vector into the TCN layer to obtain a third processing vector; The third processing vector is input into the second convolutional layer to obtain the N-dimensional fourth feature vector.

5. The speech extraction method according to claim 1, characterized in that: The optimization objective function is as follows: Among them, SI_SNR represents the optimization objective function, represents the second feature vector, s represents the feature vector of the single channel of the target speaker, ||s|| 2 =<s,s> Represents the signal energy.

6. The speech extraction method according to claim 5, characterized in that: The updating of the feature extraction module and the separation module according to the optimization target value specifically includes: The feature extraction module and the separation module are updated according to the maximum value of the optimization target value.

7. A speech extraction system, characterized in that: include: An acquisition module, used to acquire a feature vector of a pre-registered target speaker and a multi-channel feature vector of a multi-channel mixed speech; An extraction module, used for inputting the pre-registered feature vector of the target speaker and the multi-channel feature vector into a preset model to extract a target feature vector of the target speaker; A conversion module, configured to determine a target speech signal of a target speaker according to the target feature vector of the target speaker and the multi-channel feature vector; Wherein, the preset model includes a feature extraction module and a separation module, and the training method of the preset model includes: Acquire sample training data, wherein the sample training data includes feature vector samples of single-channel speech, pre-registered feature vector samples, and multi-channel feature vector samples of multi-channel mixed speech; Inputting the pre-registered feature vector sample and the multi-channel feature vector sample into the feature extraction module to obtain a first feature vector; Inputting the first feature vector and the multi-channel feature vector sample into the separation module to obtain a second feature vector; Calculate an optimization target value according to an optimization target function between the feature vector sample of the single-channel speech and the second feature vector, and update the feature extraction module and the separation module according to the optimization target value; The separation module includes a fourth neural network, the fourth neural network includes a second instance regularization layer, a third convolutional layer, a TCN unit and a fourth convolutional layer, the TCN unit includes a plurality of TCN layers, and the first feature vector and the multi-channel feature vector sample are input into the separation module to obtain a second feature vector, including: Inputting the first feature vector and the multi-channel feature vector sample into the second instance regularization layer to obtain a fourth processing vector; Inputting the fourth processing vector into the third convolutional layer to obtain a fifth processing vector; Inputting the fifth processing vector into the TCN unit to obtain a sixth processing vector; The sixth processing vector is input into the fourth convolutional layer to obtain a second feature vector.

8. A speech extraction device, characterized in that: The device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the speech extraction method as described in any one of claims 1 to 6 are realized.

9. A storage medium for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the speech extraction method as described in any one of claims 1 to 6.

Images