A voice enhancement method, system and intelligent device
By training non-specific and speaker-specific speech enhancement models through adversarial learning and combining them with speaker-specific speech data, the problem of poor performance of deep learning speech enhancement technology for specific users is solved, achieving fast and efficient speech enhancement and improving speaker recognition accuracy.
Patent Information
- Application Number
- CN202210226944.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-08
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-03-08
AI Technical Summary
Existing deep learning speech enhancement technologies have poor speech enhancement effects for specific users, consume a lot of processing resources and time, and fail to effectively reduce external non-steady-state noise interference.
Adversarial learning is used to train speaker-independent and speaker-specific speech enhancement models. By combining clean and noisy speech data of specific speakers, a deep neural network is used to identify speaker categories and perform targeted enhancement.
It improves the enhancement effect of specific person's speech, reduces external non-stationary noise interference, improves the accuracy of speaker recognition in real noisy scenarios, and reduces the amount of data training, thus achieving fast and effective speech enhancement.
Smart Images

Figure CN114743557B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a speech enhancement method, system and intelligent device. Background Art
[0002] With the continuous development of e-commerce models, live streaming has become the most popular online marketing method. At present, live streaming scenarios are no longer limited to live streaming rooms, but more often outdoor live streaming in outdoor scenarios. In outdoor scenarios, live streaming devices are usually portable smart devices such as mobile phones and IPADs. During the live streaming process, they are inevitably disturbed by external non-steady-state noise. Considering that the live streaming device is usually used by the anchor himself, how to enhance the anchor's voice information to reduce the interference of external non-steady-state noise is an urgent problem to be solved.
[0003] With the rapid development of machine learning and neural network technology, supervised neural network speech enhancement algorithms based on deep learning have gradually become a research hotspot. Deep Neural Networks (DNN), CNN, RNN and other speech enhancement models have gradually replaced traditional speech enhancement algorithms. The speech enhancement technology based on deep neural networks mainly separates speech from noisy signals through the extraction of acoustic features and deep neural network training to achieve speech enhancement. However, the processing resources required for feature extraction and neural network training are large, the processing time is long, and the speech information of specific users is not taken into account, so the speech enhancement effect for one or more specific users is poor.
[0004] Currently, there is no technology for fast and efficient speech enhancement for one or more specific users. Therefore, a speech enhancement method, system and intelligent device are proposed. Summary of the invention
[0005] In order to solve the above problems, the present invention proposes a speech enhancement method, system and intelligent device. A speech enhancement method of the present invention comprises:
[0006] Using the first training data to train a non-specific person speech enhancement model;
[0007] Based on the non-specific person speech enhancement model, the specific person training data is used to train the specific person speech enhancement model;
[0008] Extract voice feature information and identify person categories through deep neural networks;
[0009] The voice data is input into the corresponding model according to the person category to obtain enhanced voice information.
[0010] Preferably, the adopting of the first training data to train the non-specific person speech enhancement model comprises the steps of:
[0011] The speech signal synthesized by mixing the standard speech data and the noise data according to the preset signal-to-noise ratio is used as the simulated noisy speech;
[0012] A standard speech database is used as clean speech, and the simulated noisy speech and its corresponding clean speech together constitute the first training data;
[0013] The simulated noisy speech is used as the input data of the adversarial network generator G, and the clean speech is used as the target output of the generator G, which is input into the generative adversarial network;
[0014] The generator G trained by adversarial learning is the non-specific person speech enhancement model.
[0015] Further preferably, after the first training data is used to train the non-specific person speech enhancement model, the method further includes the step of training a new non-specific person speech enhancement model based on the non-specific person speech enhancement model using the second training data:
[0016] The simulated noisy speech synthesized in the first training data is processed by a statistical speech enhancement algorithm to generate a first enhanced speech, and the second training data is the simulated noisy speech and its corresponding first enhanced speech; the statistical speech enhancement algorithm includes any one or a combination of the IMCRA-OMLSA algorithm, the IMCRA minimum control iterative averaging algorithm, and the optimal modified logarithmic amplitude estimation algorithm;
[0017] The simulated noisy speech is used as input data of the adversarial network generator G, and the first enhanced speech is used as the target output of the generator G, which is input into the adversarial network of the current non-specific person speech enhancement model;
[0018] The generator G trained in an adversarial learning manner is a new non-specific person speech enhancement model. Preferably, the specific person training data includes the third training data and / or the fourth training data, the third training data includes the noisy speech of the specific person and its corresponding clean speech of the specific person, the noisy speech of the specific person is the speech synthesized by the clean speech of the specific person and the standard noise database according to the preset signal-to-noise ratio, the clean speech of the specific person is the speech of the specific person collected under the set quiet environment noise; the fourth training data includes the real noisy speech of the specific person and the corresponding fourth enhanced speech, the real noisy speech of the specific person is the noisy speech of the specific person collected under the actual scene noise environment, and the fourth enhanced speech is the speech information obtained after the real noisy speech of the specific person is processed by the statistical speech enhancement algorithm.
[0019] Preferably, the step of training the specific person speech enhancement model using specific person training data based on the non-specific person speech enhancement model comprises the following steps:
[0020] The noisy speech of a specific person in the third training data is used as the input data of the adversarial network generator G, and the clean speech of the specific person is used as the target output of the generator G.
[0021] Or using the real noisy speech of a specific person in the fourth training data as input data of the adversarial network generator G, and using the fourth enhanced speech as the target output of the generator G;
[0022] Input the data into the adversarial network corresponding to the current non-specific person speech enhancement model;
[0023] The generator G trained by adversarial learning is the speech enhancement model for a specific person.
[0024] Preferably, the step of training the specific person speech enhancement model using specific person training data based on the non-specific person speech enhancement model comprises the following steps:
[0025] The noisy speech of the specific person in the third training data is used as the input data of the adversarial network generator G, and the clean speech of the specific person is used as the target output of the generator G, which is input into the adversarial network of the current non-specific person speech enhancement model;
[0026] The generator G trained in an adversarial learning manner is the first specific person speech enhancement model;
[0027] Using the real noisy speech of a specific person in the fourth training data as input data of the adversarial network generator G, and using the fourth enhanced speech as the target output of the generator G, and inputting it into the adversarial network of the first specific person speech enhancement model;
[0028] The generator G trained again in an adversarial learning manner is a speech enhancement model for a specific person. Further preferably, the generator G trained in an adversarial learning manner comprises the steps of:
[0029] The training data is input into the discriminator D, and the target output of the discriminator D is "1". The discriminator D is trained through the error back propagation algorithm to learn the data distribution of the training data;
[0030] The generator G is trained by the back error propagation algorithm, wherein, in the training of the generator G, the training data is input into the generator G, and the target output of the generator G is to make the output of the discriminator D be "1";
[0031] The discriminator D is trained by the back error propagation algorithm. In the training of the discriminator D, the signal generated by the generator G is input into the discriminator D, and the target output of the discriminator D is "0";
[0032] The parameters of the discriminator D are fixed, and the hidden layer parameters of the generator G are adjusted. The generator G continuously generates signals and discriminates the generated signals through the discriminator D until the discriminator D outputs "1". The training ends. The mapping relationship of the generator G obtained through training is the required speech enhancement model.
[0033] In the above training process, the training input data of the generator G is expressed as The hidden layer represents the parameters z, p z (z) represents the distribution learned by the generator G, Represents the distribution of real training data, and the training process of G is to minimize the loss function: Among them, V(G) represents the loss function, and the additional L1 regularization term minimizes the loss of the generated signal of the generator G and the training data to the corresponding speech x The Euclidean distance between them, λ is the set weight parameter.
[0034] Preferably, the method of extracting speech feature information and identifying the category of a person through a deep neural network comprises the steps of:
[0035] The deep neural network is trained using fifth training data, wherein the fifth training data includes specific person voice information and corresponding category information, wherein the specific person voice information is voice information of one or more specific persons collected under a set quiet environment noise, and the category information is classification information of different specific persons;
[0036] Extracting characteristic parameters of a speech input signal, wherein the characteristic parameters include any one or a combination of logarithmic power spectrum parameters, Mel-frequency cepstrum parameters, and linear prediction parameters;
[0037] Input the characteristic parameters of the speech input signal into the deep neural network to obtain sample features;
[0038] The sample features are passed through a Softmax classifier to output a classification result, namely, a person category; the person category includes categories of different specific persons and categories of non-specific persons.
[0039] A speech enhancement system, comprising:
[0040] Voice input devices;
[0041] processor;
[0042] Memory;
[0043] as well as
[0044] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, the programs causing the computer to perform the above method.
[0045] A smart device, wherein a memory of the smart device pre-stores a trained non-specific person speech enhancement model; a microphone of the smart device collects pure speech or noisy speech signals of one or more specific persons; and a program run by a processor of the smart device enables the smart device to execute the speech enhancement method as described above.
[0046] The voice enhancement method, system and intelligent device of the present invention have the following advantages:
[0047] (1) By training a specific person speech enhancement model, the speech data of the user of the smart device (specific person) is used to perform targeted speech enhancement on one or more specific persons. Compared with traditional deep learning-based speech enhancement technology, this can effectively improve the enhancement effect of the specific person's speech and reduce the interference of external non-steady-state noise on the specific person's speech.
[0048] (2) The neural network is trained using both the clean and noisy speech of a specific person. Compared with the traditional technical solution of training with only clean speech, the accuracy of speaker recognition in actual noisy scenarios is improved.
[0049] (3) Based on the non-specific person speech enhancement model, a small amount of specific person speech information is integrated for tuning and training. Compared with the technical solution of building a huge training set in the existing technology, it can effectively reduce the amount of data training and quickly and effectively improve the speech enhancement effect of specific users of smart devices. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 is an application scenario diagram of an embodiment of the present invention;
[0051] Figure 2 is a flow chart of a speech enhancement method according to an embodiment of the present invention;
[0052] Figure 3 is a flowchart of sub-step S01 in a preferred implementation mode;
[0053] Figure 4 is a flowchart of sub-step S02 in a preferred implementation mode;
[0054] Figure 5 is a flowchart of sub-step S101 in a preferred implementation mode;
[0055] Figure 6 is a flowchart of sub-step S1014 in a preferred implementation mode;
[0056] Figure 7 is a flowchart of an additional sub-step S101' in another preferred embodiment;
[0057] Figure 8is a flowchart of sub-step S102 in a preferred implementation mode;
[0058] Fig. 9 is a flowchart of sub-step S102 in another preferred implementation manner;
[0059] Fig.10 is a flowchart of sub-step S102 in another preferred implementation manner;
[0060] Fig.11 is a flowchart of sub-step S201 in a preferred implementation mode;
[0061] Fig.12 It is a structural diagram of a speech enhancement system. DETAILED DESCRIPTION
[0062] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.
[0063] The speech enhancement method, system and intelligent device in the embodiments of the present invention are mainly used in mobile phones, cars, smart home appliances, speakers and other applications that are controlled, awakened or interacted with through voice interfaces. Figure 1 shown.
[0064] An embodiment of a speech enhancement method of the present invention is shown in the flowchart as follows: Figure 2 As shown, including:
[0065] Step S01, using different training data to train different speech enhancement models;
[0066] Step S02: Select a corresponding speech enhancement model to perform speech enhancement according to the characteristics of the input speech.
[0067] In a preferred embodiment, the step S01 uses different training data to train different speech enhancement models, as shown in the flowchart. Figure 3 As shown, including:
[0068] Step S101: using the first training data to train a non-specific speaker speech enhancement model;
[0069] Step S102: training a specific person speech enhancement model using specific person training data based on the non-specific person speech enhancement model.
[0070] In a preferred embodiment, the step S02 selects a corresponding speech enhancement model to perform speech enhancement according to the characteristics of the input speech, as shown in the flowchart. Figure 4 As shown, including:
[0071] Step S201, extracting feature information of the input speech, and identifying the category of people through a deep neural network; the category of people includes categories of different specific people and categories of non-specific people;
[0072] Step S202: selecting a speech enhancement model for a non-specific person or speech enhancement models for different specific persons according to the person category;
[0073] Step S203: Input the speech data into the selected speech enhancement model to obtain enhanced speech information. In a preferred embodiment, the step S101: Use the first training data to train the non-specific person speech enhancement model, as shown in the flowchart. Figure 5 As shown, including:
[0074] Step S1011, mixing the standard speech data and the noise data according to a preset signal-to-noise ratio to obtain a speech signal as a simulated noisy speech;
[0075] Step S1012: using a standard speech database as clean speech, simulating noisy speech and its corresponding clean speech together to form first training data;
[0076] Step S1013: using the simulated noisy speech as input data of the adversarial network generator G, and using the clean speech as the target output of the generator G, and inputting it into the generative adversarial network;
[0077] Step S1014, the generator G trained by adversarial learning is the non-specific person speech enhancement model.
[0078] In this embodiment, the generative adversarial network can train the deep neural network in a semi-supervised manner through adversarial learning, and perform self-training in a zero-sum game manner, thereby reducing the demand for training data. Among them, the generative adversarial network includes a generator G and a discriminator D. The generator G is used to generate a generated signal based on the input data to approximate the target output; the discriminator D is used to determine whether the generated signal generated by the generator G is pure speech data and judge it as "1" or "0". When it is judged as "1", the parameters of the discriminator D are adjusted, and when it is judged as "0", the parameters of the generator G are adjusted; when the generative adversarial network is trained, the generator G and the discriminator D continue to train in an adversarial learning manner until the two enter a state of equilibrium; after the training is completed, the generator G is used to perform actual speech enhancement.
[0079] In the above implementation, the non-specific person speech enhancement model is trained using at least 100,000 speech data. The standard speech data comes from a standard speech library, and preferably, the standard speech library uses the TIMIT speech database. The standard noise data comes from a standard noise library, and preferably, the standard noise library uses the NOIZEUS noise database. Among them, the TIMIT speech library is a non-specific person pure speech, and its speech sampling frequency is 16kHz, with a total of 6,300 sentences, each of which is about 3 seconds. It is specifically recorded by 630 people from different language areas, different genders, and different age groups in the United States, and is marked and segmented. The NOIZEUS noise database was collected by researchers at the University of Texas at Dallas, with a total of eight noise types, namely: Babble (crowd of people), Car, Exhibition hall, Restaurant, Street, Airport, Train station, and Train.
[0080] The simulated noisy speech database is composed of pure speech and noise mixed according to a preset signal-to-noise ratio. In the embodiment of the present invention, the pure speech of the TIMIT speech library and the 8 types of noise in the NOIZEUS noise library are mixed with different signal-to-noise ratios (signal-to-noise ratios of 0, 5, 10dB, and 15dB) to obtain noisy speech with signal-to-noise ratios of 0, 5, 10dB, and 15dB, that is, each speech can be mixed to obtain 8*4=32 noisy speeches. If 516 voices are selected from TIMIT, a total of 16512 simulated noisy speeches are obtained. The present invention adopts a multi-signal-to-noise ratio aliasing strategy, constructs training data with different noise scenes and different signal-to-noise ratios to improve the generalization performance of the model, and ensures accurate recognition rate in complex noise scenes and different signal-to-noise ratios.
[0081] In the above implementation, an end-to-end deep neural network is trained, and a zero-sum game is played between the generator G and the discriminator D. First, the discriminator D back-propagates to learn pure speech, and then the discriminator D back-propagates the speech generated by the generator G and discriminates it. In this process, the generator trains G and the discriminator D at the same time; finally, the parameters of the generator D are frozen, and the generator G back-propagates until the generator D makes a misjudgment. The generator G trained in the adversarial learning mode in step S1014 is shown in the flowchart as follows: Figure 6 As shown, including:
[0082] Step S10141, training discriminator D, inputting clean speech data into discriminator D, the target output of discriminator D is "1", and training discriminator D through error back propagation algorithm to make it learn the data distribution of clean speech;
[0083] Step S10142, training the generator G by a back error propagation algorithm, wherein in the training of the generator G, a simulated noisy speech is input to the generator G, and the target output of the generator G is to make the output of the discriminator D be "1";
[0084] Step S10143, training the discriminator D by a back error propagation algorithm, wherein, in the training of the discriminator D, the signal generated by the generator G is input into the discriminator D, and the target output of the discriminator D is "0";
[0085] Step S10144, fix the parameters of the discriminator D, adjust the hidden layer parameters of the generator G, generate a signal and input it into the discriminator D; wherein, the input data of the generator G is simulated noisy speech, and the target output of the generator G is to make the output of the discriminator D "1"; the generator G continuously generates signals and discriminates the generated signals through the discriminator D until the discriminator D outputs "1" and the training is terminated. The mapping relationship of the generator G obtained through training is the non-specific person speech enhancement model.
[0086] In the above training process, the training input of G is a noisy speech signal. and the hidden layer represents the parameters z, p z (z) represents the distribution learned by G, Represents the distribution of real training data. The actual purpose of training is to make p z (z) As close as possible The output of G is the enhanced speech Therefore, in theory, the training process of G can be described as minimizing the following loss function:
[0087]
[0088] Among them, V(G) represents the loss function, with an L1 regularization term added, which actually minimizes the generated signal of the generator G and the pure speech x The Euclidean distance between them, λ is the weight parameter; the purpose of training is to obtain G with the minimum cost of the function, As close to pure speech as possible x .
[0089] Through the above training process, the generator G can learn the mapping relationship between simulated noisy speech and clean speech for actual speech enhancement. The mapping relationship trained by the generator G is the non-specific person speech enhancement model. The noisy speech signal collected by the smart device is input into the non-specific person speech enhancement model for signal processing and then outputs the enhanced speech signal.
[0090] In another preferred embodiment, after the first training data is used to train the non-specific person speech enhancement model in step S101, step S101' is performed to train a new non-specific person speech enhancement model based on the non-specific person speech enhancement model using the second training data. The flowchart is as follows: Figure 7 As shown, the step includes:
[0091] Step S101'1, the simulated noisy speech synthesized in the first training data is processed by a statistical speech enhancement algorithm to generate a first enhanced speech, and the simulated noisy speech and the first enhanced speech are used as the second training data; the statistical speech enhancement algorithm includes any one or a combination of the IMCRA-OMLSA algorithm, the IMCRA minimum controlled iterative averaging algorithm, and the optimal modified logarithmic amplitude estimation algorithm;
[0092] Step S101'2: using the simulated noisy speech as input data of the adversarial network generator G, using the first enhanced speech as the target output of the generator G, and inputting it into the adversarial network of the current non-specific speaker speech enhancement model;
[0093] Step S101'3: The generator G trained in an adversarial learning manner is a new non-specific speaker speech enhancement model.
[0094] In this embodiment, the simulated noisy speech synthesized in the first training data is processed by the IMCRA-OMLSA algorithm, the IMCRA minimum control iterative averaging algorithm, the optimal modified logarithmic amplitude estimation algorithm, etc. to obtain the first enhanced speech, and the simulated noisy speech and the first enhanced speech are used as training data pairs, and input into the non-specific person speech enhancement model trained in the above embodiment, and adversarial network training is performed as described in the above steps S10141 to S10144, that is, the generator G is trained by the back error propagation algorithm, wherein, in the training of the generator G, The generator G is simulated to be input with noisy speech, and the target output of the generator G is to make the output of the discriminator D be "1". The discriminator D is trained by the back error propagation algorithm, wherein, during the training of the discriminator D, the signal generated by the generator G is input into the discriminator D, and the target output of the discriminator D is "0". The parameters of the discriminator D are fixed, and the hidden layer parameters of the generator G are adjusted. The generator G continuously generates signals and discriminates the generated signals through the discriminator D until the discriminator D outputs "1". The training is terminated, and the mapping relationship of the generator G obtained by training is the new non-specific person speech enhancement model.
[0095] In this embodiment, the specific person training data in step S102 includes the third training data and / or the fourth training data.
[0096] In a preferred embodiment, the third training data includes a specific person's noisy speech and its corresponding specific person's clean speech, the specific person's noisy speech is a speech synthesized from the specific person's clean speech and a standard noise database, and the specific person's clean speech is a specific person's speech collected under quiet environmental noise. In this embodiment, the specific person refers to one or more device users set in advance, the specific person's clean speech is a plurality of voice information of a specific person collected in advance under a set quiet environmental noise, and the specific person's noisy speech is a speech formed by mixing the specific person's clean speech and standard noise data according to a preset signal-to-noise ratio.
[0097] In a preferred embodiment, the fourth training data includes a specific person's real noisy speech and a corresponding fourth enhanced speech, the specific person's real noisy speech is a specific person's noisy speech collected in an actual scene noise environment, and the fourth enhanced speech is a speech information obtained after the specific person's real noisy speech is processed by a statistical speech enhancement algorithm. In this embodiment, the specific person refers to one or more device users set in advance, the specific person's real noisy speech is a plurality of speech information of a specific person collected in advance in an actual scene or a specific person's speech information collected in real time when the device is used in an actual scene, and any one or a combination of the IMCRA-OMLSA algorithm, the IMCRA minimum control iterative average algorithm, and the optimal modified logarithmic amplitude estimation algorithm processes the specific person's real noisy speech into the fourth enhanced speech.
[0098] In a preferred embodiment, the step S102 uses the specific person training data to train the specific person speech enhancement model based on the non-specific person speech enhancement model, and the flow chart is as follows: Figure 8 As shown, the steps include:
[0099] S1021, using the noisy speech of the specific person in the third training data as input data of the adversarial network generator G, and using the pure speech of the specific person as the target output of the generator G;
[0100] S1022, inputting the data into the adversarial network corresponding to the current non-specific person speech enhancement model;
[0101] S1023. The generator G trained by adversarial learning is a speech enhancement model for a specific person.
[0102] In this embodiment, the noisy speech of a specific person and the pure speech of a specific person are used as training data pairs and input into the non-specific person speech enhancement model obtained by training in any of the above embodiments. Adversarial network training is performed as described in the above steps S10141 to S10144, that is, the generator G is trained by the reverse error propagation algorithm, wherein, in the training of the generator G, the noisy speech of the specific person is input into the generator G, and the target output of the generator G is to make the output of the discriminator D "1"; the discriminator D is trained by the reverse error propagation algorithm, wherein, in the training of the discriminator D, the signal generated by the generator G is input into the discriminator D, and the target output of the discriminator D is "0"; the parameters of the discriminator D are fixed, and the hidden layer parameters of the generator G are adjusted. The generator G continuously generates signals and discriminates the generated signals through the discriminator D until the discriminator D outputs "1" and the training is terminated. The mapping relationship of the generator G obtained by training is the specific person speech enhancement model.
[0103] Since the specific person speech enhancement model of this embodiment is parameter-tuned on the basis of the non-specific person speech enhancement model, the non-specific person speech enhancement model itself is fully trained using large-scale data. On this basis, a small amount of specific person speech information is integrated for tuning and training, which can improve the speech enhancement effect of specific users of smart devices. Usually, the third training data generally only requires dozens to hundreds of speech data.
[0104] In another preferred embodiment, the step S102 uses the specific person training data to train the specific person speech enhancement model based on the non-specific person speech enhancement model, as shown in the flowchart. Fig. 9 As shown, the steps include:
[0105] S1021′, using the real noisy speech of a specific person in the fourth training data as input data of the adversarial network generator G, and using the fourth enhanced speech as the target output of the generator G;
[0106] S1022', inputting the data into the adversarial network corresponding to the current non-specific person speech enhancement model;
[0107] S1023', the generator G trained by adversarial learning is the speech enhancement model for a specific person.
[0108] In this embodiment, the real noisy speech of a specific person and the fourth enhanced speech are used as training data pairs and input into the non-specific person speech enhancement model trained by any of the above embodiments, and the adversarial network training is performed using the method of steps S10141 to S10144, that is, the generator G is trained by the reverse error propagation algorithm, wherein, in the training of the generator G, the real noisy speech of the specific person is input into the generator G, and the target output of the generator G is to make the output of the discriminator D "1"; the discriminator D is trained by the reverse error propagation algorithm, wherein, in the training of the discriminator D, the signal generated by the generator G is input into the discriminator D, and the target output of the discriminator D is "0"; the parameters of the discriminator D are fixed, and the hidden layer parameters of the generator G are adjusted, the generator G continuously generates signals and discriminates the generated signals through the discriminator D until the discriminator D outputs "1" and the training is terminated, and the mapping relationship of the generator G obtained by training is the specific person speech enhancement model. The specific person speech enhancement model of this embodiment uses real noisy speech and statistically processed enhanced speech as training data pairs, which can be collected in real time when the device is used in actual scenarios, and the training can be completed without collecting speech in a quiet environment in advance.
[0109] In another preferred embodiment, the step S102 uses the specific person training data to train the specific person speech enhancement model based on the non-specific person speech enhancement model, as shown in the flowchart. Fig.10 As shown, the steps include:
[0110] Step S1021", using the noisy speech of the specific person in the third training data as input data of the adversarial network generator G, and using the clean speech of the specific person as the target output of the generator G, and inputting it into the adversarial network of the current non-specific person speech enhancement model;
[0111] Step S1022”: the generator G trained in an adversarial learning manner is a first specific person speech enhancement model;
[0112] Step S1023″, using the real noisy speech of the specific person in the fourth training data as input data of the adversarial network generator G, and using the fourth enhanced speech as the target output of the generator G, and inputting it into the adversarial network of the first specific person speech enhancement model;
[0113] In step S1024", the generator G obtained by retraining in an adversarial learning manner is the speech enhancement model for a specific person.
[0114] In this embodiment, the noisy speech of a specific person and the pure speech of a specific person are first used as training data pairs and input into the non-specific person speech enhancement model obtained by training in any of the above embodiments, and the adversarial network training is performed using the method of steps S10141 to S10144, that is, the generator G is trained by the reverse error propagation algorithm, wherein, in the training of the generator G, the noisy speech of the specific person is input into the generator G, and the target output of the generator G is to make the output of the discriminator D "1"; the discriminator D is trained by the reverse error propagation algorithm, wherein, in the training of the discriminator D, the signal generated by the generator G is input into the discriminator D, and the target output of the discriminator D is "0"; the parameters of the discriminator D are fixed, the hidden layer parameters of the generator G are adjusted, the generator G continuously generates signals and discriminates the generated signals through the discriminator D until the discriminator D outputs "1" and the training is terminated, and the mapping relationship of the generator G obtained by training is the first specific person speech enhancement model (preliminary specific Human speech enhancement model), on this basis, the real noisy speech of the specific person and the third enhanced speech are used as training data pairs, and are input into the above-mentioned first specific person speech enhancement model, and the method as the above-mentioned steps S10141 to S10144 is continued to be used for adversarial network training, that is, the generator G is trained by the reverse error propagation algorithm, wherein, in the training of the generator G, the real noisy speech of the specific person is input into the generator G, and the target output of the generator G is to make the output of the discriminator D "1"; the discriminator D is trained by the reverse error propagation algorithm, wherein, in the training of the discriminator D, the signal generated by the generator G is input into the discriminator D, and the target output of the discriminator D is "0"; the parameters of the discriminator D are fixed, and the hidden layer parameters of the generator G are adjusted, the generator G continuously generates signals and discriminates the generated signals through the discriminator D until the discriminator D outputs "1" and the training is ended, and the mapping relationship of the generator G obtained by training is the final specific person speech enhancement model. The specific person speech enhancement model of this embodiment is based on the non-specific person speech enhancement model trained in any of the above embodiments. The third training data and the fourth training data are successively used for adversarial training, and the network model parameters are multiply optimized. The obtained specific person speech enhancement model can improve the speech enhancement effect of specific users of smart devices.
[0115] In a preferred embodiment, the step S201 extracts voice feature information and identifies the person category through a deep neural network, as shown in the flowchart. Fig.11 As shown, the steps include:
[0116] Step S2011, using fifth training data to train the deep neural network; the fifth training data includes specific person voice information and corresponding category information, the specific person voice information is voice information of one or more specific persons collected under a set quiet environment noise, and the category information is classification information of different specific persons;
[0117] Step S2012: extracting characteristic parameters of the speech input signal, wherein the characteristic parameters include any one or a combination of logarithmic power spectrum parameters, Mel-frequency cepstrum parameters, and linear prediction parameters;
[0118] Step S2013, inputting the characteristic parameters of the speech input signal into the deep neural network to obtain sample features;
[0119] Step S2014: The sample features are passed through a Softmax classifier to output a classification result, i.e., a personnel category.
[0120] In this embodiment, one or more specific persons are set in the device in advance and the voice information of the corresponding specific person is collected under the set quiet environment noise. For example, the smart device has three common users (specific persons) A, B, and C. The voice data of A, B, and C in a quiet environment are collected respectively, and the classification label is specified as the category information to obtain the fifth training data. The set quiet environment noise is a noise decibel value range set according to different application scenarios (such as conference rooms, outdoor, hospitals, etc.) (the decibel value range of the quiet environment noise in this embodiment is below 40 decibels). Through the deep neural network learning algorithm, the fifth training data is used to train a deep neural network for identifying personnel categories. In this embodiment, the Mel inverse spectrum, speech signal power spectrum, and perceptual linear prediction coefficient of the speech input signal are first extracted and input into the deep neural network, and then the DNN is used to further extract the sample features of the specific person's voice. Finally, the sample features are output through the Softmax classifier to classify the results, that is, the personnel belong to a specific person (A or B or C) or a non-specific person.
[0121] In this embodiment, step S202 selects a non-specific person speech enhancement model or speech enhancement models of different specific persons according to the person category, and selects a specific person speech enhancement model or a non-specific person speech enhancement model of the corresponding specific person trained in any embodiment according to whether the identified person belongs to a specific person (A or B or C) or a non-specific person.
[0122] In this embodiment, step S203, the voice data is input into the selected voice enhancement model to obtain enhanced voice information, and the voice data of the person is input into the selected voice enhancement model. The voice enhancement model calculates the mapping relationship of the generator G to obtain enhanced voice data corresponding to the voice data of the person.
[0123] A computer-readable storage medium stores a computer program for electronic data exchange, wherein when the computer program is executed by a processor, the computer executes the above method.
[0124] A speech enhancement system, the system structure diagram is as follows Fig.12 As shown, including:
[0125] Voice input devices;
[0126] processor;
[0127] Memory;
[0128] as well as
[0129] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, the programs causing the computer to perform the above method.
[0130] In a preferred embodiment, a smart device is pre-trained with a non-specific person speech enhancement model (the non-specific person speech enhancement model trained using any of the above embodiments) before leaving the factory, the trained non-specific person speech enhancement model is stored in the memory of the smart device, and the microphone of the smart device collects pure speech or noisy speech signals of one or more specific persons; the program run by the processor of the smart device enables the smart device to execute the speech enhancement method as described above, and when a person uses it, the specific person is set and the corresponding speech is collected according to the requirements of the software interface of the smart device.
[0131] Of course, those skilled in the art should realize that the above embodiments are only used to illustrate the present invention, and are not intended to limit the present invention. As long as they are within the scope of the present invention, any changes or modifications to the above embodiments will fall within the protection scope of the present invention.
Claims
1. A speech enhancement method, characterized in that: include: Using the first training data to train a non-specific person speech enhancement model; Based on the non-specific person speech enhancement model, a new non-specific person speech enhancement model is trained using the second training data, including: the simulated noisy speech synthesized in the first training data is processed by a statistical speech enhancement algorithm to generate a first enhanced speech, and the second training data is the simulated noisy speech and its corresponding first enhanced speech; the statistical speech enhancement algorithm includes any one or a combination of the IMCRA-OMLSA algorithm, the IMCRA minimum control iterative average algorithm, and the optimal modified logarithmic amplitude estimation algorithm; the simulated noisy speech is used as the input data of the adversarial network generator G, and the first enhanced speech is used as the target output of the generator G, and is input into the adversarial network of the current non-specific person speech enhancement model; the generator G trained in the adversarial learning manner is the new non-specific person speech enhancement model; Based on the non-specific person speech enhancement model, the specific person training data is used to train the specific person speech enhancement model; Extract voice feature information and identify person categories through deep neural networks; The voice data is input into the corresponding model according to the person category to obtain enhanced voice information.
2. The speech enhancement method according to claim 1, characterized in that: The method of using the first training data to train the non-specific person speech enhancement model comprises the following steps: The speech signal synthesized by mixing the standard speech data and the noise data according to the preset signal-to-noise ratio is used as the simulated noisy speech; A standard speech database is used as clean speech, and the simulated noisy speech and its corresponding clean speech together constitute the first training data; The simulated noisy speech is used as the input data of the adversarial network generator G, and the clean speech is used as the target output of the generator G, which is input into the generative adversarial network; The generator G trained by adversarial learning is the non-specific person speech enhancement model.
3. The speech enhancement method according to claim 1, characterized in that: The specific person training data includes third training data and / or fourth training data, the third training data includes the noisy speech of the specific person and its corresponding clean speech of the specific person, the noisy speech of the specific person is the speech synthesized by the clean speech of the specific person and the standard noise database according to a preset signal-to-noise ratio, and the clean speech of the specific person is the speech of the specific person collected under a set quiet environment noise; the fourth training data includes the real noisy speech of the specific person and the corresponding fourth enhanced speech, the real noisy speech of the specific person is the noisy speech of the specific person collected in an actual scene noise environment, and the fourth enhanced speech is the speech information obtained after the real noisy speech of the specific person is processed by a statistical speech enhancement algorithm.
4. The speech enhancement method according to claim 3, characterized in that: The method of training the specific person speech enhancement model using the specific person training data based on the non-specific person speech enhancement model comprises the following steps: The noisy speech of a specific person in the third training data is used as the input data of the adversarial network generator G, and the pure speech of the specific person is used as the target output of the generator G. Or using the real noisy speech of a specific person in the fourth training data as input data of the adversarial network generator G, and using the fourth enhanced speech as the target output of the generator G; Input the data into the adversarial network corresponding to the current non-specific person speech enhancement model; The generator G trained by adversarial learning is the speech enhancement model for a specific person.
5. The speech enhancement method according to claim 3, characterized in that: The method of training the specific person speech enhancement model using the specific person training data based on the non-specific person speech enhancement model comprises the following steps: The noisy speech of the specific person in the third training data is used as the input data of the adversarial network generator G, and the clean speech of the specific person is used as the target output of the generator G, which is input into the adversarial network of the current non-specific person speech enhancement model; The generator G trained in an adversarial learning manner is a first specific person speech enhancement model; Using the real noisy speech of a specific person in the fourth training data as input data of the adversarial network generator G, and using the fourth enhanced speech as the target output of the generator G, and inputting it into the adversarial network of the first specific person speech enhancement model; The generator G trained again in an adversarial learning manner is the speech enhancement model for a specific person.
6. The speech enhancement method according to claim 1, characterized in that: The generator G trained in an adversarial learning manner comprises the following steps: The training data is input into the discriminator D, and the target output of the discriminator D is "1". The discriminator D is trained through the error back propagation algorithm to learn the data distribution of the training data; The generator G is trained by the back error propagation algorithm, wherein the training data is input into the generator G during the training of the generator G, and the target output of the generator G is to make the output of the discriminator D be "1"; The discriminator D is trained by the back error propagation algorithm. In the training of the discriminator D, the signal generated by the generator G is input into the discriminator D, and the target output of the discriminator D is "0"; The parameters of the discriminator D are fixed, and the hidden layer parameters of the generator G are adjusted. The generator G continuously generates signals and discriminates the generated signals through the discriminator D until the discriminator D outputs "1", and the training ends. The mapping relationship of the generator G obtained through training is the speech enhancement model; The training input data of generator G is represented as The hidden layer represents the parameters z, p z (z) represents the distribution learned by the generator G, Represents the distribution of real training data, and the training process of G is to minimize the loss function: Where V(G) represents the loss function, the additional L1 regularization term minimizes the Euclidean distance between the generated signal of the generator G and the corresponding speech x of the training data, and λ is the set weight parameter.
7. The speech enhancement method according to claim 1, characterized in that: The method of extracting speech feature information and identifying the category of a person through a deep neural network comprises the following steps: The deep neural network is trained using fifth training data, wherein the fifth training data includes specific person voice information and corresponding category information, wherein the specific person voice information is voice information of one or more specific persons collected under a set quiet environment noise, and the category information is classification information of different specific persons; Extracting characteristic parameters of a speech input signal, wherein the characteristic parameters include any one or a combination of logarithmic power spectrum parameters, Mel-frequency cepstrum parameters, and linear prediction parameters; Input the characteristic parameters of the speech input signal into the deep neural network to obtain sample features; The sample features are passed through a Softmax classifier to output a classification result, namely, a person category; the person category includes categories of different specific persons and categories of non-specific persons.
8. A speech enhancement system, characterized in that: include: Voice input devices; processor; Memory; as well as One or more programs, wherein the one or more programs are stored in a memory and configured to be executed by the processor, the programs causing the computer to execute the method according to any one of claims 1 to 7.
9. A smart device, characterized in that: The memory of the smart device pre-stores a trained non-specific person speech enhancement model; the microphone of the smart device collects pure speech or noisy speech signals of one or more specific persons; the program run by the processor of the smart device enables the smart device to execute the method described in any one of claims 1-7.
