Speech enhancement model training method, speech enhancement method and related device
By training the speech enhancement model, using multi-channel speech data to generate single-channel data for speech enhancement, the problem of poor overall real-time performance in the prior art is solved, and more efficient speech enhancement processing is achieved.
Patent Information
- Application Number
- CN202411353630.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art has poor overall real-time performance in the speech enhancement process of multi-channel voice data, making it difficult to engineer on the hardware platform, and each channel needs to be pre-processed separately.
By acquiring multi-channel pure speech data, generating single-channel noisy voice data and single-channel enhanced speech data, extracting sample feature data and generating sample label data, training speech enhancement models to process based on single-channel data during speech enhancement, Reduce the calculation amount and calculation complexity and improve real-time performance.
On the basis of ensuring the voice enhancement effect, the calculation amount and calculation complexity are reduced, and the processing speed and overall real-time nature of voice enhancement are improved.
Smart Images

Figure CN119091896A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of audio signal processing, and in particular to a training method for a speech enhancement model, a speech enhancement method and related devices. Background Art
[0002] With the development of audio signal processing technology, multi-channel voice data has been widely used in many scenarios because it has higher audio quality and can support more accurate audio analysis and more flexible audio processing and transmission. Under the influence of noise interference, transmission distortion, signal attenuation and other factors, the data quality of multi-channel voice data will be reduced. In order to improve the signal-to-noise ratio and intelligibility of speech, speech enhancement technology can be used to analyze and process the collected multi-channel voice data to enhance the useful part of the speech.
[0003] At present, considering that the beamforming algorithm has a better effect on enhancing far-field speech, the existing technology mostly uses a beamforming algorithm based on a microphone array to perform speech enhancement processing on multi-channel speech data. During the enhancement process, the existing technology needs to pre-process the speech data of each channel separately, such as performing echo cancellation on the speech data of each channel separately, which has the problem of poor overall real-time performance and is difficult to engineer on the hardware platform. Summary of the invention
[0004] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the technical defect of poor overall real-time performance in the prior art.
[0005] In a first aspect, an embodiment of the present application provides a method for training a speech enhancement model, comprising:
[0006] Obtain multi-channel pure voice data;
[0007] Generate single-channel noisy speech data according to the multi-channel clean speech data and preset noise data, and perform feature extraction on the single-channel noisy speech data to obtain sample feature data;
[0008] Generate single-channel enhanced voice data according to the multi-channel clean voice data and a preset AGC algorithm;
[0009] Generate sample label data based on the multi-channel clean speech data, the single-channel noisy speech data and the single-channel enhanced speech data;
[0010] The sample feature data is used as the input of the speech enhancement model to be trained, and the sample label data is used as the target output of the speech enhancement model to be trained. The speech enhancement model to be trained is trained, and a target speech enhancement model is obtained after the training is completed.
[0011] In a second aspect, an embodiment of the present application provides a speech enhancement method, comprising:
[0012] Obtain target multi-channel voice data;
[0013] Performing channel separation on the target multi-channel voice data to obtain target single-channel voice data;
[0014] Extracting features from the target single-channel speech data to obtain target speech feature data;
[0015] Inputting the target speech feature data into a target speech enhancement model, and obtaining model output data output by the target speech enhancement model; wherein the target speech enhancement model is trained based on the training method of the speech enhancement model described in any of the above embodiments;
[0016] Target enhanced speech data is obtained according to the model output data.
[0017] In a third aspect, an embodiment of the present application provides a training device for a speech enhancement model, comprising:
[0018] A first voice data acquisition module, used to acquire multi-channel pure voice data;
[0019] A first feature extraction module, used to generate single-channel noisy speech data according to the multi-channel clean speech data and preset noise data, and perform feature extraction on the single-channel noisy speech data to obtain sample feature data;
[0020] A first speech enhancement module, used for generating single-channel enhanced speech data according to the multi-channel clean speech data and a preset AGC algorithm;
[0021] A sample label generation module, used to generate sample label data based on the multi-channel clean speech data, the single-channel noisy speech data and the single-channel enhanced speech data;
[0022] The model training module is used to use the sample feature data as the input of the speech enhancement model to be trained, use the sample label data as the target output of the speech enhancement model to be trained, train the speech enhancement model to be trained, and obtain the target speech enhancement model after the training is completed.
[0023] In a fourth aspect, an embodiment of the present application provides a speech enhancement device, comprising:
[0024] A second voice data acquisition module, used to acquire target multi-channel voice data;
[0025] A channel separation module is used to perform channel separation on the target multi-channel voice data to obtain target single-channel voice data;
[0026] A second feature extraction module is used to extract features from the target single-channel speech data to obtain target speech feature data;
[0027] A model processing module, used for inputting the target speech feature data into a target speech enhancement model and obtaining model output data output by the target speech enhancement model; wherein the target speech enhancement model is trained based on the training method of the speech enhancement model described in any of the above embodiments;
[0028] The second speech enhancement module is used to obtain target enhanced speech data according to the model output data.
[0029] In a fifth aspect, an embodiment of the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the training method of the speech enhancement model described in any of the above embodiments, and / or the steps of the speech enhancement method described in any of the above embodiments.
[0030] In a sixth aspect, an embodiment of the present application provides a computer device, the computer device comprising: one or more processors, and a memory;
[0031] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the training method of the speech enhancement model described in any of the above embodiments and / or the steps of the speech enhancement method described in any of the above embodiments are executed.
[0032] In the speech enhancement model training method, speech enhancement method and related devices provided in some embodiments of the present application, multi-channel pure speech data is used to generate single-channel noisy data and single-channel enhanced speech data, and sample feature data is extracted from the single-channel noisy data. The present application generates sample label data based on multi-channel pure speech data, single-channel noisy speech data and single-channel enhanced speech data. And the sample feature data and sample label data are used for model training to obtain a target speech enhancement model. In this way, in the process of speech enhancement, when multi-channel speech data is collected, the target speech enhancement model can perform speech enhancement based on the single channel data of the multi-channel speech data, without processing the speech data of each channel separately, thereby reducing the amount of calculation and the complexity of calculation on the basis of ensuring the speech enhancement effect, thereby improving the processing speed of speech enhancement and improving the overall real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0034] Figure 1 A flowchart of a method for training a speech enhancement model in one embodiment;
[0035] Figure 2 A flowchart of the training steps of a speech enhancement model in one embodiment;
[0036] Figure 3 A network structure diagram of a speech enhancement model in one embodiment;
[0037] Figure 4 One of the flowcharts of the speech enhancement method in one embodiment;
[0038] Figure 5 The second flowchart of the speech enhancement method in one embodiment is shown;
[0039] Figure 6 is a schematic diagram of the structure of a training device for a speech enhancement model in one embodiment;
[0040] Figure 7 is a structural schematic diagram of a speech enhancement device in one embodiment;
[0041] Figure 8 FIG. 1 is a diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0043] In one embodiment, Figure 1 As shown, the present application provides a method for training a speech enhancement model, comprising:
[0044] S102: Acquire multi-channel pure voice data.
[0045] The multi-channel clean voice data may be multi-channel voice data with a signal-to-noise ratio greater than a specific signal-to-noise ratio threshold, for example, multi-channel voice data without noise, interference, or other irrelevant signals. In one example, the multi-channel clean voice data may only include human voice data.
[0046] It is understood that the multi-channel pure voice data can be obtained by any method, and this document does not impose any specific restrictions on this. For example, it can be obtained based on the voice data of an open source voice dataset.
[0047] In one example, multi-channel clean voice data can be recorded through multiple audio input devices. Furthermore, the setting orientation and / or the distance between each two audio input devices and the sound source position may be different. For example, the distance between each audio input device and the sound source position may be 1 meter, 3 meters, and 6 meters, respectively. In this way, the collected multi-channel clean voice data can be suitable for actual application scenarios. Subsequent model training based on multi-channel clean voice data can improve the performance of the model in actual use.
[0048] S104: Generate single-channel noisy speech data according to the multi-channel clean speech data and the preset noise data, and perform feature extraction on the single-channel noisy speech data to obtain sample feature data.
[0049] The single-channel noisy data may be single-channel speech data with a signal-to-noise ratio less than a specific signal-to-noise ratio threshold, and the single-channel noisy data includes both speech data and noise data. The sample feature data refers to speech feature data of noisy speech data, which is used as sample data for model training.
[0050] In this step, single-channel noise data can be generated based on the multi-channel clean speech data and the preset noise data. The noise data can be preset according to the actual situation, and this document does not impose any specific restrictions on this. For example, the noise data can include an open source noise data set and far-field and near-field noise data recorded by an audio input device.
[0051] It can be understood that the specific method of generating single-channel noise data can be determined according to actual conditions, and this article does not impose specific restrictions on this. It only needs to be generated based on multi-channel clean speech data and noise data. In one example, multi-channel clean speech data can be split into single-channel clean speech data, and the single-channel clean speech data and noise data can be mixed in a random mixing manner to obtain multiple single-channel noisy speech data with different signal-to-noise ratios. Multiple single-channel noisy speech data can constitute a noisy data set for model training.
[0052] S106: Generate single-channel enhanced voice data according to the multi-channel pure voice data and a preset AGC algorithm.
[0053] The single-channel enhanced speech data may be the single-channel speech data obtained by performing speech enhancement using a preset AGC (Automatic Gain Control). It is understood that the AGC algorithm may be determined according to actual conditions, and this document does not impose any specific restrictions on this, for example, it may be a beamforming algorithm.
[0054] In this step, the specific generation method of the single-channel enhanced voice data can be determined according to the actual situation, and this application does not impose specific restrictions on this. It only needs to be generated based on the multi-channel clean voice data and the preset AGC algorithm. Taking the AGC algorithm as a beamforming algorithm as an example, this application can perform beam enhancement processing on the multi-channel clean voice data based on the beamforming algorithm, and perform channel separation on the enhanced multi-channel clean voice data, and perform single-channel enhancement processing on the separated single-channel voice data to obtain single-channel enhanced data. Furthermore, the beam enhancement processing can be performed in an offline processing manner.
[0055] S108: Generate sample label data based on the multi-channel clean speech data, the single-channel noisy speech data and the single-channel enhanced speech data.
[0056] Among them, sample label data refers to label data corresponding to sample feature data, which is used to guide the model to adjust its model parameters during the model training process.
[0057] In this step, sample label data may be generated according to the multi-channel clean speech data, the single-channel noisy speech data, and the single-channel enhanced speech data. For example, a first signal-to-noise ratio corresponding to the multi-channel clean speech data, a second signal-to-noise ratio corresponding to the single-channel noisy speech data, and a third signal-to-noise ratio corresponding to the single-channel enhanced speech data may be determined respectively, and sample label data may be generated based on the first signal-to-noise ratio, the second signal-to-noise ratio, and the third signal-to-noise ratio.
[0058] S110: Using the sample feature data as the input of the speech enhancement model to be trained, using the sample label data as the target output of the speech enhancement model to be trained, training the speech enhancement model to be trained, and obtaining the target speech enhancement model after the training is completed.
[0059] Specifically, the sample feature data and the sample label data can constitute the training set of the speech enhancement model, which is used to guide the speech enhancement model to be trained to adjust its model parameters. The present application can use the sample feature data as the input of the speech enhancement model to be trained, and adjust the model parameters of the speech enhancement model to be trained according to the actual output data of the speech enhancement model to be trained and the sample label data, until the trained speech enhancement model is obtained. The trained speech enhancement model is the target speech enhancement model.
[0060] In the present application, multi-channel pure speech data is used to generate single-channel noisy data and single-channel enhanced speech data, and sample feature data is extracted from the single-channel noisy data. The present application generates sample label data based on multi-channel pure speech data, single-channel noisy speech data and single-channel enhanced speech data. And the sample feature data and sample label data are used for model training to obtain a target speech enhancement model. In this way, in the process of speech enhancement, when multi-channel speech data is collected, the target speech enhancement model can perform speech enhancement based on the single channel data of the multi-channel speech data, without processing the speech data of each channel separately, thereby reducing the amount of calculation and the complexity of calculation on the basis of ensuring the speech enhancement effect, thereby improving the processing speed of speech enhancement and improving the overall real-time performance.
[0061] In one embodiment, the speech enhancement model to be trained includes a noise reduction module and an enhancement module, wherein the noise reduction module refers to an artificial intelligence module for performing speech noise reduction, and the enhancement module refers to an artificial intelligence module for performing speech enhancement.
[0062] The sample label data includes noise reduction label data and enhancement label data. The noise reduction label data can be used to guide the parameter adjustment of the noise module, which can be obtained based on single-channel noisy speech data and multi-channel clean speech data. The enhancement label data can be used to guide the parameter adjustment of the enhancement module, which can be obtained based on single-channel noisy speech data and single-channel enhanced speech data.
[0063] like Figure 2 As shown, the sample feature data is used as the input of the speech enhancement model to be trained, and the sample label data is used as the target output of the speech enhancement model to be trained, and the speech enhancement model to be trained is trained, including:
[0064] S202: In the current training round, the sample feature data is input into the speech enhancement model to be trained, and the noise reduction training data generated by the noise reduction module of the speech enhancement model to be trained and the enhanced training data generated by the enhancement module of the speech enhancement model to be trained are obtained;
[0065] S204: Calculate the total loss value of the current training round based on the denoised training data, the enhanced training data, the denoised label data, and the enhanced label data;
[0066] S206: If it is determined according to the total loss value that the preset training end condition is not met, then enter the next training round.
[0067] Specifically, the training process of the speech enhancement model to be trained may include training rounds, and steps S202 to S206 may be performed in each training round. The following description takes one training round as an example, and the execution process of other training rounds may refer to the following description.
[0068] If the current training round is the Kth training round, then in the Kth training round, the sample feature data can be input into the speech enhancement model to be trained, so that the noise reduction module can generate the noise reduction training data of the Kth training round based on the model parameters of the Kth training round, and the gain module can generate the enhanced training data of the Kth training round based on the model parameters of the Kth training round. The present application can calculate the total loss value of the Kth training round based on the noise reduction training data of the Kth training round, the enhanced training data of the Kth training round, the noise reduction label data and the enhanced label data, and judge whether the preset training end condition is met accordingly. If the training end condition is not met, enter the (K+1)th training round and execute steps S202 to S206 again. Further, if the training end condition is met, it can be determined that the training is completed and the target speech enhancement data can be obtained.
[0069] In this way, the artificial intelligence module can be used to achieve speech noise reduction. Compared with traditional noise reduction algorithms such as Wiener filtering, the artificial intelligence module has a better suppression effect on non-steady-state noise. Therefore, using the trained target speech enhancement model for speech enhancement can have a better noise suppression effect.
[0070] It can be understood that the present application can use any method to calculate the total loss value of the current training round. This article does not impose any specific restrictions on this. It only needs to be calculated based on the denoised training data, enhanced training data, denoised label data and enhanced label data.
[0071] In one embodiment, the present application may calculate the first loss value of the current training round according to the denoised training data and denoised label data of the current training round. Also, the second loss value of the current training round may be calculated according to the enhanced training data and enhanced label data of the current training round, and the total loss value of the current training round may be obtained by weighted summing the first loss value of the current training round and the second loss value of the current training round.
[0072] It can be understood that the weighting coefficients corresponding to the first loss value and the second loss value can be determined according to the actual situation, and this document does not impose specific restrictions on this. Further, in one example, in order to make the model output of the speech enhancement model to be trained closer to the enhanced label data and take into account the noise reduction effect of the noise reduction module, the weighting coefficient corresponding to the first loss value can be smaller than the weighting coefficient corresponding to the second loss value.
[0073] In one embodiment, Figure 3As shown, the noise reduction module may include a first fully connected layer and a first gated loop layer, and the enhancement module includes a second gated loop layer and a third gated loop layer. It can be understood that the activation function of each fully connected layer and the activation function of each gated loop layer can be determined according to actual conditions. In one example, the activation function of the first fully connected layer, the activation function of the first gated loop layer, the activation function of the second gated loop layer, and the activation function of the third gated loop layer can all be tanh functions.
[0074] It can be understood that the dimensions of the first gated loop layer, the second gated loop layer and the third gated loop layer can be determined according to actual conditions such as the hardware configuration of the execution device and / or algorithm performance indicators, and this document does not make any specific restrictions on this.
[0075] Inputting the sample feature data into the speech enhancement model to be trained, and obtaining the noise reduction training data generated by the noise reduction module of the speech enhancement model to be trained, and the enhanced training data generated by the enhancement module of the speech enhancement model to be trained, including:
[0076] Step A1: inputting sample feature data into the first fully connected layer, and obtaining first fully connected result data output by the first fully connected layer;
[0077] Step A3: inputting the first fully connected result data into the first gated recurrent layer, and obtaining the first gated result data output by the first gated recurrent layer;
[0078] Step A5: splicing the sample feature data, the first fully connected result data and the first gated result data to obtain first spliced data;
[0079] Step A7: inputting the first spliced data into the second gated circulation layer, and obtaining second gated result data output by the second gated circulation layer;
[0080] Step A8: splicing the sample characteristic data, the first gated result data and the second gated result data to obtain second spliced data;
[0081] Step A11: inputting the second spliced data into the third gated circulation layer, and obtaining third gated result data output by the third gated circulation layer;
[0082] Step A13: generating denoised training data according to the first gating result data, and generating enhanced training data according to the third gating result data.
[0083] Please combine Figure 3In this embodiment, the noise reduction module may include a first fully connected layer and a first gated recurrent layer connected in sequence, and the gain module may include a second gated recurrent layer and a third gated recurrent layer connected in sequence. After the sample feature data is input into the speech enhancement model to be trained, the speech enhancement model to be trained may process the sample feature data through the first fully connected layer and generate first fully connected result data. The first fully connected result data is used as the input of the first gated recurrent layer, so that the first gated recurrent layer can generate the first gated result data based on the first fully connected result data.
[0084] The input of the second gated loop layer can be spliced by the sample feature data, the first fully connected result data and the first gated result data, so that the second gated loop layer can better utilize the information generated by the noise reduction module and generate the second gated result data accordingly. The input of the third gated loop layer can be spliced by the sample feature data, the first gated result data and the second gated result data, so that the third gated loop layer can better utilize the generated information of the noise reduction module and the generated information of the second gated loop layer, and generate the third gated result data accordingly. The speech enhancement model to be trained can generate noise reduction training data according to the first gated result data, and generate enhanced training data according to the third gated result data.
[0085] In this way, by using the first concatenated data as the input of the second gated recurrent layer and the second concatenated data as the input of the third gated recurrent layer, more data can be used for speech enhancement, thereby improving the speech enhancement effect of the target speech enhancement model. At the same time, overfitting of the model can also be prevented.
[0086] In one embodiment, generating denoised training data according to the first gating result data, and generating enhanced training data according to the third gating result data, comprises:
[0087] A131: input the first gating result into the second fully connected layer, and obtain the denoised training data generated by the second fully connected layer;
[0088] A133: Input the third gating result data into the third fully connected layer, and obtain the enhanced training data generated by the third fully connected layer.
[0089] Specifically, please combine Figure 3, the denoising module may include 2 fully connected layers and 1 gated recurrent layer, which are the first fully connected layer, the second fully connected layer and the first gated recurrent layer. The first fully connected layer, the first gated recurrent layer and the second fully connected layer are connected in sequence. The enhancement module may include 2 gated recurrent layers and 1 fully connected layer, which are the second gated recurrent layer, the third gated recurrent layer and the third fully connected layer. The second gated recurrent layer, the third gated recurrent layer and the third fully connected layer are connected in sequence. In one example, the activation function of the second fully connected layer may be a sigmoid function, and the activation function of the third fully connected layer may be a relu function.
[0090] The denoising module can use the first gating result data as the input of the second fully connected layer, so that the second fully connected layer generates denoising training data for supervised model training according to the first gating result data, thereby achieving the noise separation task in a supervised form. The gain module can input the third gating result data into the third fully connected layer, so that the third fully connected layer generates enhanced training data according to the third gating result data.
[0091] This embodiment can simplify the model structure of the speech enhancement model while ensuring the speech noise reduction and enhancement effect, and can perform speech enhancement through compact and effective model results. In this way, the computational complexity can be further reduced and the overall real-time performance can be improved. It can suppress non-steady-state noise and enhance far-field speech.
[0092] In one embodiment, feature extraction is performed on single-channel noisy speech data to obtain sample feature data, including:
[0093] Step B1: performing frame processing on the single-channel noisy speech data and obtaining multiple frames of noisy speech segments;
[0094] Step B3: For each frame of noisy speech segment, obtain the noisy spectrum corresponding to the frame of noisy speech segment, and use a preset filtering algorithm to compress the noisy spectrum into N frequency bands, calculate the frequency band energy value corresponding to each frequency band respectively, and generate speech feature data of the frame of noisy speech segment according to the frequency band energy values corresponding to the frame of noisy speech segment; wherein N is a positive integer, and the sample feature data includes the speech feature data of each frame of noisy speech segment.
[0095] In this embodiment, in order to further improve the real-time reasoning of speech enhancement, the input feature dimension of the model can be compressed.
[0096] Specifically, considering that speech signals are generally short-term signals, the present application can frame the single-channel noisy speech data to divide the single-channel noisy speech data into M short-term frames (i.e., M noisy speech segments), and process the short-term frames, where M is a positive integer.
[0097] For each noisy speech segment obtained after framing, the present application can generate speech feature data of the noisy speech segment according to steps B31 to B37. The following is an example of the i-th noisy speech segment, where i is a positive integer and i≤M.
[0098] Step B31: Perform time-frequency transformation on the noisy speech segment of the i-th frame, and obtain the noisy spectrum corresponding to the noisy speech segment of the i-th frame. For example, the time-frequency transformation can be performed using a Fourier transform algorithm.
[0099] Step B33: Using a preset filtering algorithm, compress the noisy spectrum corresponding to the noisy speech segment of the i-th frame into N frequency bands. For example, a triangular filter can be used for compression. It can be understood that the specific value of N can be determined based on actual factors such as the hardware configuration of the execution platform, and this document does not make specific restrictions on this.
[0100] Step B35: Calculate the band energy value of the noisy speech segment of the i-th frame in each frequency band respectively. For example, when N=2, the band energy value of the noisy speech segment of the i-th frame in the first frequency band and the band energy value of the noisy speech segment of the i-th frame in the second frequency band can be calculated respectively. It can be understood that the present application can calculate the band energy value in any way. In one example, the present application can normalize the energy spectrum of the noisy speech segment of the i-th frame in each frequency band respectively, and calculate the initial energy value corresponding to the frequency band according to the normalization result, and use the log value of the initial energy value as the band energy value of the noisy speech segment of the i-th frame in the frequency band.
[0101] Step B37: Generate speech feature data of the i-th frame of noisy speech segment according to the band energy value of each frequency band of the i-th frame of noisy speech segment. For example, the speech feature data of the i-th frame of noisy speech segment may include the band energy value of each frequency band of the i-th frame of noisy speech segment.
[0102] In one embodiment, for each frame of noisy speech segment, speech feature data of the frame of noisy speech segment is generated according to energy values of each frequency band corresponding to the frame of noisy speech segment, including:
[0103] For each frame of noisy speech segment, the total energy value of the frame of noisy speech segment is calculated, and according to the energy value of each frequency band corresponding to the frame of noisy speech segment, the first-order difference data and the second-order difference data corresponding to each frequency band are respectively calculated, and the total energy value of the frame of noisy speech segment, the energy values of each frequency band corresponding to the frame of noisy speech segment, each first-order difference data and each second-order difference data are used as the speech feature data of the frame of noisy speech segment.
[0104] In this embodiment, for the i-th frame of noisy speech segment, its speech feature data includes not only N frequency band energy values, but also the total energy value of the i-th frame of noisy speech segment and the first-order difference data and second-order difference data of the i-th frame of noisy speech segment in each frequency band. In one example, the total energy value may be a logarithmic value, and the first-order difference data and second-order difference data corresponding to each frequency band may be obtained by performing DCT (Discrete Cosine Transform) transformation on the frequency band energy corresponding to the frequency band.
[0105] In this way, by using the total frame energy value, each frequency band energy value, each first-order difference data and each second-order difference data as the speech feature data of the noisy speech segment, the speech feature data can more comprehensively and accurately reflect the speech information in the noisy speech segment on the basis of compressing the input feature dimension. Model training can be performed in this way to further improve the enhancement effect of the target speech enhancement model.
[0106] In one embodiment, the sample label data includes denoised label data and enhanced label data. The relevant description of the denoised label data and the enhanced label data can be found above, and will not be repeated here.
[0107] Generate sample label data based on multi-channel clean speech data, single-channel noisy speech data, and single-channel enhanced speech data, including:
[0108] Step C1: performing frame processing on the multi-channel clean speech data and obtaining multiple frames of clean speech segments;
[0109] Step C3: performing frame processing on the single-channel enhanced speech data and obtaining multiple frames of enhanced speech segments;
[0110] Step C5: for each clean speech segment, respectively calculate the power value of the first frequency band energy ratio of the clean speech segment and the corresponding noisy speech segment in each frequency band; wherein the denoising label data includes the power values of the first frequency band energy ratio corresponding to each clean speech segment;
[0111] Step C7: For each frame of enhanced speech segment, respectively calculate the power value of the second frequency band energy ratio between the frame of enhanced speech segment and the corresponding noisy speech segment in each frequency band; wherein the enhanced label data includes each second frequency band energy ratio power value corresponding to each frame of enhanced speech segment.
[0112] In this embodiment, the multi-channel clean speech data, the single-channel enhanced speech data and the single-channel noisy speech data can be processed by framing with the same number of frames, and M frames of multi-frame clean speech segments, M frames of enhanced speech segments and M frames of noisy speech segments are obtained respectively. The i-th frame multi-frame clean speech segment, the i-th frame enhanced speech segment and the i-th frame noisy speech segment correspond to each other.
[0113] The present application can respectively calculate the band energy values of each frame of pure speech segment in N frequency bands, and the band energy values of each frame of enhanced speech segment in N frequency bands. The specific calculation process can refer to the above-mentioned relevant description of calculating the band energy values of noisy speech segments in N frequency bands, which will not be repeated here.
[0114] For the jth frequency band of the i-th frame, the present application can calculate the power value of the ratio between the frequency band energy value of the i-th frame clean speech segment in the j-th frequency band and the frequency band energy value of the i-th noisy speech segment in the j-th frequency band, and obtain the power value of the first frequency band energy ratio of the i-th frame clean speech segment in the j-th frequency band. Wherein, j is a positive integer, and j≤N.
[0115] According to the above process, the present application can obtain the power of the first frequency band energy ratio of each frame of clean speech segment in N frequency bands. The power of the first frequency band energy ratio of M frames of clean speech segments in N frequency bands can constitute the noise reduction label data of the present application.
[0116] For the jth frequency band of the i-th frame, the present application can calculate the power value of the ratio between the frequency band energy value of the enhanced speech segment of the i-th frame in the j-frequency band and the frequency band energy value of the i-th noisy speech segment in the j-frequency band, and obtain the power value of the second frequency band energy ratio of the enhanced speech segment of the i-th frame in the j-frequency band.
[0117] According to the above process, the present application can obtain the power of the second frequency band energy ratio of each frame enhanced speech segment in N frequency bands. The power value of the second frequency band energy ratio of M frames enhanced speech segments in N frequency bands can constitute the enhanced label data of the present application.
[0118] The sample label data of this embodiment is used for model training, and the model output data is the power of the target frequency band energy ratio corresponding to N frequency bands, which can be used to reflect the gain ratio of speech enhancement.
[0119] Furthermore, when the activation function of the second fully connected layer is a sigmoid function, the value range of the denoising label data is [0,1]. If the maximum gain ratio represented by the model output data is max_scale, and max_scale is greater than 1, the output data of the third fully connected layer has a value range of [0,max_scale]. Based on this, the activation function of the third fully connected layer can be a relu function, and its maximum value is set to max_scale. Furthermore, max_scale can be determined based on the gain ratio of the far-field energy minimum speech data and its enhanced data.
[0120] In one embodiment, Figure 4 As shown, the present application provides a speech enhancement method, comprising:
[0121] S402: Acquire target multi-channel voice data;
[0122] S404: performing channel separation on the target multi-channel voice data to obtain target single-channel voice data;
[0123] S406: extracting features from the target single-channel speech data to obtain target speech feature data;
[0124] S408: Inputting the target speech feature data into the target speech enhancement model, and obtaining model output data output by the target speech enhancement model; wherein the target speech enhancement model is trained based on the training method of the speech enhancement model in any of the above embodiments;
[0125] S410: Obtain target enhanced speech data according to the model output data.
[0126] The target multi-channel speech data refers to the multi-channel speech data to be enhanced, which may include a noise signal and a speech signal. It is understood that the target multi-channel speech data may be obtained by any method, and this document does not impose any specific restrictions on this. For example, it may be collected by a microphone array.
[0127] In this embodiment, after acquiring the target multi-channel speech data, it can be channel-separated to obtain the corresponding target single-channel speech data, and the speech features of the target single-channel speech data can be extracted to obtain the target speech feature data. The present application can input the target speech feature data into the target speech enhancement model to perform noise reduction and enhancement operations on the target single-channel speech data through the target speech enhancement model, and obtain the model output data output by the target speech enhancement model.
[0128] It can be understood that the data form of the model output data is associated with the data form of the sample label data in the model training data, and can be used to guide the speech enhancement operation of the target single-channel speech data. Therefore, the present application can perform speech enhancement operation on the target single-channel speech data according to the model output data, and obtain the target enhanced speech data.
[0129] For example, when the sample label data includes the power values of each first frequency band energy ratio and the power values of each frequency band energy ratio, after training is completed, the model output data of the target speech enhancement model may be the power value of the frequency band energy ratio corresponding to the target enhanced speech data and the target single-channel speech data.
[0130] In this embodiment, when multi-channel speech data is collected, the target speech enhancement model can perform speech enhancement based on a single channel data of the multi-channel speech data, without processing the speech data of each channel separately. This can reduce the amount and complexity of calculations while ensuring the speech enhancement effect, thereby improving the processing speed of speech enhancement and improving overall real-time performance.
[0131] In one embodiment, feature extraction is performed on target single-channel speech data to obtain target speech feature data, including:
[0132] Step D1: performing frame processing on the target single-channel speech data and obtaining multiple frames of target speech segments;
[0133] Step D3: for each frame of the target speech segment, the target speech spectrum of the frame of the target speech segment is calculated, and the target speech spectrum is compressed into N frequency bands by using a preset filtering algorithm, and the frequency band energy value, first-order difference data and second-order difference data of the target speech spectrum in each frequency band are calculated respectively;
[0134] Step D5: For each frame of target speech segment, calculate the total energy value of the frame of target speech segment, and use the total energy value of the frame of target speech segment, the band energy value of the frame of target speech segment in each frequency band, the first-order difference data and the second-order difference data as the speech feature data of the frame of target speech segment; wherein the target speech feature data includes the speech feature data of each frame of target speech segment.
[0135] In this embodiment, in the process of extracting the target speech feature data, after obtaining the target single-channel speech data, the target single-channel speech data may be preprocessed to divide the target single-channel speech data into multiple short-time frames and obtain multiple target speech segments. It can be understood that the specific operation mode of the preprocessing operation can be determined according to the actual situation, and this document does not make specific restrictions on this, as long as it at least includes framing processing. Exemplarily, in addition to framing processing, the preprocessing operation may also include windowing processing, etc.
[0136] For each frame of target speech segment, the present application can calculate the speech feature data of the target speech segment of the frame according to the process shown in step D3 and step D5, and use the speech feature data of all target speech segments as target speech feature data.
[0137] Specifically, in the process of calculating the speech feature data of a single-frame target speech segment, the present application may first perform a time-frequency transform on the frame target speech segment (for example, a Fourier transform may be performed), and obtain the spectrum corresponding to the frame target speech segment, that is, the target speech spectrum, and calculate the total energy value of the frame target speech segment. In addition, the present application may also use a preset filtering algorithm to compress the spectrum corresponding to the frame target speech segment to N frequency bands, and respectively calculate the frequency band energy value, first-order difference data, and second-order difference data of the spectrum corresponding to the frame target speech segment in each frequency band. The speech feature data of the frame target speech segment includes the total energy of the frame target speech segment, N frequency band energy values, each first-order difference data, and each second-order difference data.
[0138] It can be understood that the relevant calculation process of the total energy value, frequency band energy value, first-order difference data and second-order difference data can refer to the relevant description of the speech feature data of the noisy speech segment in the above embodiment, and will not be repeated here.
[0139] In this way, the input feature dimensions of the model can be compressed, thereby further improving the real-time reasoning of the speech enhancement model while ensuring the noise reduction and enhancement effect.
[0140] In one embodiment, the model output data is a target mask value, that is, a target mask value. When the network structure of the target speech enhancement model is as follows Figure 3 As shown in the figure, when the activation function of the second fully connected layer is the sigmoid function and the activation function of the third fully connected layer is the relu function, the target mask value has a value range of [0,1]. Since the scale transformation value scale has a value range of [0, max_scale] and max_scale is generally greater than 1, it is necessary to transform the target mask value and perform speech enhancement based on the target scale transformation value obtained by the transformation. The scale transformation value scale is the power value of the frequency band energy ratio.
[0141] Specifically, the target enhanced speech data is obtained according to the model output data, including:
[0142] According to the target mask value and the preset scale mapping relationship, the target scale transformation value is obtained;
[0143] The target enhanced speech data is obtained based on the target scale transformation value.
[0144] In this embodiment, the target mask value can be mapped to a target scale transformation value according to a preset scale mapping relationship, and the target scale transformation value can be applied to the frequency band to obtain target enhanced speech data, thereby further improving the speech enhancement effect.
[0145] In one example, if Figure 5 As shown, obtaining target enhanced speech data based on the target scale transformation value may include the following steps:
[0146] S502: Calculate the amplitude corresponding to each frequency point after the scale transformation according to the target scale transformation value and the interpolation calculation algorithm, and obtain an enhanced spectrum;
[0147] S504: performing inverse Fourier transform on the enhanced spectrum to obtain an enhanced time domain signal;
[0148] S506: Perform overlap processing on the enhanced time domain signal to obtain target enhanced speech data.
[0149] The training device for the speech enhancement model provided in an embodiment of the present application is described below. The training device for the speech enhancement model described below and the training method for the speech enhancement model described above can be referenced to each other.
[0150] In one embodiment, Figure 6 As shown, the present application provides a training device 600 for a speech enhancement model, comprising:
[0151] A first voice data acquisition module 602, used to acquire multi-channel pure voice data;
[0152] A first feature extraction module 604 is used to generate single-channel noisy speech data according to the multi-channel clean speech data and preset noise data, and perform feature extraction on the single-channel noisy speech data to obtain sample feature data;
[0153] A first speech enhancement module 606, configured to generate single-channel enhanced speech data according to the multi-channel clean speech data and a preset AGC algorithm;
[0154] A sample label generation module 608, configured to generate sample label data based on the multi-channel clean speech data, the single-channel noisy speech data and the single-channel enhanced speech data;
[0155] The model training module 610 is used to use the sample feature data as the input of the speech enhancement model to be trained, use the sample label data as the target output of the speech enhancement model to be trained, train the speech enhancement model to be trained, and obtain the target speech enhancement model after the training is completed.
[0156] In one embodiment, the sample label data includes noise reduction label data and enhanced label data; the noise reduction label data is obtained based on the single-channel noisy speech data and the multi-channel clean speech data, and the enhanced label data is obtained based on the single-channel noisy speech data and the single-channel enhanced speech data.
[0157] The model training module 610 of the present application includes:
[0158] A training data acquisition unit, used to input the sample feature data into the speech enhancement model to be trained in the current training round, and obtain the noise reduction training data generated by the noise reduction module of the speech enhancement model to be trained, and the enhanced training data generated by the enhancement module of the speech enhancement model to be trained;
[0159] A loss calculation unit, configured to calculate a total loss value of a current training round based on the denoised training data, the enhanced training data, the denoised label data and the enhanced label data;
[0160] The training end judgment unit is used to enter the next training round if it is determined according to the total loss value that the preset training end condition is not met.
[0161] In one embodiment, the noise reduction module includes a first fully connected layer and a first gated recurrent layer, and the enhancement module includes a second gated recurrent layer and a third gated recurrent layer.
[0162] The training data acquisition unit of the present application includes:
[0163] A first fully connected unit, used for inputting the sample feature data into the first fully connected layer, and obtaining first fully connected result data output by the first fully connected layer;
[0164] A first gated recurrent unit, configured to input the first fully-connected result data into the first gated recurrent layer, and obtain first gated result data output by the first gated recurrent layer;
[0165] A first splicing unit, used for splicing the sample feature data, the first fully connected result data and the first gated result data to obtain first spliced data;
[0166] A second gated circulation unit, used for inputting the first spliced data into the second gated circulation layer, and obtaining second gated result data output by the second gated circulation layer;
[0167] A second splicing unit, configured to splice the sample feature data, the first gating result data and the second gating result data to obtain second spliced data;
[0168] A third gated circulation unit, used for inputting the second spliced data into the third gated circulation layer, and obtaining third gated result data output by the third gated circulation layer;
[0169] A training data generating unit is used to generate the noise reduction training data according to the first gating result data, and to generate the enhanced training data according to the third gating result data.
[0170] In one embodiment, the noise reduction module further includes a second fully connected layer, and the enhancement module further includes a third fully connected layer.
[0171] The training data generating unit of the present application comprises:
[0172] a second fully connected unit, configured to input the first gating result into the second fully connected layer, and obtain the denoising training data generated by the second fully connected layer;
[0173] The third fully connected unit is used to input the third gating result data into the third fully connected layer and obtain the enhanced training data generated by the third fully connected layer.
[0174] In one embodiment, the first feature extraction module 604 of the present application includes:
[0175] A first framing unit, configured to perform framing processing on the single-channel noisy speech data and obtain multiple frames of noisy speech segments;
[0176] The first frame feature generating unit is used to obtain, for each frame of the noisy speech segment, a noisy spectrum corresponding to the noisy speech segment in the frame, and use a preset filtering algorithm to compress the noisy spectrum into N frequency bands, respectively calculate the frequency band energy value corresponding to each of the frequency bands, and generate speech feature data of the noisy speech segment in the frame according to the respective frequency band energy values corresponding to the noisy speech segment in the frame; wherein N is a positive integer; wherein the sample feature data includes the speech feature data of the noisy speech segment in each frame.
[0177] In one embodiment, the first frame feature generation unit of the present application includes:
[0178] The first frequency band energy calculation unit is used to calculate the total energy value of the noisy speech segment in each frame, and calculate the first-order difference data and the second-order difference data corresponding to each frequency band according to each frequency band energy value corresponding to the noisy speech segment in the frame, and use the total energy value of the noisy speech segment in the frame, the energy values of each frequency band corresponding to the noisy speech segment in the frame, each first-order difference data and each second-order difference data as the speech feature data of the noisy speech segment in the frame.
[0179] In one embodiment, the sample label data includes denoised label data and enhanced label data.
[0180] The sample label generation module 608 of the present application includes:
[0181] A second framing unit, configured to perform framing processing on the multi-channel clean voice data and obtain multiple frames of clean voice segments;
[0182] A third framing unit, configured to perform framing processing on the single-channel enhanced speech data and obtain multiple frames of enhanced speech segments;
[0183] A first ratio power calculation unit is used to calculate, for each frame of the clean speech segment, the first frequency band energy ratio power value of the clean speech segment and the corresponding noisy speech segment in each frequency band; wherein the noise reduction label data includes each first frequency band energy ratio power value corresponding to the clean speech segment of each frame;
[0184] The second ratio power calculation unit is used to calculate the second frequency band energy ratio power value of the enhanced speech segment in each frame and the corresponding noisy speech segment in each frequency band; wherein the enhanced label data includes the respective second frequency band energy ratio power values corresponding to the enhanced speech segment in each frame.
[0185] The speech enhancement device provided in the embodiment of the present application is described below. The speech enhancement device described below and the speech enhancement method described above can be referenced to each other.
[0186] In one embodiment, Figure 7 As shown, the present application provides a speech enhancement device 700, comprising:
[0187] The second voice data acquisition module 702 is used to acquire target multi-channel voice data;
[0188] A channel separation module 704 is used to perform channel separation on the target multi-channel voice data to obtain target single-channel voice data;
[0189] A second feature extraction module 706 is used to extract features from the target single-channel speech data to obtain target speech feature data;
[0190] A model processing module 708, used for inputting the target speech feature data into a target speech enhancement model, and obtaining model output data output by the target speech enhancement model; wherein the target speech enhancement model is trained based on the training method of the speech enhancement model described in any of the above embodiments;
[0191] The second speech enhancement module 710 is used to obtain target enhanced speech data according to the model output data.
[0192] In one embodiment, the second feature extraction module 706 of the present application includes:
[0193] A fourth framing unit, configured to perform framing processing on the target single-channel voice data and obtain multiple frames of target voice segments;
[0194] A second frequency band energy calculation unit is used to calculate the target speech spectrum of the target speech segment of each frame, and compress the target speech spectrum into N frequency bands by using a preset filtering algorithm, and respectively calculate the frequency band energy value, first-order difference data and second-order difference data of the target speech spectrum in each frequency band;
[0195] The second frame feature generating unit is used to calculate the total energy value of the target speech segment in each frame, and use the total energy value of the target speech segment in the frame, the band energy value of the target speech segment in the frame in each frequency band, the first-order difference data and the second-order difference data as the speech feature data of the target speech segment in the frame; wherein the target speech feature data includes the speech feature data of the target speech segment in each frame.
[0196] In one embodiment, the model output data is a target mask value. The second speech enhancement module 710 of the present application includes:
[0197] A mapping unit, configured to obtain a target scale transformation value according to the target mask value and a preset scale mapping relationship;
[0198] The speech enhancement unit is used to obtain the target enhanced speech data based on the target scale transformation value.
[0199] In one embodiment, the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the training method of the speech enhancement model described in any of the above embodiments, and / or the steps of the speech enhancement method described in any of the above embodiments.
[0200] In one embodiment, the present application also provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the training method of the speech enhancement model described in any of the above embodiments, and / or the steps of the speech enhancement method described in any of the above embodiments.
[0201] Indicatively, Figure 8 The internal structure diagram of a computer device provided in an embodiment of the present application is shown in FIG. 1 . In one example, the computer device may be a server. Figure 8 The computer device 900 includes a processing component 902, which further includes one or more processors, and a memory resource represented by a memory 901 for storing instructions executable by the processing component 902, such as an application. The application stored in the memory 901 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 902 is configured to execute instructions to perform the steps of the training method of the speech enhancement model described in any of the above embodiments and / or the steps of the speech enhancement method described in any of the above embodiments.
[0202] The computer device 900 may further include a power supply component 903 configured to perform power management of the computer device 900, a wired or wireless network interface 904 configured to connect the computer device 900 to a network, and an input / output (I / O) interface 905. The computer device 900 may operate based on an operating system stored in the memory 901, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.
[0203] Those skilled in the art will understand that the internal structure of the computer device shown in the present application is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0204] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. Herein, "one", "one", "said", "the" and "it" may also include plural forms, unless the context clearly indicates another way. A plurality refers to at least two cases, such as 2, 3, 5 or 8, etc. "And / or" includes any and all combinations of the relevant listed items.
[0205] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.
[0206] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A training method for a speech enhancement model, It is characterized in that include: Obtain multi-channel pure voice data; Generate single-channel noisy speech data according to the multi-channel clean speech data and preset noise data, and perform feature extraction on the single-channel noisy speech data to obtain sample feature data; Generate single-channel enhanced voice data according to the multi-channel clean voice data and a preset AGC algorithm; Generate sample label data based on the multi-channel clean speech data, the single-channel noisy speech data and the single-channel enhanced speech data; The sample feature data is used as the input of the speech enhancement model to be trained, and the sample label data is used as the target output of the speech enhancement model to be trained. The speech enhancement model to be trained is trained, and a target speech enhancement model is obtained after the training is completed.
2. The method according to claim 1, It is characterized in that The sample label data includes noise reduction label data and enhanced label data; the noise reduction label data is obtained based on the single-channel noisy speech data and the multi-channel clean speech data, and the enhanced label data is obtained based on the single-channel noisy speech data and the single-channel enhanced speech data; The method of using the sample feature data as the input of the speech enhancement model to be trained, using the sample label data as the target output of the speech enhancement model to be trained, and training the speech enhancement model to be trained includes: In the current training round, the sample feature data is input into the speech enhancement model to be trained, and the noise reduction training data generated by the noise reduction module of the speech enhancement model to be trained and the enhanced training data generated by the enhancement module of the speech enhancement model to be trained are obtained; Calculating a total loss value of a current training round based on the denoised training data, the enhanced training data, the denoised label data, and the enhanced label data; If it is determined according to the total loss value that the preset training end condition is not met, the next training round is entered.
3. The method according to claim 2, It is characterized in that The denoising module includes a first fully connected layer and a first gated recurrent layer, and the enhancement module includes a second gated recurrent layer and a third gated recurrent layer; The step of inputting the sample feature data into the speech enhancement model to be trained, and obtaining the noise reduction training data generated by the noise reduction module of the speech enhancement model to be trained, and the enhanced training data generated by the enhancement module of the speech enhancement model to be trained, comprises: Inputting the sample feature data into the first fully connected layer, and obtaining first fully connected result data output by the first fully connected layer; Inputting the first fully connected result data into the first gated recurrent layer, and obtaining first gated result data output by the first gated recurrent layer; splicing the sample feature data, the first fully connected result data, and the first gated result data to obtain first spliced data; Inputting the first spliced data into the second gated recurrent layer, and obtaining second gated result data output by the second gated recurrent layer; splicing the sample characteristic data, the first gating result data and the second gating result data to obtain second spliced data; Inputting the second spliced data into the third gated circulation layer, and obtaining third gated result data output by the third gated circulation layer; The noise reduction training data is generated according to the first gating result data, and the enhanced training data is generated according to the third gating result data.
4. The method according to claim 3, It is characterized in that The noise reduction module further includes a second fully connected layer, and the enhancement module further includes a third fully connected layer; The generating the noise reduction training data according to the first gating result data, and the generating the enhanced training data according to the third gating result data, comprises: Inputting the first gating result into the second fully connected layer, and obtaining the denoising training data generated by the second fully connected layer; The third gating result data is input into the third fully connected layer, and the enhanced training data generated by the third fully connected layer is obtained.
5. The method according to claim 1, It is characterized in that The step of extracting features from the single-channel noisy speech data to obtain sample feature data includes: Performing frame processing on the single-channel noisy speech data to obtain multiple frames of noisy speech segments; For each frame of the noisy speech segment, a noisy spectrum corresponding to the noisy speech segment in the frame is obtained, and a preset filtering algorithm is used to compress the noisy spectrum into N frequency bands, and the frequency band energy value corresponding to each of the frequency bands is calculated respectively, and according to the energy values of each frequency band corresponding to the noisy speech segment in the frame, speech feature data of the noisy speech segment in the frame is generated; wherein N is a positive integer; The sample feature data includes speech feature data of each frame of the noisy speech segment.
6. The method according to claim 5, It is characterized in that For each frame of the noisy speech segment, generating speech feature data of the noisy speech segment in the frame according to energy values of each frequency band corresponding to the noisy speech segment in the frame, including: For each frame of the noisy speech segment, the total energy value of the noisy speech segment in the frame is calculated, and according to the energy value of each frequency band corresponding to the noisy speech segment in the frame, the first-order difference data and the second-order difference data corresponding to each frequency band are respectively calculated, and the total energy value of the noisy speech segment in the frame, the energy values of each frequency band corresponding to the noisy speech segment in the frame, each first-order difference data and each second-order difference data are used as the speech feature data of the noisy speech segment in the frame.
7. The method according to claim 5 or 6, It is characterized in that The sample label data includes noise reduction label data and enhanced label data; The generating sample label data based on the multi-channel clean speech data, the single-channel noisy speech data and the single-channel enhanced speech data includes: Performing frame processing on the multi-channel clean speech data to obtain multiple frames of clean speech segments; Performing frame processing on the single-channel enhanced speech data to obtain multiple frames of enhanced speech segments; For each frame of the clean speech segment, respectively calculate the first frequency band energy ratio power value of the clean speech segment and the corresponding noisy speech segment in each frequency band; wherein the denoising label data includes each first frequency band energy ratio power value corresponding to the clean speech segment of each frame; For the enhanced speech segment of each frame, the power value of the second-frequency band energy ratio between the enhanced speech segment of the frame and the corresponding noisy speech segment in each frequency band is calculated respectively; wherein the enhanced label data includes the power values of the second-frequency band energy ratio corresponding to the enhanced speech segment of each frame.
8. A method for speech enhancement, It is characterized in that include: Obtain target multi-channel voice data; Performing channel separation on the target multi-channel voice data to obtain target single-channel voice data; Extracting features from the target single-channel speech data to obtain target speech feature data; Inputting the target speech feature data into a target speech enhancement model, and obtaining model output data output by the target speech enhancement model; wherein the target speech enhancement model is trained based on the training method of the speech enhancement model according to any one of claims 1 to 7; Target enhanced speech data is obtained according to the model output data.
9. The method according to claim 8, It is characterized in that The step of extracting features from the target single-channel speech data to obtain target speech feature data includes: Performing frame processing on the target single-channel voice data and obtaining multiple frames of target voice segments; For each frame of the target speech segment, a target speech spectrum of the target speech segment of the frame is calculated, and a preset filtering algorithm is used to compress the target speech spectrum into N frequency bands, and the frequency band energy value, first-order difference data and second-order difference data of the target speech spectrum in each frequency band are calculated respectively; For each frame of the target speech segment, the total energy value of the target speech segment in the frame is calculated, and the total energy value of the target speech segment in the frame, the frequency band energy value of the target speech segment in the frame in each frequency band, the first-order difference data and the second-order difference data are used as speech feature data of the target speech segment in the frame; The target speech feature data includes speech feature data of the target speech segment in each frame.
10. The method according to claim 8 or 9, It is characterized in that The model output data is a target mask value; The step of obtaining target enhanced speech data according to the model output data comprises: Obtaining a target scale transformation value according to the target mask value and a preset scale mapping relationship; The target enhanced speech data is obtained based on the target scale transformation value.
11. A training device for a speech enhancement model, It is characterized in that include: A first voice data acquisition module, used to acquire multi-channel pure voice data; A first feature extraction module, used to generate single-channel noisy speech data according to the multi-channel clean speech data and preset noise data, and perform feature extraction on the single-channel noisy speech data to obtain sample feature data; A first speech enhancement module, used for generating single-channel enhanced speech data according to the multi-channel clean speech data and a preset AGC algorithm; A sample label generation module, used to generate sample label data based on the multi-channel clean speech data, the single-channel noisy speech data and the single-channel enhanced speech data; The model training module is used to use the sample feature data as the input of the speech enhancement model to be trained, use the sample label data as the target output of the speech enhancement model to be trained, train the speech enhancement model to be trained, and obtain the target speech enhancement model after the training is completed.
12. A speech enhancement device, It is characterized in that include: A second voice data acquisition module, used to acquire target multi-channel voice data; A channel separation module is used to perform channel separation on the target multi-channel voice data to obtain target single-channel voice data; A second feature extraction module is used to extract features from the target single-channel speech data to obtain target speech feature data; A model processing module, used for inputting the target speech feature data into a target speech enhancement model and obtaining model output data output by the target speech enhancement model; wherein the target speech enhancement model is trained based on the training method of the speech enhancement model according to any one of claims 1 to 7; The second speech enhancement module is used to obtain target enhanced speech data according to the model output data.
13. A storage medium, It is characterized in that The storage medium stores computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to execute the steps of the training method for the speech enhancement model as described in any one of claims 1 to 7, and / or execute the steps of the speech enhancement method as described in any one of claims 8 to 10.
14. A computer device, It is characterized in that include: one or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, execute the steps of the training method of the speech enhancement model as described in any one of claims 1 to 7, and / or execute the steps of the speech enhancement method as described in any one of claims 8 to 10.
Citation Information
Cited By
High-fidelity audio generation method and device and storage medium
CN119673199A
