A Chinese speech enhancement method

Abstract

The present invention discloses a method for enhancing Chinese speech, comprising: obtaining a noisy speech signal, and dividing the time domain waveform of the noisy speech signal into a plurality of speech signal sequences; inputting the speech signal sequences into a speech enhancement model, obtaining enhanced speech signals corresponding to each speech signal sequence, and splicing them in sequence; wherein the speech enhancement model is: inputting a preset number of speech sequence signals into a speech enhancement model, and then <x noisy ,x clean x in > noisy After inputting the time domain speech enhancement model to be trained, based on the scale coefficient matrix K α 、x noisy The corresponding x clean And the enhanced speech signal x output by the time domain speech enhancement model to be trained enhanced The loss value of the multi-resolution loss function is calculated, and the deep learning model is obtained by adjusting the model parameters of the time domain speech enhancement model to be trained through back propagation. The present invention introduces the proportional coefficient matrix K when calculating the loss value. α , different attention can be given to frequency bands based on their importance, thereby improving the speech enhancement performance of the speech enhancement model.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech processing, and in particular relates to a Chinese speech enhancement method. Background Art

[0002] Speech enhancement tasks are often abstracted into supervised learning problems. In supervised learning, the loss function measures the degree of data dispersion, thereby calculating the loss value based on the predicted and true values ​​output by the network model. During neural network training, the network model continuously adjusts its parameters to improve the accuracy of predictions with the goal of minimizing the loss value. In speech enhancement tasks, the frequency domain loss function measures the fit of speech data in the frequency domain. The loss value is generally calculated using the amplitude spectrum features of the clean and predicted speech.

[0003] In the existing technology, there are three types of frequency domain loss functions: the first is to directly use the amplitude spectra of the clean speech and the predicted speech for calculation. Specifically, after subtracting the amplitude spectra of the two bit by bit, the Frobenius norm (F-norm) is used to measure the similarity between the two to obtain the loss value; the second idea is that the logarithmic amplitude spectrum of speech is more in line with the auditory perception characteristics of the human ear. Therefore, the error value calculated using the logarithmic amplitude spectrum of the clean speech and the predicted speech can enable the model to obtain better speech processing effect after training; the third idea starts from calculating the short-time Fourier transform (STFT) of the speech amplitude spectrum. By setting different parameters in the STFT calculation, such as the number of Fourier points, window length, frame shift, etc., speech amplitude spectra at different resolutions are generated, and then the loss values ​​of the amplitude maps at different resolutions are calculated separately and summed to obtain the final loss value.

[0004] In the three frequency-domain loss function implementations described above, the same calculation method is used for all frequency bands in the amplitude spectrum when calculating the fit of speech data in the frequency domain. This gives each frequency band equal importance in the loss calculation. This approach can limit the model's ability to perceive information in important frequency bands, hindering performance. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a method for enhancing Chinese speech. The technical problem to be solved by the present invention is achieved by the following technical solutions:

[0006] The present invention provides a Chinese speech enhancement method, comprising:

[0007] Acquire a noisy speech signal, and divide the time domain waveform of the noisy speech signal into multiple speech signal sequences;

[0008] The speech signal sequence is input into the speech enhancement model to obtain the enhanced speech signal corresponding to each speech signal sequence and splice them in sequence; wherein the speech enhancement model is: <x noisy , x clean > Noisy speech x noisy After inputting the time domain speech enhancement model to be trained, based on the scale coefficient matrix Kα, x noisy Corresponding pure voice x clean And the enhanced speech signal x output by the time domain speech enhancement model to be trained enhanced Calculating a loss value of a multi-resolution loss function and adjusting model parameters of the time-domain speech enhancement model to be trained by backpropagation to obtain a deep learning model;

[0009] Among them, the proportional coefficient matrix K α Calculate it as follows:

[0010] Obtain all clean speech data x in the Chinese corpus and perform short-time Fourier transform according to P groups of parameters to obtain P amplitude spectra |STFT(x)|;

[0011] For each amplitude spectrum |STFT(x)|, calculate the mean parameter M(f) of the Chinese speech amplitude spectrum, where f is the element index in M(f);

[0012] P proportional coefficient matrices K are calculated based on the Chinese speech amplitude spectrum mean parameter M(f) α .

[0013] In one embodiment of the present invention, the dimension of the amplitude spectrum |STFT(x)| is F spec ×T spec , where F spec is the number of frequency bins in the amplitude spectrum |STFT(x)|, T spec is the number of time frames in the amplitude spectrum |STFT(x)|;

[0014] The step of calculating the mean parameter M(f) of the Chinese speech amplitude spectrum for each amplitude spectrum |STFT(x)| comprises:

[0015] For each amplitude spectrum |STFT(x)|, at T spec The mean value of the amplitude spectrum |STFT(x)| is calculated in the dimension to obtain the mean parameter M(f) of the amplitude spectrum of Chinese speech:

[0016]

[0017] Where t represents the summation variable.

[0018] In one embodiment of the present invention, P proportional coefficient matrices K are calculated based on the Chinese speech amplitude spectrum mean parameter M(f) α The steps include:

[0019] The Chinese speech amplitude spectrum mean parameter M(f) is subjected to minimum and maximum normalization processing respectively to obtain P normalized amplitude spectrum mean sequences M norm (f);

[0020] The hyperparameter α∈[0,1] is used to normalize each of the amplitude spectrum mean sequences M norm The elements in (f) are remapped to obtain P sequences M α (f);

[0021] According to the sequence M α Elements in (f) Generate a 2D diagonal matrix After that, the obtained P two-dimensional diagonal matrices K α as a scaling factor matrix.

[0022] In one embodiment of the present invention, the Chinese speech amplitude spectrum mean parameter M(f) is subjected to minimum and maximum normalization processing according to the following formula:

[0023]

[0024] In the formula, min(M(f)) represents the minimum value of the mean parameter M(f) of the amplitude spectrum of Chinese speech, max(M(f)) represents the maximum value of the mean parameter M(f) of the amplitude spectrum of Chinese speech, and M norm (f) represents the normalized amplitude spectrum mean sequence.

[0025] In one embodiment of the present invention, the normalized amplitude spectrum mean sequence M is calculated using the hyperparameter α∈[0,1] according to the following formula: norm The elements in (f) are remapped:

[0026] M α (f) = αM norm (f)+(1-α)Ones

[0027] Among them, Ones represents the length and M norm (f) The same all-1 sequence.

[0028] In one embodiment of the present invention, the speech enhancement model is trained according to the following steps:

[0029] Reading B data pairs consisting of clean speech data and noisy speech data from the Chinese corpus, wherein each noisy speech data is obtained by additively mixing each clean speech data with noise;

[0030] Divide each data pair into multiple speech sequence signal pairs of preset length <x noisy , x clean >

[0031] The noisy speech x in the speech sequence signal pair noisy Input the time domain speech enhancement model Wave-U-Net to be trained for forward propagation to obtain the enhanced speech signal x enhanced ;

[0032] According to the proportional coefficient matrix K α , enhanced speech signal x enhanced And the noisy speech x noisy Corresponding pure voice x clean , calculate the loss value of the multi-resolution loss function;

[0033] When the loss value does not meet the preset conditions, back propagation is performed according to the loss value to adjust the model parameters of the time-domain speech enhancement model to be trained; when the loss value meets the preset conditions, the time-domain speech enhancement model to be trained is used as the trained speech enhancement model.

[0034] In one embodiment of the present invention, according to the scale coefficient matrix K α , enhanced speech signal x enhanced And the noisy speech x noisy Corresponding pure voice x clean , the step of calculating the loss value of the multi-resolution loss function comprises:

[0035] According to the P group of parameters, the noisy speech x noisy Corresponding pure voice x clean Perform short-time Fourier transform respectively to obtain P amplitude spectra |STFT(x clean )|;

[0036] Use the proportional coefficient matrix K corresponding to each set of parameters α , calculate each of the amplitude spectrum |STFT(x clean )|The enhanced amplitude spectrum of STFT(x clean )| K ;

[0037] Using each enhanced magnitude spectrum |STFT(x clean )| K Calculate the loss value of the single-resolution loss function;

[0038] The loss value of the multi-resolution loss function is calculated based on the obtained P loss values ​​of the single-resolution loss functions.

[0039] In one embodiment of the present invention, the amplitude spectrum |STFT(x clean )|The enhanced amplitude spectrum of STFT(x clean )| K :

[0040] |STFT(x)| K =K α |STFT(x clean )|.

[0041] In one embodiment of the present invention, the loss value of the single-resolution loss function is calculated according to the following formula:

[0042]

[0043] In the formula, ||·|| F and ||·||1 represent the Frobenius norm and L1 norm respectively, and N represents the enhanced amplitude spectrum |STFT(x clean )| K The number of elements in .

[0044] In one embodiment of the present invention, the loss value of the multi-resolution loss function is calculated according to the following formula:

[0045]

[0046] Where, p = 1, 2,…, P.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention provides a Chinese speech enhancement method, which adopts a speech enhancement model: a preset number of speech sequence signals are <x noisy , x clean > Noisy speech x noisy After inputting the time domain speech enhancement model to be trained, based on the scale coefficient matrix K α 、x noisy Corresponding pure voice x clean And the enhanced speech signal x output by the time domain speech enhancement model to be trained enhanced The loss value of the multi-resolution loss function is calculated, and the deep learning model is obtained by adjusting the model parameters of the time domain speech enhancement model to be trained by back propagation. The present invention introduces the proportional coefficient matrix K when calculating the loss value. α, different attention can be given to frequency bands based on the importance of each frequency band of the noisy speech signal, thereby improving the speech enhancement performance of the speech enhancement model.

[0049] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of a Chinese speech enhancement method provided by an embodiment of the present invention;

[0051] Figure 2 Schematic diagram of the structure of the speech enhancement model provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0053] Figure 1 Flowchart of the Chinese speech enhancement method provided by the embodiment of the present invention. Figure 1 As shown, the embodiment of the present invention provides a Chinese speech enhancement method, comprising:

[0054] S1. Acquire a noisy speech signal and divide the time domain waveform of the noisy speech signal into multiple speech signal sequences;

[0055] S2. Input the speech signal sequence into the speech enhancement model, obtain the enhanced speech signal corresponding to each speech signal sequence and splice them in sequence; wherein the speech enhancement model is: input a preset number of speech sequence signals into the speech enhancement model, obtain the enhanced speech signal corresponding to each speech signal sequence and splice them in sequence; <x noisy , x clean > Noisy speech x noisy After inputting the time domain speech enhancement model to be trained, based on the scale coefficient matrix K α 、x noisy Corresponding pure voice x clean And the enhanced speech signal x output by the time domain speech enhancement model to be trained enhanced A deep learning model is obtained by calculating the loss value of the multi-resolution loss function and adjusting the model parameters of the time-domain speech enhancement model to be trained through backpropagation;

[0056] Among them, the above-mentioned proportional coefficient matrix K α Calculate it as follows:

[0057] Obtain all clean speech data x in the Chinese corpus and perform short-time Fourier transform according to P groups of parameters to obtain P amplitude spectra |STFT(x)|;

[0058] For each amplitude spectrum |STFT(x)|, calculate the mean parameter M(f) of the Chinese speech amplitude spectrum, where f is the element index in M(f);

[0059] P proportional coefficient matrices K are calculated based on the mean parameter M(f) of the Chinese speech amplitude spectrum α .

[0060] Figure 2 Schematic diagram of the structure of the speech enhancement model provided by the embodiment of the present invention. Figure 2 As shown, the model used for Chinese speech enhancement in the present invention is an end-to-end time-domain speech enhancement model Wave-U-Net, whose input and output are both speech time-domain waveforms of length Length. The time-domain speech enhancement model consists of an encoder, an intermediate layer, a decoder, and an output layer, and has a U-shaped mesh structure; specifically, the encoder includes L encoding layers, each of which consists of a one-dimensional convolutional layer, a BN layer, a LeakRelu activation function, and a downsampling layer. After the features are input into the encoding layer, the encoding layer will extract higher-level features based on the input features, and the intermediate layer is a one-dimensional convolutional layer. The decoder includes L decoding layers, each of which consists of a one-dimensional convolutional layer, a BN layer, a LeakRelu activation function, and an upsampling layer. The input of each decoding layer comes from two parts, one part is the output features of the previous layer, and the other part is the output features of the one-dimensional convolutional layer in the decoding layer at a symmetrical position. These two parts of features are spliced ​​in the channel dimension and then input into the decoding layer. The output layer is a one-dimensional convolutional layer, and the Tanh activation function can be used to restore the features into waveform sequence information. The Tanh activation function is then used to perform nonlinear mapping on them and limit the amplitude to the range of (O, 1) to produce an enhanced speech signal.

[0061] In the above step S2, after the time domain waveform of the noisy speech signal is divided into multiple speech signal sequences, the multiple speech signal sequences are sequentially input into the speech enhancement model for forward propagation to obtain multiple enhanced speech signals. The enhanced speech signal sequences are spliced ​​in the order of input to obtain the speech signal after the noisy speech signal is enhanced.

[0062] It should be understood that the Chinese corpus contains a large amount of Chinese speech and audio data with diverse content. Analyzing and computing this audio data in the corpus can yield parameters characterizing the vocal characteristics of Chinese speech. In this embodiment, a short-time Fourier transform is used to extract the mean amplitude spectrum parameters of all clean speech data x in the Chinese corpus to characterize the vocal characteristics of Chinese speech.

[0063] Specifically, when calculating the coefficient matrix K αWhen , all the pure speech data x without noise in the Chinese corpus are first subjected to short-time Fourier transform according to P groups of parameters, and P amplitude spectra |STFT(x)| are obtained. The dimension of each amplitude spectrum |STFT(x)| is F spec ×T spec , where T spec Indicates the number of time frames in the amplitude spectrum |STFT(x)|. The more speech points there are, the more time frames there are in the corresponding amplitude spectrum. spec Indicates the number of frequency bins in the amplitude spectrum |STFT(x)|, F spec The value of is determined by the window length winLength parameter in the short-time Fourier transform process. The specific calculation formula is:

[0064]

[0065] The longer the window length, the more frequency bins there are in the amplitude spectrum.

[0066] Furthermore, the ROF (Range Of Frequencies) contained in the amplitude spectrum is determined by the sampling rate of the speech, specifically:

[0067]

[0068] It should be noted that the first element of each row in the amplitude spectrum |STFT(x)| represents the magnitude of the DC component of the original speech, and the elements in the remaining positions represent the ROF divided into F spec -1, the amplitude of the original speech in the corresponding frequency band.

[0069] In this embodiment, the step of calculating the mean parameter M(f) of the Chinese speech amplitude spectrum for each amplitude spectrum |STFT(x)| includes:

[0070] For each amplitude spectrum |STFT(x)|, at T spec Calculate the mean of the amplitude spectrum |STFT(x)| in the dimension to obtain the mean parameter M(f) of the amplitude spectrum of Chinese speech:

[0071]

[0072] Where t represents the summation variable. M(f) is a summation variable with a length of F. spec The position of the elements in M(f) corresponds to the frequency bands of each row in the amplitude spectrum |STFT(x)|, and the values ​​of the elements in M(f) represent the average amplitude of the corresponding frequency band, which can characterize the energy distribution of Chinese speech in each frequency band.

[0073] Optionally, P proportional coefficient matrices K are calculated based on the mean parameter M(f) of the Chinese speech amplitude spectrum. α The steps include:

[0074] Perform minimum and maximum normalization on the amplitude spectrum mean parameter M(f) of Chinese speech respectively, and obtain P normalized amplitude spectrum mean sequences M norm (f);

[0075] Use the hyperparameter α∈[0,1] to normalize each amplitude spectrum mean sequence M norm The elements in (f) are remapped to obtain P sequences M α (f);

[0076] According to the sequence M α Elements in (f) Generate a 2D diagonal matrix After that, the obtained P two-dimensional diagonal matrices K α as a scaling factor matrix.

[0077] Specifically, the minimum and maximum normalization processing is first performed on the Chinese speech amplitude spectrum mean parameter M(f) according to the following formula:

[0078]

[0079] In the formula, min(M(f)) represents the minimum value of the mean parameter M(f) of the amplitude spectrum of Chinese speech, max(M(f)) represents the maximum value of the mean parameter M(f) of the amplitude spectrum of Chinese speech, and M norm (f) represents the normalized amplitude spectrum mean sequence.

[0080] After normalization, M norm The size of the elements in (f) is mapped in the interval [0, 1]. The larger the value of the element, the more likely it is that Chinese speech will have frequency components with larger energy in the corresponding frequency band, that is, the richer the speech information contained in the frequency band.

[0081] Generally speaking, the main characteristic frequencies of Chinese speech are relatively concentrated. norm In (f), the values ​​of the elements representing the frequency band below 1000 Hz are large, and the values ​​of the elements representing other frequency bands are close to zero. norm (f) Calculating the loss value will result in missing information in the loss function at higher frequency bands. Therefore, in order to give different attention to frequency bands according to the difference in importance between frequency bands, this embodiment introduces a hyperparameter α∈[0,1] to M normThe elements in (f) are remapped so that the values ​​of smaller elements are increased, so that relatively unimportant frequency bands can also be given some attention when calculating the loss value. For example, the normalized amplitude spectrum mean sequence M is calculated using the hyperparameter α∈[0,1] according to the following formula: norm The elements in (f) are remapped:

[0082] M α (f) = αM norm (f)+(1-α)Ones

[0083] Among them, Ones represents the length and M norm (f) The same all-1 sequence, M α The values ​​of the elements in (f) are distributed in the interval [1-α, 1].

[0084] Furthermore, M α (f) The elements in the sequence are recorded as The two-dimensional diagonal matrix As a scaling coefficient matrix, its dimension is F spec ×F spec .

[0085] In this embodiment, the speech enhancement model is trained according to the following steps:

[0086] Read B data pairs consisting of clean speech data and noisy speech data from the Chinese corpus, where each noisy speech data is obtained by additively mixing each clean speech data with noise;

[0087] Divide each data pair into multiple speech sequence signal pairs of preset length <x noisy , x clean >

[0088] The noisy speech x in the speech sequence signal pair noisy Input the time domain speech enhancement model Wave-U-Net to be trained for forward propagation to obtain the enhanced speech signal x enhanced ;

[0089] According to the proportional coefficient matrix K α , enhanced speech signal x enhanced and noisy speech x noisy Corresponding pure voice x clean , calculate the loss value of the multi-resolution loss function;

[0090] When the loss value does not meet the preset conditions, back propagation is performed according to the loss value to adjust the model parameters of the time-domain speech enhancement model to be trained; when the loss value meets the preset conditions, the time-domain speech enhancement model to be trained is used as the trained speech enhancement model.

[0091] During the training process, a certain number of Chinese speech samples are first randomly selected from the Chinese corpus as clean speech data for the training set. Another number of Chinese speech samples are then randomly selected as clean speech data for the test set. A certain amount of noise, such as white noise or interior noise, is then selected from the noise library as additive noise for the speech in the training set. The noise can be white noise or vehicle noise. The clean speech data in the training set are then mixed with the selected noise to create a training set containing noisy speech data with varying signal-to-noise ratios. Similarly, a certain amount of noise, such as high-frequency channel noise or pink noise, is also selected as additive noise for the speech in the test set. The clean speech data in the test set are then mixed with the noise to create a test set containing noisy speech data with varying signal-to-noise ratios. This results in a Chinese speech training set for training the deep learning model and a Chinese speech test set for testing the trained model.

[0092] The calculation formula for additive mixing of clean speech data and noise is as follows:

[0093] y(t)=x(t)+n(t)

[0094] Where y(t) represents the time domain signal of the synthesized noisy speech data at time t, x(t) represents the time domain signal of the clean speech data at time t, and n(t) is the noise signal at time t.

[0095] The training process will be carried out in multiple rounds. In each round of training, firstly, B pairs of clean speech data and noisy speech data that have not been read in this round of training are randomly read from the training set, and each data pair is divided into multiple speech signal sequence pairs of length Length. <x noisy , x clean >, then the noisy speech x in the speech signal sequence pair noisy As input, forward propagation is performed in the time domain speech enhancement model to be trained to obtain the enhanced speech signal x enhanced , then use x clean 、x enhanced And the scale coefficient matrix K α The loss value of the multi-resolution loss function is calculated, and the time-domain speech enhancement model to be trained performs backpropagation based on the loss value and adjusts the model parameters. A training round ends when all speech signal sequences in the training set are processed.

[0096] As the number of training rounds increases, the multi-resolution loss function gradually decreases. The model's performance can be tested using the test set after each fixed number of training rounds. The trained speech enhancement model is considered complete when the multi-resolution loss function no longer decreases with increasing training rounds.

[0097] During the research, the inventors discovered that the pronunciation characteristics of Chinese speech are different from those of other common languages, and this difference is reflected in the voiceless and voiced sounds. Speech is divided into voiceless and voiced sounds according to whether the vocal cords vibrate during pronunciation. Sounds in which the vocal cords vibrate during pronunciation are voiced sounds, and their waveform amplitude in the time domain is large and has a short periodicity. There are resonance peaks in the frequency domain, and the energy is mainly distributed in the low-frequency region. Sounds in which the vocal cords do not vibrate during pronunciation are voiceless sounds, and their overall amplitude in the time domain is smaller than that of voiced sounds, and they have no significant characteristics in the frequency domain. Different from languages ​​such as English, French, German, Japanese, and Russian that contain a large number of voiced sounds, only the four initials m, n, l, and r in the Chinese phonetic spelling are voiced sounds, and the rest of the initials and finals are voiceless sounds, which makes the frequency distribution of Chinese speech more concentrated, and the semantic information in Chinese speech appears frequently in certain frequency bands and less frequently in other frequency bands.

[0098] Based on the above analysis, this embodiment combines the pronunciation characteristics of Chinese speech with the loss function to improve the performance of the loss function in the Chinese speech enhancement task.

[0099] Optionally, according to the scale factor matrix K α , enhanced speech signal x enhanced and noisy speech x noisy Corresponding pure voice x clean , the steps for calculating the loss value of the multi-resolution loss function include:

[0100] According to the P group parameters, the noisy speech x noisy Corresponding pure voice x clean Perform short-time Fourier transform respectively to obtain P amplitude spectra |STFT(x clean )|;

[0101] Use the proportional coefficient matrix K corresponding to each set of parameters α , calculate each amplitude spectrum|STFT(x clean )|The enhanced amplitude spectrum of STFT(x clean )| K ;

[0102] Using each enhanced magnitude spectrum |STFT(x clean )| K Calculate the loss value of the single-resolution loss function;

[0103] The loss value of the multi-resolution loss function is calculated based on the loss values ​​of the P single-resolution loss functions obtained.

[0104] Specifically, the noisy speech x noisy Corresponding pure voice x clean The amplitude spectrum|STFT(x clean)|, and then the proportional coefficient matrix K under each set of parameters α With the magnitude spectrum | STFT(x clean )|Multiply: |STFT(x)| K =K α |STFT(x clean )|, so that the amplitude spectrum|STFT(x clean )| is assigned different proportional coefficients according to the importance of the frequency interval, and the P enhanced amplitude spectra |STFT(x)| are obtained. K Dimension and magnitude spectrum of STFT(x clean )| have the same dimensions.

[0105] For each enhanced magnitude spectrum |STFT(x)| K , the loss value of the single-resolution loss function can be calculated according to the following formula:

[0106]

[0107] In the formula, ||·|| F and ||·||1 represent the Frobenius norm and L1 norm respectively, and N represents the enhanced amplitude spectrum |STFT(x clean )| K The number of elements in .

[0108] Finally, the loss value of the multi-resolution loss function is calculated according to the following formula:

[0109]

[0110] In the formula, M represents the different parameters The number of , m = 1, 2,…, M.

[0111] Below, the Chinese speech enhancement method provided by the present invention is further described using the dataset Primewords Chinese Corpus Set 1 as a Chinese corpus.

[0112] The Chinese corpus used in this example comes from the Primewords Chinese Corpus Set 1 dataset. The Chinese speech data contained therein are all PCM-encoded and stored in WAV format at a sampling rate of 16000 Hz. Short-time Fourier transform (STFT) operations were performed on all clean speech data in the corpus, with window lengths of 1024, 2048, and 512, and frame shifts of 120, 240, and 50. The resulting amplitude spectra |STFT(x)| at three different resolutions were obtained. The frequency dimension lengths of each amplitude spectrum |STFT(x)| were 513, 1025, and 257, respectively. The first position in the frequency dimension represents the magnitude of the DC component in each speech frame, while the remaining positions represent the corresponding frequency intervals after equally dividing the 0-8000 Hz frequency range into 512, 1024, and 2048 portions. The time frame dimension length of the amplitude spectrum |STFT(x)| is the number of speech frames after sliding windowing by the window function, and its value increases with the duration of the processed speech.

[0113] For each amplitude spectrum |STFT(x)|, it is accumulated in the time frame dimension and divided by the total number of frames to obtain the amplitude spectrum mean parameter M(f), which is a one-dimensional sequence with lengths of 513, 1025, and 257. Then, M(f) is normalized to the minimum and maximum values, and the smaller values ​​in the amplitude spectrum mean parameter are amplified by the hyperparameter to obtain M α (f), where the hyperparameter α = 0.25, using M α (f) Generate the diagonal matrix K α＝0.25 Will.

[0114] 10,000 Chinese speech samples were randomly selected from the Primewords Chinese Corpus Set 1 dataset as the clean speech data for the training set. Another 800 different Chinese speech samples were randomly selected from the same dataset as the clean speech data for the test set. The noise set used was from the NoiseX-92 noise set, with each noise sampled to 16,000 Hz. Ten noise types from this set were selected as additive noise for the training set: white noise, car interior noise, military vehicle noise, tank interior noise, restaurant noise, machine gun noise, factory floor noise 1, destroyer aircraft noise, destroyer operations room background noise, and cockpit noise 1. The 10,000 clean signals from the training set were mixed with these 10 noise types to generate 10,000 noisy speech samples with signal-to-noise ratios of 0, 5, 10, and 15 dB. The specific mixing method is to sequentially select a noise type, then combine it with four clean speech samples to generate four noisy speech samples with different signal-to-noise ratios. Then, the next noise type is selected and mixed with another four speech samples, repeating this process until 10,000 clean speech signals are generated. Five other noise types from the NoiseX-92 noise set are selected as additive noise for the test set: high-frequency channel noise, pink noise, Buccaneer cockpit noise 2, and factory floor noise 2. The 800 clean speech samples from the test set are mixed with these five noise types using the same method as used to generate the training set, generating 800 noisy speech samples with signal-to-noise ratios of 2.5, 7.5, 12.5, and 17.5 dB.

[0115] Please continue to see Figure 2The input and output data of the speech enhancement model used in this embodiment are speech time-domain waveforms of length 16384. Both the encoding and decoding layers have L = 12 levels. The one-dimensional convolutional layers used in the 12 encoding layers have a convolution kernel size of 15, the number of input channels is [1, 24, 48, 72, 96, 120, 144, 168, 192, 216, 240, 264], and the number of output channels is [24, 48, 72, 96, 120, 144, 168, 192, 216, 240, 264, 288]. The one-dimensional convolution layer used as the intermediate layer has a convolution kernel size of 15 and 288 input and output channels. The one-dimensional convolution layer used in the 12 decoding layers has a convolution kernel size of 5, input channels of [576, 552, 504, 456, 408, 360, 312, 264, 216, 168, 120, 72], and output channels of [288, 264, 240, 216, 192, 168, 144, 120, 96, 72, 48, 24]. The one-dimensional convolution layer used as the output layer has a convolution kernel size of 1 and input and output channels of [25, 1], respectively. The data is restored to waveform sequence information through the output layer, and then nonlinearly mapped using the Tanh activation function, resulting in an enhanced speech signal of length 16384.

[0116] The simulation environment used for model training in this implementation is as follows: an Intel Xeon Gold 6330 CPU with a main frequency of 2.0 GHz, 30 GB of RAM, an Ubuntu 16.04 operating system, the PyTorch deep learning framework, and a Python 3.9.12 development platform. Graphics card: GTX A5000. Under these environmental conditions, the model was trained using the dataset obtained in step 2. The loss function obtained in step 1 was used during training. During model training, the Adam optimization algorithm was used, with a learning rate set to 0.001 and Adam parameters set to β1 = 0.9 and β2 = 0.999. Training was repeated for 500 rounds to obtain a trained speech enhancement model.

[0117] The trained speech enhancement model is used to enhance noisy speech. First, the noisy speech is segmented into speech signal segments of length L = 16384. If the remaining speech length is less than 16384, the end is padded with zeros. These segments are then fed into the model for forward propagation to obtain enhanced speech signal segments of length L = 16384. The signal segments are then concatenated to obtain the complete enhanced speech.

[0118] To verify the effectiveness of this invention, we used several speech enhancement methods as references. These included the Wiener filter method, a traditional speech enhancement algorithm, and a deep learning-based speech enhancement method that used the same dataset and model structure but different loss functions. The model's speech enhancement performance was tested using the test set, and the objective evaluation of speech quality (PESQ) was used as the performance metric. The results of the Chinese speech enhancement performance comparison are shown in Table 1.

[0119] Table 1 Comparison results of Chinese speech enhancement performance

[0120]

[0121] In Table 1, L MSE is the mean square error loss function, L STFT With L M-STFT The proportional coefficient matrix K α The single-resolution loss function and multi-resolution loss function when the amplitude spectrum is enhanced and the loss value is calculated directly using the amplitude spectrum. As shown in Table 1, the performance of the Chinese speech enhancement method based on deep learning is better than the Wiener filter method in the traditional algorithm. Under the same model structure, the STFT loss function under single resolution combined with Chinese pronunciation characteristics is Better than directly using the loss function L of the spectrogram STFT , multi-resolution STFT loss function combined with Chinese pronunciation characteristics Better than using the multi-resolution STFT loss function L directly using the spectrogram M-STFT , which verifies the effectiveness of using speech features to improve the loss function. In the loss function, The model achieved the best speech enhancement performance, verifying the effectiveness of the loss function in the present invention.

[0122] It can be seen from the above embodiments that the beneficial effects of the present invention are:

[0123] The present invention provides a Chinese speech enhancement method, which adopts a speech enhancement model: a preset number of speech sequence signals are <x noisy , x clean > Noisy speech x noisy After inputting the time domain speech enhancement model to be trained, based on the scale coefficient matrix K α 、x noisy Corresponding pure voice x clean And the enhanced speech signal x output by the time domain speech enhancement model to be trained enhanced The loss value of the multi-resolution loss function is calculated, and the deep learning model is obtained by adjusting the model parameters of the time domain speech enhancement model to be trained by back propagation. The present invention introduces the proportional coefficient matrix K when calculating the loss value. α, different attention can be given to frequency bands based on the importance of each frequency band of the noisy speech signal, thereby improving the speech enhancement performance of the speech enhancement model.

[0124] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0125] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art can understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims.

[0126] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A Chinese speech enhancement method, characterized in that: include: Acquire a noisy speech signal, and divide the time domain waveform of the noisy speech signal into multiple speech signal sequences; The speech signal sequence is input into the speech enhancement model to obtain the enhanced speech signal corresponding to each speech signal sequence and splice them in sequence; wherein the speech enhancement model is: <x noisy , x clean > Noisy speech x noisy After inputting the time domain speech enhancement model to be trained, based on the scale coefficient matrix K α 、x noisy Corresponding pure voice x clean And the enhanced speech signal x output by the time domain speech enhancement model to be trained enhanced Calculating a loss value of a multi-resolution loss function and adjusting model parameters of the time-domain speech enhancement model to be trained by backpropagation to obtain a deep learning model; Among them, the proportional coefficient matrix K α Calculate it as follows: Obtain all clean speech data x in the Chinese corpus and perform short-time Fourier transform according to P groups of parameters to obtain P amplitude spectra |STFT(x)|; For each amplitude spectrum |STFT(x)|, calculate the mean parameter M(f) of the Chinese speech amplitude spectrum, where f is the element index in M(f); P proportional coefficient matrices K are calculated based on the Chinese speech amplitude spectrum mean parameter M(f) α .

2. The Chinese speech enhancement method according to claim 1, characterized in that: The dimension of the amplitude spectrum |STFT(x)| is F spec ×T spec , where F spec is the number of frequency bins in the amplitude spectrum |STFT(x)|, T spec is the number of time frames in the amplitude spectrum |STFT(x)|; The step of calculating the mean parameter M(f) of the Chinese speech amplitude spectrum for each amplitude spectrum |STFT(x)| comprises: For each amplitude spectrum |STFT(x)|, at Ts pec The mean value of the amplitude spectrum |STFT(x)| is calculated in the dimension to obtain the mean parameter M(f) of the amplitude spectrum of Chinese speech: Where t represents the summation variable.

3. The Chinese speech enhancement method according to claim 2, characterized in that: P proportional coefficient matrices K are calculated based on the Chinese speech amplitude spectrum mean parameter M(f) α The steps include: The Chinese speech amplitude spectrum mean parameter M(f) is subjected to minimum and maximum normalization processing respectively to obtain P normalized amplitude spectrum mean sequences M norm (f); The hyperparameter α∈[0,1] is used to normalize each of the amplitude spectrum mean sequences M norm The elements in (f) are remapped to obtain P sequences M α (f); According to the sequence M α Elements in (f) Generate a 2D diagonal matrix Finally, the obtained P two-dimensional diagonal matrices Kα are used as the proportional coefficient matrix.

4. The Chinese speech enhancement method according to claim 3, characterized in that: The minimum and maximum normalization processing is performed on the Chinese speech amplitude spectrum mean parameter M(f) according to the following formula: In the formula, min(M(f)) represents the minimum value of the mean parameter M(f) of the amplitude spectrum of Chinese speech, max(M(f)) represents the maximum value of the mean parameter M(f) of the amplitude spectrum of Chinese speech, and M norm (f) represents the normalized amplitude spectrum mean sequence.

5. The Chinese speech enhancement method according to claim 3, characterized in that: According to the following formula, the hyperparameter α∈[0,1] is used to calculate the normalized amplitude spectrum mean sequence M norm The elements in (f) are remapped: M α (f)=αM norm (f)+(1-α)Ones Among them, Ones represents the length and M norm (f) The same all-1 sequence.

6. The Chinese speech enhancement method according to claim 1, wherein: The speech enhancement model is trained according to the following steps: Reading B data pairs consisting of clean speech data and noisy speech data from the Chinese corpus, wherein each noisy speech data is obtained by additively mixing each clean speech data with noise; Divide each data pair into multiple speech sequence signal pairs of preset length <x noisy , x clean > The noisy speech x in the speech sequence signal pair noisy Input the time domain speech enhancement model Wave-U-Net to be trained for forward propagation to obtain the enhanced speech signal x enhanced ; According to the proportional coefficient matrix K α , enhanced speech signal x enhanced And the noisy speech x noisy Corresponding pure voice x clean , calculate the loss value of the multi-resolution loss function; When the loss value does not meet the preset conditions, back propagation is performed according to the loss value to adjust the model parameters of the time-domain speech enhancement model to be trained; when the loss value meets the preset conditions, the time-domain speech enhancement model to be trained is used as the trained speech enhancement model.

7. The Chinese speech enhancement method according to claim 6, characterized in that: According to the proportional coefficient matrix K α , enhanced speech signal x enhanced And the noisy speech x noisy Corresponding pure voice x clean , the step of calculating the loss value of the multi-resolution loss function comprises: According to the P group of parameters, the noisy speech x noisy Corresponding pure voice x clean Perform short-time Fourier transform respectively to obtain P amplitude spectra |STFT(x clean )|; Use the proportional coefficient matrix K corresponding to each set of parameters α , calculate each of the amplitude spectrum |STFT(x clean )|The enhanced amplitude spectrum of STFT(x clean )| K ; Using each enhanced magnitude spectrum |STFT(x clean )| K Calculate the loss value of the single-resolution loss function; The loss value of the multi-resolution loss function is calculated based on the obtained P loss values ​​of the single-resolution loss functions.

8. The Chinese speech enhancement method according to claim 7, characterized in that: The amplitude spectrum |STFT(x clean )|The enhanced amplitude spectrum of STFT(x clean )| K : |STFT(x)| K =K α |STFT(x clean )|。 9. The Chinese speech enhancement method according to claim 7, characterized in that: The loss value of the single-resolution loss function is calculated according to the following formula: In the formula, ||·|| F and ||·||1 denote the Frobenius norm and L1 norm, respectively, and N denotes the number of elements in the amplitude spectrum.

10. The Chinese speech enhancement method according to claim 7, characterized in that: The loss value of the multi-resolution loss function is calculated according to the following formula: Where, p = 1, 2,…, P.

Images