Speech enhancement model training method and device, speech enhancement method and device, equipment and medium

Abstract

The invention discloses a speech enhancement model training method and device, a speech enhancement method and device, equipment and a medium. Nonlinear adjustment is carried out on predicted speech features and ideal speech features through an equal-loudness curve and c power compression, perception differences of human ears for sounds of different frequencies are considered, the problem that low-frequency noise suppression is affected by too small low-frequency weights is solved, power compression is carried out on weight coefficients, the equal-loudness curve form is kept, but the dynamic range is compressed, and the low-frequency noise suppression effect is improved. And the model can better conform to auditory habits of human ears in a noise reduction or enhancement process. And calculating the difference between the adjusted predicted speech features and ideal speech features by adopting a loss function related to a time domain, thereby realizing fine feature difference calculation and loss value calculation, quantifying the prediction performance of the model, and guiding the training process of the model. Therefore, the model is ensured to control and adjust different frequency bands and voice frames more accurately in the noise reduction or enhancement process, so that the overall voice processing effect is improved.

Description

Technical Field

[0001] The present application relates to the field of digital signal processing and deep learning technology, and in particular to a speech enhancement model training and speech enhancement method, device, equipment and medium. Background Art

[0002] In the field of audio processing, speech enhancement technology is a crucial research direction, which aims to restore clear, high-quality speech from noise-contaminated speech signals. This technology has wide application value in multiple application scenarios such as speech recognition, voice communication, hearing aid design, and audio editing. However, traditional speech enhancement methods often rely on complex signal processing algorithms or statistical models. These methods often have unsatisfactory results when dealing with complex and changeable noise environments, and it is difficult to ensure the naturalness and intelligibility of the restored speech.

[0003] In recent years, with the rapid development of deep learning technology, speech enhancement models based on neural networks have gradually become a research hotspot. These models can achieve more effective speech restoration in complex noisy environments by automatically learning the mapping relationship between noise and clean speech. However, the frequency response of the human ear to speech signals varies significantly. For example, the sound frequency range that the human ear can perceive is from 20Hz to 20kHz, but the human ear feels different about sounds in different frequency bands. For example, sounds in the low-frequency band (such as 20Hz-60Hz) give people a strong sense of shock, while sounds in the mid-frequency band (such as 250Hz-4kHz) mainly affect the clarity and timbre of the sound. When processing speech signals, many existing speech enhancement models often do not fully consider the differences in the sensitivity of the human ear to speech of different frequencies, and cannot accurately reflect the true feelings of the human ear to speech signals, resulting in very poor listening experience of the final speech enhancement results. Summary of the invention

[0004] The present application provides a speech enhancement model training and speech enhancement method, device, equipment and medium, which are used to solve the problem that the existing speech enhancement model does not fully consider the differences in the sensitivity of the human ear to speech of different frequencies, and cannot accurately reflect the real perception of the human ear to the speech signal, resulting in the final speech enhancement result having a very poor listening experience.

[0005] In a first aspect, the present application provides a method for training a speech enhancement model, the method comprising:

[0006] Determine a sample speech signal based on any clean speech signal in the clean speech signal set and at least one noise signal included in the noise signal set;

[0007] Acquire predicted speech features based on the sample speech signal through an original speech enhancement model;

[0008] The predicted speech feature and the ideal speech feature of the clean speech signal are adjusted respectively by using equal loudness curves compressed by c power to obtain adjusted predicted speech features and adjusted ideal speech features; wherein c is greater than 0 and less than 1;

[0009] Determining a loss value corresponding to the sample speech signal based on the adjusted predicted speech feature and the adjusted ideal speech feature through a loss function associated with the time domain;

[0010] Based on the loss value, the original speech enhancement model is trained to obtain a trained speech enhancement model.

[0011] In a second aspect, the present application also provides a speech enhancement method based on the above-mentioned speech enhancement model, the method comprising:

[0012] Acquire a noisy speech signal;

[0013] The predicted clean speech is obtained based on the noisy speech signal through a pre-trained speech enhancement model.

[0014] In a third aspect, the present application also provides a training device for a speech enhancement model, the device comprising:

[0015] An acquisition unit, configured to determine a sample speech signal based on any clean speech signal in the clean speech signal set and at least one noise signal included in the noise signal set;

[0016] A first processing unit, configured to obtain predicted speech features based on the sample speech signal through an original speech enhancement model;

[0017] A second processing unit is used to adjust the predicted speech feature and the ideal speech feature of the clean speech signal respectively through equal loudness curves compressed by c power to obtain adjusted predicted speech features and adjusted ideal speech features; wherein c is greater than 0 and less than 1;

[0018] A third processing unit, configured to determine a loss value corresponding to the sample speech signal based on the adjusted predicted speech feature and the adjusted ideal speech feature by using a loss function associated with the time domain;

[0019] A training unit is used to train the original speech enhancement model based on the loss value to obtain a trained speech enhancement model.

[0020] In a fourth aspect, the present application also provides a speech enhancement device based on the above-mentioned model, the device comprising:

[0021] An acquisition module, used for acquiring a noisy speech signal;

[0022] The processing module is used to obtain predicted clean speech based on the noisy speech signal through a pre-trained speech enhancement model.

[0023] In a fifth aspect, the present application provides a computer device, which includes a processor, and the processor is used to implement the steps of the training method of the speech enhancement model as described above when executing a computer program stored in a memory, or to implement the steps of the speech enhancement method as described above.

[0024] In a sixth aspect, the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the training method of the speech enhancement model as described above, or implements the steps of the speech enhancement method as described above.

[0025] The beneficial effects of this application are as follows:

[0026] 1. Through equal loudness curves and c power compression (where c is greater than 0 and less than 1), the predicted speech features and ideal speech features are nonlinearly adjusted. This adjustment method takes into account the differences in the human ear's perception of sounds of different frequencies, overcomes the problem of low-frequency noise suppression affected by too small low-frequency weights, and performs power compression on the weight coefficients, maintaining the shape of the equal loudness curves, but compressing the dynamic range, so that the model is more in line with the human ear's auditory habits during noise reduction or enhancement.

[0027] 2. The loss function related to the time domain is used to calculate the difference between the adjusted predicted speech features and the ideal speech features, and the detailed feature difference calculation and loss value calculation are realized for each frequency band and each speech frame. The prediction performance of the model is quantified and used to guide the training process of the model, ensuring that the model has more precise control and adjustment of different frequency bands and speech frames during noise reduction or enhancement, thereby improving the overall speech processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 A schematic diagram of a training process of a speech enhancement model provided in an embodiment of the present application;

[0030] Figure 2 A schematic diagram of an equal loudness curve provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of the training process of a specific speech enhancement model provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of a speech enhancement process provided by an embodiment of the present application;

[0033] Figure 5 A schematic diagram of the structure of a training device for a speech enhancement model provided in an embodiment of the present application;

[0034] Figure 6 A schematic diagram of the structure of a speech enhancement device provided in an embodiment of the present application;

[0035] Figure 7 It is a structural schematic diagram of a computer device provided in an optional embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0037] In order to better simulate the auditory perception characteristics of the human ear and improve the speech enhancement effect, the present application provides a speech enhancement model training and speech enhancement method, device, equipment and medium.

[0038] Embodiment 1:

[0039] This application provides a method for training a speech enhancement model. Figure 1 A schematic diagram of a training process of a speech enhancement model provided in an embodiment of the present application, the process comprising:

[0040] S101: Determine a sample speech signal based on any clean speech signal in a clean speech signal set and at least one noise signal included in a noise signal set.

[0041] In the present application, the training method of the speech enhancement model is applied to a computer device, which may be an intelligent terminal, such as a computer, a robot, etc., or a server, such as an application server, a business server, etc.

[0042] In order to improve the auditory quality of the predicted clean speech, it is necessary to collect a clean speech signal set and a noise signal set in advance, and iteratively train the original speech enhancement model to obtain a trained speech enhancement model. The noise signal set includes a large number of noise signals, and the clean speech signal set includes a large number of clean speech signals, which are a batch of clean human voice signals collected in advance, that is, a batch of noise-free speech signals emitted by humans collected in advance.

[0043] In a possible implementation, noises that may appear in voice signals may be collected in advance for each working scenario of the smart device, and these noises may be determined as noise signals in the noise signal set according to their types. According to statistics, in working scenarios of smart devices, the more common types of noises include white noise, Gaussian noise, salt and pepper noise, etc.

[0044] Exemplarily, a clean speech signal and at least one noise signal are randomly obtained from the clean speech signal set and the noise signal set, respectively. The clean speech signal is mixed with the at least one noise signal, that is, the clean speech signal is subjected to noise processing to obtain a sample speech signal, and the clean speech signal is a clean speech signal corresponding to the sample speech signal. Subsequently, the original speech enhancement model is trained based on the sample speech signal and the clean speech signal corresponding to the sample speech signal to obtain a trained speech enhancement model.

[0045] It should be noted that the types of noise used in the process of adding noise to the clean speech signal should be as many as possible to make the sample speech signal more diverse, thereby improving the robustness and accuracy of the speech enhancement model trained subsequently.

[0046] S102: Acquire predicted speech features based on the sample speech signal through the original speech enhancement model.

[0047] After obtaining the sample speech signal based on the above embodiment, the sample speech signal can be input into the original speech enhancement model. The input sample speech signal is processed by the original speech enhancement model to obtain a predicted speech signal. The predicted speech feature represents the enhancement effect of the speech enhancement model on the noisy speech signal, and the speech enhancement model may include spectral features or time domain features, etc.

[0048] For example, the original speech enhancement model is roughly divided into three parts, where Conv2d is a convolutional layer used to extract deep-level features, the LSTM layer is used to model temporal relationships, and the last Dense layer, i.e., a linear layer, is used for dimensionality conversion to obtain the final mask value.

[0049] S103: adjusting the predicted speech feature and the ideal speech feature of the clean speech signal respectively through equal loudness curves compressed by c power to obtain adjusted predicted speech features and adjusted ideal speech features; wherein c is greater than 0 and less than 1.

[0050] In actual application scenarios, the frequency response of the human ear to speech signals has significant differences. For example, the sound frequency range that the human ear can perceive is from 20Hz to 20kHz, but the sound of different frequency bands is not the same to the human ear. For example, the sound of the low frequency band (such as 20Hz-60Hz) gives people a strong sense of shock, while the mid-frequency band (such as 250Hz-4kHz) mainly affects the clarity and timbre of the sound. Based on this, in the present application, the speech features of different frequency bands can be adjusted so that the speech signal in the frequency range that the human ear is sensitive to is strengthened. Exemplarily, the weighting coefficients corresponding to different frequency bands can be used to adjust the predicted speech features and the ideal speech features of the clean speech signal, so that the adjusted speech features can better fit the human ear.

[0051] In one example, the predicted speech feature and the ideal speech feature of the clean speech signal can be adjusted using c-power compressed equal loudness curves to obtain adjusted predicted speech features and adjusted ideal speech features, where c is a number greater than 0 and less than 1.

[0052] It should be noted that the equal loudness curve is a series of curves with consistent loudness levels obtained through subjective measurement. In simple terms, the sound pressure levels of different frequency bands on the same curve are the same to human ears. According to the sampling rate of 16000Hz (cutoff frequency band 8000Hz), the weighting coefficient W (μ) calculated by ISO 226:2003 standard is as follows: Figure 2 As shown, the horizontal axis is the frequency band and the vertical axis is the weighting coefficient value. Figure 2 The equal loudness curves shown show that the human ear is most sensitive to mid-frequency and not so sensitive to low frequency. Moreover, people from different countries have the same trend in sound sensitivity, which is that they are most sensitive to mid-frequency, followed by high frequency, and the lowest sensitivity to low frequency.

[0053] In the present application, if the speech feature includes the predicted speech feature and the ideal speech feature, the speech feature is a spectrum feature or a time domain feature.

[0054] Taking the speech feature as the short-time discrete cosine transform feature as an example, the sample speech signal and the clean speech signal are framed and windowed, and then the short-time discrete cosine transform feature is calculated. Taking the speech with a sampling rate of 16000Hz as an example, a 512-point window length, a 256-point frame shift, and a 512-point square root Hanning window can be selected as the window function. It should be noted that in practice, other parameter configurations can also be selected to adapt to system delays.

[0055] The short-time discrete cosine transform characteristic form is as follows:

[0056]

[0057] in,

[0058] The noisy speech features F of the sample speech signal can be obtained by formula (1): x (t, μ), ideal speech feature F of clean speech signal s (t, μ), where t, μ represent time and frequency respectively, N is the window size, which represents the number of samples considered in the short-time transform, z(μ) is a normalization factor used to ensure that the transform is orthogonal, and f(t) represents the input speech signal.

[0059] Assuming that the mask predicted by the original speech enhancement model is M(t, μ), the predicted speech feature is in, Representation prediction speech features, F x (t, μ) represents the characteristics of noisy speech.

[0060] S104: Determine a loss value corresponding to the sample speech signal based on the adjusted predicted speech feature and the adjusted ideal speech feature through a loss function related to the time domain.

[0061] After obtaining the adjusted predicted speech features and the adjusted ideal speech features based on the above-mentioned embodiment, a loss value can be determined based on the adjusted predicted speech features and the adjusted ideal speech features, so as to train the original speech enhancement model through the loss value and obtain a trained speech enhancement model. Exemplarily, a loss value can be determined based on the adjusted predicted speech features and the adjusted ideal speech features through a loss function related to the time domain, so as to better capture dynamic changes in time, such as transient characteristics and envelope information of speech signals, to ensure that the speech denoising model can have more precise control and adjustment of different frequency bands and speech frames during the denoising process, thereby improving the overall perceived quality of the denoising effect, making the denoised speech data more natural and clear, and enhancing the generalization ability of the model.

[0062] Among them, the time domain related loss function may include an L1 loss function, an L2 loss function, etc., which are not specifically limited here.

[0063] In a possible implementation, if the speech feature is a spectral feature, determining the loss value corresponding to the sample speech signal based on the adjusted predicted speech feature and the adjusted ideal speech feature through a loss function related to the time domain includes:

[0064] Performing time domain transformation on the adjusted predicted speech feature and the adjusted ideal speech feature respectively to obtain an adjusted predicted time domain feature and an adjusted ideal time domain feature;

[0065] The loss value corresponding to the sample speech signal is determined based on the adjusted predicted time domain features and the adjusted ideal time domain features through a loss function related to the time domain.

[0066] Since the speech features (including predicted speech features and ideal speech features) may be processed in the spectral domain (frequency domain) (such as short-time discrete cosine transform features, Mel cepstral coefficients, etc.), in order to utilize time-domain related information (such as the sensitivity of the human ear to sound changes at different time points), the spectral features can be converted back to time-domain features. Exemplarily, the spectral representation of the speech features (such as a spectrogram) is inversely transformed (such as an inverse discrete cosine transform, an inverse short-time Fourier transform, etc.) to obtain the corresponding time-domain waveform. Then, through a loss function related to the time domain, based on the adjusted predicted time-domain features and the adjusted ideal time-domain features, the loss value corresponding to the sample speech signal is determined.

[0067] In a possible implementation, determining the loss value corresponding to the sample speech signal based on the adjusted predicted time domain feature and the adjusted ideal time domain feature through a loss function related to the time domain includes:

[0068] Obtaining an inner product of the adjusted predicted time domain feature and the adjusted ideal time domain feature, and obtaining energy of the adjusted ideal time domain feature;

[0069] Determining, based on the inner product and the energy, a projection intensity of the adjusted predicted time domain feature in the direction of the adjusted ideal time domain feature;

[0070] Determining similarity according to the projection intensity and the adjusted ideal time domain feature;

[0071] Obtaining an error vector between the similarity and the adjusted predicted time-domain feature;

[0072] Determining a ratio between the square of the modulus of the similarity and the square of the modulus of the error vector;

[0073] The loss value is determined based on the logarithm of the ratio with a base of 10.

[0074] In order to ensure the processing effect of the speech enhancement model, in the present application, after obtaining the adjusted predicted time domain features and the adjusted ideal time domain features, the inner product of the adjusted predicted time domain features and the adjusted ideal time domain features can be determined, so as to measure the similarity of the two vectors in the direction through the inner product, which is helpful for calculating the correlation between the predicted speech and the ideal speech in the subsequent steps. And, the energy of the adjusted ideal time domain features is obtained. Then, according to the inner product and the energy, the projection intensity of the adjusted predicted time domain features in the direction of the adjusted ideal time domain features is determined to determine the accuracy of the predicted speech signal. According to the projection coefficient and the adjusted ideal time domain features, the similarity is determined to determine the best approximation of the predicted speech signal in the direction of the ideal speech. The error vector between the projection vector and the adjusted predicted time domain features is obtained to determine the difference between the predicted speech signal and the clean speech signal. The ratio between the square of the modulus of the projection vector and the square of the modulus of the error vector is determined to determine the ratio between the contribution and error of the predicted speech signal in the direction of the ideal speech through the ratio, which is helpful for evaluating the overall quality of the predicted speech signal. The obtained ratio is taken as the logarithm with base 10, and the loss value is obtained according to the ratio. The ratio is converted into the loss value through logarithmic transformation, so that the calculation of the loss value is more in line with the auditory perception characteristics of the human ear and is convenient for optimization during the training process. Among them, the smaller the loss value, the higher the similarity between the predicted signal and the ideal signal, that is, the better the noise reduction effect.

[0075] For example, by using a loss function related to the time domain, based on the adjusted predicted time domain features and the adjusted ideal time domain features, determining the loss value corresponding to the sample speech signal can be expressed by the following formula:

[0076]

[0077] B=As

[0078]

[0079] Among them, A represents the projection intensity, ~s represents the adjusted predicted time domain features, s represents the adjusted ideal time domain features, B represents the similarity, C represents the error vector, and Loss x Characterize the loss value.

[0080] In another possible implementation, determining the loss value corresponding to the sample speech signal based on the adjusted predicted time domain feature and the adjusted ideal time domain feature through a loss function related to the time domain includes:

[0081] Obtaining an inner product of the adjusted predicted time domain feature and the adjusted ideal time domain feature, and obtaining energy of the adjusted ideal time domain feature;

[0082] Determining, based on the inner product and the energy, a projection intensity of the adjusted predicted time domain feature in the direction of the adjusted ideal time domain feature;

[0083] Determining similarity according to the projection intensity and the adjusted ideal time domain feature;

[0084] Obtaining an error vector between the adjusted ideal time domain feature and the adjusted predicted time domain feature;

[0085] Determining a ratio between the square of the modulus of the similarity and the square of the modulus of the error vector;

[0086] The loss value is determined based on the logarithm of the ratio with a base of 10.

[0087] In order to ensure the processing effect of the speech enhancement model, in the present application, after obtaining the adjusted predicted time domain features and the adjusted ideal time domain features, the inner product of the adjusted predicted time domain features and the adjusted ideal time domain features can be determined, so as to measure the similarity of the two vectors in the direction through the inner product, which is helpful for calculating the correlation between the predicted speech and the ideal speech in the subsequent steps. And, the energy of the adjusted ideal time domain features is obtained. Then, according to the inner product and the energy, the projection intensity of the adjusted predicted time domain features in the direction of the adjusted ideal time domain features is determined to determine the accuracy of the predicted speech signal. According to the projection coefficient and the adjusted ideal time domain features, the similarity is determined to determine the best approximation of the predicted speech signal in the direction of the ideal speech. The error vector between the adjusted ideal time domain features and the adjusted predicted time domain features is obtained to determine the difference between the predicted speech signal and the clean speech signal. The ratio between the square of the modulus of the projection vector and the square of the modulus of the error vector is determined to determine the ratio between the contribution and error of the predicted speech signal in the direction of the ideal speech through the ratio, which is helpful for evaluating the overall quality of the predicted speech signal. The obtained ratio is taken as the logarithm with base 10, and the loss value is obtained according to the ratio. The ratio is converted into the loss value through logarithmic transformation, so that the calculation of the loss value is more in line with the auditory perception characteristics of the human ear and is convenient for optimization during the training process. Among them, the smaller the loss value, the higher the similarity between the predicted signal and the ideal signal, that is, the better the noise reduction effect.

[0088] For example, by using a loss function related to the time domain, based on the adjusted predicted time domain features and the adjusted ideal time domain features, determining the loss value corresponding to the sample speech signal can be expressed by the following formula:

[0089]

[0090] B=As

[0091]

[0092] Among them, A represents the projection intensity, ~s represents the adjusted predicted time domain features, s represents the adjusted ideal time domain features, B represents the similarity, C represents the error vector, and Loss y Characterize the loss value.

[0093] S105: Based on the loss value, the original speech enhancement model is trained to obtain a trained speech enhancement model.

[0094] Through the above embodiment, several sample speech signals can be obtained, and the above operation is performed for each sample speech signal. When the preset convergence condition is met, the training of the original speech enhancement model is completed.

[0095] In a possible implementation, the mask used to predict the predicted speech signal in the original speech enhancement model may be adjusted based on a loss function related to the time domain, until a trained speech enhancement model is obtained when the loss function satisfies a preset convergence condition.

[0096] The preset convergence condition may be that the sum of the loss values ​​determined based on each sample speech signal is always in a downward trend and tends to be flat, or the number of iterations for training the original speech enhancement model reaches the set maximum number of iterations, etc. The setting can be flexibly made in the specific implementation and is not specifically limited here.

[0097] As a possible implementation method, when training the original speech enhancement model, the sample speech signal can be divided into training samples and test samples. The original speech enhancement model is first trained based on the training samples, and then the reliability of the trained speech enhancement model is verified based on the test samples.

[0098] The beneficial effects of this application are as follows:

[0099] 1. Through equal loudness curves and c power compression (where c is greater than 0 and less than 1), the predicted speech features and ideal speech features are nonlinearly adjusted. This adjustment method takes into account the differences in the human ear's perception of sounds of different frequencies, overcomes the problem of low-frequency noise suppression affected by too small low-frequency weights, and performs power compression on the weight coefficients, maintaining the shape of the equal loudness curves, but compressing the dynamic range, so that the model is more in line with the human ear's auditory habits during noise reduction or enhancement.

[0100] 2. The loss function related to the time domain is used to calculate the difference between the adjusted predicted speech features and the ideal speech features, and the detailed feature difference calculation and loss value calculation are realized for each frequency band and each speech frame. The prediction performance of the model is quantified and used to guide the training process of the model, ensuring that the model has more precise control and adjustment of different frequency bands and speech frames during noise reduction or enhancement, thereby improving the overall speech processing effect.

[0101] Embodiment 2:

[0102] The training process of the speech enhancement model provided by the present application is described below through specific embodiments. Figure 3 A schematic diagram of a specific training process of a speech enhancement model provided in an embodiment of the present application, the process includes:

[0103] S301: Determine a sample speech signal based on any clean speech signal in a clean speech signal set and at least one noise signal included in a noise signal set.

[0104] S302: Preprocess the sample speech signal to obtain a preprocessed sample speech signal.

[0105] Taking the preprocessed sample speech signal as the short-time discrete cosine transform feature as an example, the sample speech signal is framed and windowed, and then the short-time discrete cosine transform feature is calculated. Taking the speech with a sampling rate of 16000Hz as an example, a 512-point window length, a 256-point frame shift, and a 512-point square root Hanning window can be selected as the window function.

[0106] The short-time discrete cosine transform characteristic form is as follows:

[0107]

[0108] in,

[0109] The noisy speech features F of the sample speech signal can be obtained by formula (1): x (t, μ), where t, μ represent time and frequency respectively, N is the window size, which represents the number of samples considered in the short-time transform, z(μ) is a normalization factor used to ensure that the transform is orthogonal, and f(t) represents the input sample speech signal.

[0110] S303: Obtain predicted speech features based on the sample speech signal through the original speech enhancement model. Assuming that the mask predicted by the original speech enhancement model is M(t, μ), the predicted speech features are in, Representation prediction speech features, F x (t, μ) represents the characteristics of noisy speech.

[0111] S304: adjusting the predicted speech feature and the ideal speech feature of the clean speech signal respectively through the equal loudness curve of c-power compression to obtain the adjusted predicted speech feature and the adjusted ideal speech feature.

[0112] For example, the adjusted predicted speech features and the adjusted ideal speech features can be expressed by the following formula:

[0113] W=W c

[0114]

[0115] Among them, ^W represents the equal loudness curve of c power compression, f represents the frequency band, t represents the time, and F represents s (t, f) represents the ideal speech features, Representation predicts speech features, Characterize the ideal speech features after adjustment, Characterize the adjusted predicted speech features.

[0116] S305: Obtain the inner product of the adjusted predicted time domain feature and the adjusted ideal time domain feature, and obtain the energy of the adjusted ideal time domain feature; based on the inner product and the energy, determine the projection intensity of the adjusted predicted time domain feature in the direction of the adjusted ideal time domain feature; determine the similarity according to the projection intensity and the adjusted ideal time domain feature; obtain the error vector between the adjusted ideal time domain feature and the adjusted predicted time domain feature; determine the ratio between the square of the modulus of the similarity and the square of the modulus of the error vector; determine the loss value according to the logarithm of the ratio with base 10.

[0117] For example, the loss value can be obtained using the following formula:

[0118]

[0119] B=As

[0120]

[0121] Among them, A represents the projection intensity, s represents the adjusted predicted time domain features, s represents the adjusted ideal time domain features, B represents the similarity, C represents the error vector, and Loss y Characterize the loss value.

[0122] S306: Based on the loss value, the original speech enhancement model is trained to obtain a trained speech enhancement model.

[0123] Embodiment 3:

[0124] The present application provides a speech enhancement method based on the speech enhancement model described in the above embodiment. Figure 4 A schematic diagram of a speech enhancement process provided in an embodiment of the present application, the process includes:

[0125] S401: Acquire a noisy speech signal.

[0126] S402: Obtain predicted clean speech based on the noisy speech signal using a pre-trained speech enhancement model.

[0127] The speech enhancement method provided in the present application is applied to a computer device, which may be an intelligent device or a server. The computer device for performing speech enhancement in the present application may be the same as or different from the computer device for performing speech enhancement model training described above.

[0128] In a possible implementation, the speech enhancement model training is generally performed in an offline manner.

[0129] Smart devices can continuously collect voice signals. Based on the voice signals collected by the smart devices, computer devices that perform voice enhancement can process the collected voice signals to remove noise contained in the voice signals.

[0130] If the computer device for voice enhancement is an intelligent device, after a voice signal is collected, the intelligent device can directly remove the noise contained in the voice signal based on the voice enhancement method provided by this application. If the computer device for voice enhancement is a server, after the intelligent device collects a voice signal, it sends the voice signal to the server, and the server removes the noise contained in the voice signal based on the voice enhancement method provided by this application.

[0131] In the present application, a speech enhancement model is pre-trained, and the noise contained in the speech signal is removed by the speech enhancement model. Exemplarily, a computer device for speech enhancement obtains a collected speech signal, determines the speech signal as a noisy speech signal, and inputs the noisy speech signal into a pre-trained speech enhancement model. The speech enhancement model receives the input noisy speech signal, processes the noisy speech signal, and obtains a predicted clean speech.

[0132] It should be noted that the specific training process of the speech enhancement model has been described in the above embodiment, and the repeated parts will not be repeated here.

[0133] Embodiment 4:

[0134] Based on the same inventive concept, the present application also provides a training device for a speech enhancement model. Figure 5A schematic diagram of a training device for a speech enhancement model provided in an embodiment of the present application, the device comprising:

[0135] An acquisition unit 51 is used to determine a sample speech signal based on any clean speech signal in the clean speech signal set and at least one noise signal included in the noise signal set;

[0136] A first processing unit 52 is used to obtain predicted speech features based on the sample speech signal through an original speech enhancement model;

[0137] A second processing unit 53 is used to adjust the predicted speech feature and the ideal speech feature of the clean speech signal respectively through the equal loudness curve compressed by c power to obtain the adjusted predicted speech feature and the adjusted ideal speech feature; wherein c is greater than 0 and less than 1;

[0138] A third processing unit 54 is used to determine a loss value corresponding to the sample speech signal based on the adjusted predicted speech feature and the adjusted ideal speech feature through a loss function related to the time domain;

[0139] The training unit 55 is used to train the original speech enhancement model based on the loss value to obtain a trained speech enhancement model.

[0140] The training device of the speech enhancement model in this embodiment is presented in the form of a functional module, where the module refers to an application specific integrated circuit (ASIC), a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0141] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0142] Embodiment 5:

[0143] The present application also provides a speech enhancement device based on the speech enhancement model described in the above embodiment, Figure 6 A schematic diagram of the structure of a speech enhancement device provided in an embodiment of the present application, the device comprising:

[0144] An acquisition module 61, used to acquire a noisy speech signal;

[0145] The processing module 62 is used to obtain predicted clean speech based on the noisy speech signal through a pre-trained speech enhancement model.

[0146] The speech enhancement device in this embodiment is presented in the form of a functional module, where the module refers to an application specific integrated circuit (ASIC), a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0147] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0148] Embodiment 6:

[0149] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present application, such as Figure 7 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0150] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0151] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0152] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0153] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0154] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0155] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0156] Embodiment 7:

[0157] On the basis of the above embodiments, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program executable by a processor is stored. When the program runs on the processor, the processor implements the following steps when executing:

[0158] Determine a sample speech signal based on any clean speech signal in the clean speech signal set and at least one noise signal included in the noise signal set;

[0159] Acquire predicted speech features based on the sample speech signal through an original speech enhancement model;

[0160] The predicted speech feature and the ideal speech feature of the clean speech signal are adjusted respectively by using equal loudness curves compressed by c power to obtain adjusted predicted speech features and adjusted ideal speech features; wherein c is greater than 0 and less than 1;

[0161] Determining a loss value corresponding to the sample speech signal based on the adjusted predicted speech feature and the adjusted ideal speech feature through a loss function associated with the time domain;

[0162] Based on the loss value, the original speech enhancement model is trained to obtain a trained speech enhancement model.

[0163] Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to the training method of the speech enhancement model, the implementation of the above-mentioned computer-readable storage medium can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0164] Embodiment 8:

[0165] On the basis of the above embodiments, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program executable by a processor is stored. When the program runs on the processor, the processor implements the following steps when executing:

[0166] Acquire a noisy speech signal;

[0167] The predicted clean speech is obtained based on the noisy speech signal through a pre-trained speech enhancement model.

[0168] Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to that of the speech enhancement method, the implementation of the above-mentioned computer-readable storage medium can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0169] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for training a speech enhancement model, characterized in that: The method comprises: Determine a sample speech signal based on any clean speech signal in the clean speech signal set and at least one noise signal included in the noise signal set; Acquire predicted speech features based on the sample speech signal through an original speech enhancement model; The predicted speech feature and the ideal speech feature of the clean speech signal are adjusted respectively by using equal loudness curves compressed by c power to obtain adjusted predicted speech features and adjusted ideal speech features; wherein c is greater than 0 and less than 1; Determining a loss value corresponding to the sample speech signal based on the adjusted predicted speech feature and the adjusted ideal speech feature through a loss function associated with the time domain; Based on the loss value, the original speech enhancement model is trained to obtain a trained speech enhancement model.

2. The method according to claim 1, characterized in that If the speech feature includes the predicted speech feature and the ideal speech feature, the speech feature is a spectrum feature or a time domain feature.

3. The method according to claim 2, characterized in that If the speech feature is a spectral feature, determining the loss value corresponding to the sample speech signal based on the adjusted predicted speech feature and the adjusted ideal speech feature by using a loss function related to the time domain includes: Performing time domain transformation on the adjusted predicted speech feature and the adjusted ideal speech feature respectively to obtain an adjusted predicted time domain feature and an adjusted ideal time domain feature; The loss value corresponding to the sample speech signal is determined based on the adjusted predicted time domain features and the adjusted ideal time domain features through a loss function related to the time domain.

4. The method according to claim 3, characterized in that Determining a loss value corresponding to the sample speech signal based on the adjusted predicted time domain feature and the adjusted ideal time domain feature through a loss function related to the time domain includes: Obtaining an inner product of the adjusted predicted time domain feature and the adjusted ideal time domain feature, and obtaining energy of the adjusted ideal time domain feature; Determining, based on the inner product and the energy, a projection intensity of the adjusted predicted time domain feature in the direction of the adjusted ideal time domain feature; Determining similarity according to the projection intensity and the adjusted ideal time domain feature; Obtaining an error vector between the similarity and the adjusted predicted time-domain feature; Determining a ratio between the square of the modulus of the similarity and the square of the modulus of the error vector; The loss value is determined based on the logarithm of the ratio with a base of 10.

5. The method according to claim 3, characterized in that Determining a loss value corresponding to the sample speech signal based on the adjusted predicted time domain feature and the adjusted ideal time domain feature through a loss function related to the time domain includes: Obtaining an inner product of the adjusted predicted time domain feature and the adjusted ideal time domain feature, and obtaining energy of the adjusted ideal time domain feature; Determining, based on the inner product and the energy, a projection intensity of the adjusted predicted time domain feature in the direction of the adjusted ideal time domain feature; Determining similarity according to the projection intensity and the adjusted ideal time domain feature; Obtaining an error vector between the adjusted ideal time domain feature and the adjusted predicted time domain feature; Determining a ratio between the square of the modulus of the similarity and the square of the modulus of the error vector; The loss value is determined based on the logarithm of the ratio with a base of 10.

6. A speech enhancement method based on the speech enhancement model according to any one of claims 1 to 5, characterized in that: The method comprises: Acquire a noisy speech signal; The predicted clean speech is obtained based on the noisy speech signal through a pre-trained speech enhancement model.

7. A training device for a speech enhancement model, characterized in that: The device comprises: An acquisition unit, configured to determine a sample speech signal based on any clean speech signal in the clean speech signal set and at least one noise signal included in the noise signal set; A first processing unit, configured to obtain predicted speech features based on the sample speech signal through an original speech enhancement model; A second processing unit is used to adjust the predicted speech feature and the ideal speech feature of the clean speech signal respectively through equal loudness curves compressed by c power to obtain adjusted predicted speech features and adjusted ideal speech features; wherein c is greater than 0 and less than 1; A third processing unit, configured to determine a loss value corresponding to the sample speech signal based on the adjusted predicted speech feature and the adjusted ideal speech feature by using a loss function associated with the time domain; A training unit is used to train the original speech enhancement model based on the loss value to obtain a trained speech enhancement model.

8. A speech enhancement device based on any one of the models described in claims 1-5, characterized in that: The device comprises: An acquisition module, used for acquiring a noisy speech signal; The processing module is used to obtain predicted clean speech based on the noisy speech signal through a pre-trained speech enhancement model.

9. A computer device, characterized in that: The computer device includes a processor, which is used to implement the steps of the training method of the speech enhancement model as described in any one of claims 1 to 5 above when executing the computer program stored in the memory, or to implement the steps of the speech enhancement method as described in claim 6 above.

10. A computer-readable storage medium, characterized in that: It stores a computer program that can be executed by a computer device. When the program is run on the computer device, the computer device executes the steps of the training method of the speech enhancement model as described in any one of claims 1 to 5 above, or implements the steps of the speech enhancement method as described in claim 6 above.

Images