Speech enhancement method, device, computer equipment and storage medium

Abstract

The present application discloses a speech enhancement method, device, computer equipment and storage medium. The method comprises acquiring speech data; based on a pre-trained speech enhancement model, acquiring a human voice detection result, a signal-to-noise ratio detection result and a speech gain parameter of the speech data, wherein the human voice detection result is used to characterize the data corresponding to the human voice part in the speech data, the signal-to-noise ratio detection result is used to characterize the signal-to-noise ratio corresponding to the speech data, and the speech gain parameter is used to enhance the speech data; based on the human voice detection result and the signal-to-noise ratio detection result, adjusting the speech gain parameter; based on the adjusted speech gain parameter, performing enhancement processing on the speech data to obtain enhanced speech data. The present method can realize dynamic enhancement of speech data and improve the clarity and intelligibility of speech data.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and more specifically, to a speech enhancement method, apparatus, computer equipment and storage medium. Background Art

[0002] Currently, deep learning has begun to be applied to speech enhancement technology. Deep learning can effectively deal with sudden transient noise by processing speech data. Compared with traditional signal processing algorithms, the clarity and intelligibility of enhanced speech data have been greatly improved. However, the enhancement of speech data by neural networks is often a global gain of speech, which may excessively suppress some valid data in the speech. Summary of the invention

[0003] In view of the above problems, the present application proposes a speech enhancement method, apparatus, computer device and storage medium to achieve dynamic enhancement of speech data.

[0004] In a first aspect, an embodiment of the present application provides a speech enhancement method, the method comprising: obtaining speech data; based on a pre-trained speech enhancement model, obtaining a human voice detection result, a signal-to-noise ratio detection result, and a speech gain parameter of the speech data, the human voice detection result is used to characterize the data corresponding to the human voice part in the speech data, the signal-to-noise ratio detection result is used to characterize the signal-to-noise ratio corresponding to the speech data, and the speech gain parameter is used to enhance the speech data; based on the human voice detection result and the signal-to-noise ratio detection result, adjusting the speech gain parameter; and enhancing the speech data based on the adjusted speech gain parameter to obtain enhanced speech data.

[0005] In the second aspect, the embodiment of the present application provides a speech enhancement device, which includes: a speech acquisition module, a data acquisition module, a gain adjustment module and a speech enhancement module. The speech acquisition module is used to acquire speech data; the data acquisition module is used to acquire the human voice detection result, the signal-to-noise ratio detection result and the speech gain parameter of the speech data based on the pre-trained speech enhancement model, the human voice detection result is used to characterize the data corresponding to the human voice part in the speech data, the signal-to-noise ratio detection result is used to characterize the signal-to-noise ratio corresponding to the speech data, and the speech gain parameter is used to enhance the speech data; the gain adjustment module is used to adjust the speech gain parameter based on the human voice detection result and the signal-to-noise ratio detection result; the speech enhancement module is used to enhance the speech data based on the adjusted speech gain parameter to obtain enhanced speech data.

[0006] In a third aspect, an embodiment of the present application provides a computer device, comprising: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the speech enhancement method provided in the first aspect above.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the speech enhancement method provided in the first aspect above.

[0008] The solution provided by the present application obtains voice data, obtains the human voice detection result, signal-to-noise ratio detection result and voice gain parameter of the voice data based on a pre-trained voice enhancement model, adjusts the voice gain parameter based on the human voice detection result and the signal-to-noise ratio detection result, and performs enhancement processing on the voice data based on the adjusted voice gain parameter to obtain enhanced voice data. The method can suppress noise in the voice data and improve the clarity and intelligibility of the voice data by dynamically adjusting the gain parameter for the voice data. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A flow chart of a speech enhancement method provided in one embodiment of the present application is shown.

[0011] Figure 2 A flow chart of a speech enhancement method provided in another embodiment of the present application is shown.

[0012] Figure 3 A specific flow chart of step S230 in another embodiment of the present application is shown.

[0013] Figure 4 A flow chart of a speech enhancement method provided in yet another embodiment of the present application is shown.

[0014] Figure 5 A specific flow chart of step S310 in another embodiment of the present application is shown.

[0015] Figure 6 A specific flow chart of step S320 in another embodiment of the present application is shown.

[0016] Figure 7 A specific flow chart of step S323 in another embodiment of the present application is shown.

[0017] Figure 8 It shows a structural block diagram of a speech enhancement device provided in an embodiment of the present application.

[0018] Fig. 9 A structural block diagram of a computer device provided in an embodiment of the present application is shown.

[0019] Fig.10 A structural block diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0021] Neural networks are often used to process audio data. By enhancing speech data, they can suppress noise in speech data and improve the clarity of speech data. However, the adjustment of speech data by neural networks is often a global adjustment. It may suppress the effective human voice in the speech data while suppressing the noise in the speech data.

[0022] In view of the above problems, the inventors have proposed a speech enhancement method, apparatus, computer device and storage medium provided in the embodiments of the present application to achieve dynamic adjustment of gain parameters of speech data. The specific speech enhancement method will be described in detail in the subsequent embodiments.

[0023] The speech enhancement method provided in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0024] See also Figure 1 , Figure 1 A flow chart of a speech enhancement method provided by an embodiment of the present application is shown below. Figure 1 The process shown is described in detail, and the speech enhancement method may specifically include the following steps:

[0025] Step S110: Acquire voice data.

[0026] In an embodiment of the present application, a computer device can obtain voice data that needs to be voice enhanced, and enhance the voice data based on a gain value obtained by a voice enhancement model, thereby suppressing noise in the voice data and improving the clarity of the voice data. It is understandable that when people are recording or talking through a device, the relevant device may not only collect human voice data containing useful information, but also collect the noise in the surrounding environment. Therefore, it is necessary to perform noise reduction and other processing on the acquired voice data to suppress the noise in the voice data and amplify the human voice in the voice data.

[0027] In some implementations, in order to more accurately obtain the gain parameter corresponding to the speech data, the computer device may process the pre-acquired speech data to make it conform to the input conditions of the speech enhancement model, so as to more conveniently enhance the speech data. For example, after acquiring the speech data, the computer device may perform a fast Fourier transform on the speech data to obtain its corresponding complex spectrum, and then extract the amplitude of the complex spectrum to obtain the corresponding amplitude spectrum.

[0028] It can be understood that the voice data acquired by the computer device is continuous data in the time domain. After fast Fourier transform, the corresponding complex spectrum in the frequency domain is obtained. The complex spectrum is a function that characterizes the relationship between frequency and amplitude. The value of the complex spectrum at each frequency point is a complex number, and the complex number can be uniquely determined by the modulus and the angle of argument, so the complex spectrum can be decomposed into an amplitude spectrum and a phase spectrum. The amplitude spectrum can be used as the input of the speech enhancement model. After obtaining the gain parameter corresponding to the speech data, the amplitude spectrum can be enhanced based on the gain parameter, and the enhanced amplitude spectrum is recombined with the phase spectrum to obtain the enhanced complex spectrum, and then the enhanced speech data is obtained, thereby suppressing the noise in the speech data.

[0029] In some implementations, the computer device may be a mobile terminal such as a smart phone, a tablet computer, or a smart watch that is equipped with a recording device, and the computer device may obtain voice data through the recording device.

[0030] In other embodiments, the computer device can obtain voice data locally, that is, obtain voice data from locally stored files. For example, when the computer device is a mobile terminal, it can obtain voice data stored locally from a recording folder, or download voice data from the network in advance and store it in a local folder. When the voice data needs to be processed, the voice data is read from the corresponding folder.

[0031] In some other implementations, when the computer device is a computer or a mobile terminal, the voice data can also be downloaded from the network. For example, the computer device can download the required voice data from the corresponding server through a wireless network, a data network, etc.

[0032] In some further implementations, the computer device may also receive input voice data through input operations of the user in other devices, thereby acquiring the voice data.

[0033] There is no limitation on the specific way in which the computer device obtains the voice data.

[0034] Step S120: Based on a pre-trained speech enhancement model, a voice detection result, a signal-to-noise ratio detection result, and a speech gain parameter of the speech data are obtained, wherein the voice detection result is used to characterize the data corresponding to the voice part in the speech data, the signal-to-noise ratio detection result is used to characterize the signal-to-noise ratio corresponding to the speech data, and the speech gain parameter is used to enhance the speech data.

[0035] In the embodiment of the present application, after acquiring the voice data, the computer device may input the voice data into a pre-trained voice enhancement model to obtain a human voice detection result, a signal-to-noise ratio detection result, and a voice gain parameter corresponding to the voice data, so that the computer device can enhance and adjust the voice data based on the acquired parameters.

[0036] It can be understood that the voice detection result, signal-to-noise ratio detection result and voice gain parameter generated by the speech enhancement model based on the input voice data can generate corresponding voice detection results, signal-to-noise ratio detection results and voice gain parameters for each frame of the voice data. That is, the voice data can input each frame of data into the speech enhancement model in time sequence, and the speech enhancement model can also generate corresponding voice detection results, signal-to-noise ratio detection results and voice gain parameters in real time based on each frame of voice data input. Finally, the voice detection result, signal-to-noise ratio detection result and voice gain parameters corresponding to the entire voice data can be obtained.

[0037] In some embodiments, the human voice detection result corresponding to the speech data generated by the speech enhancement model can be used to characterize the data corresponding to the human voice part in the speech data. For the speech data as a whole, the human voice detection result can represent the partial data segments in which the human voice exists in the speech data and the partial data segments in which the human voice does not exist. For each frame of speech data, the corresponding human voice detection result can characterize whether there is a human voice in the frame data. For example: if there is a human voice in the frame data, the speech enhancement model can mark the human voice detection result as "1", and if there is no human voice, the human voice detection result is marked as "0". The human voice detection result corresponding to each frame of speech data can form the human voice detection result of the speech data as a whole. For example: the human voice detection result corresponding to each frame of speech data is used as an element to form a vector. If the speech data includes 200 frames of data, the human voice detection result corresponding to the speech data can be a vector containing 200 elements.

[0038] In some embodiments, the signal-to-noise ratio detection result generated by the speech enhancement model can be used to characterize the signal-to-noise ratio corresponding to the speech data, that is, the ratio between the signal strength and the noise strength in the speech data. For the speech data as a whole, the signal-to-noise ratio detection result can characterize the ratio between the strength of the human voice signal and the strength of the noise signal in the speech data. For each frame of speech data, its corresponding signal-to-noise ratio detection result can be used to characterize the ratio of human voice to noise in this frame of data. It can be understood that the greater the signal-to-noise ratio, the smaller the noise intensity included in the frame of speech data. Similar to the human voice detection result, the signal-to-noise ratio detection result corresponding to the speech data as a whole can also be composed of the signal-to-noise ratio detection results corresponding to each frame of speech data.

[0039] Step S130: adjusting the speech gain parameter based on the human voice detection result and the signal-to-noise ratio detection result.

[0040] In an embodiment of the present application, after the computer device obtains the human voice detection result, the signal-to-noise ratio detection result and the semantic gain parameter corresponding to the semantic data based on the semantic enhancement model, the voice gain parameter can be adjusted based on the human voice detection result and the signal-to-noise ratio detection result. It is used to enhance the voice data based on the semantic gain parameter to achieve the effect of noise reduction. It can be understood that the human voice detection result is used to characterize whether there is a human voice in the voice data. The computer device can separate the human voice segment from the non-human voice segment in the voice data based on the human voice detection result, so as to appropriately reduce the voice gain parameter corresponding to the human voice segment in the voice data to avoid suppressing the effective human voice; and the signal-to-noise ratio detection result is used to characterize the signal-to-noise ratio corresponding to the voice data. The computer device can obtain the part of the voice data with a larger noise intensity based on the signal-to-noise ratio detection result, that is, the part of the data segment with a lower signal-to-noise ratio, to increase the voice gain parameter corresponding to this part, so as to achieve the effect of noise reduction on the voice data.

[0041] Step S140: performing enhancement processing on the speech data based on the adjusted speech gain parameter to obtain enhanced speech data.

[0042] In an embodiment of the present application, after the computer device adjusts the speech gain parameter based on the human voice detection result and the signal-to-noise ratio detection result, the speech data can be enhanced based on the speech gain parameter. Specifically, the speech gain parameter acquired by the computer device can be a matrix, and the amplitude spectrum is enhanced based on the speech gain parameter, that is, the matrix is ​​convolved with the amplitude spectrum corresponding to the speech data to obtain an enhanced amplitude spectrum, and then the enhanced amplitude spectrum can be combined with the phase spectrum corresponding to the speech data to obtain the enhanced complex spectrum of the speech data, and then the enhanced speech data is obtained by inverse Fourier transform. It can be understood that the speech gain parameter may include the speech gain parameter corresponding to each frame of data in the speech data, and the computer device enhances each frame of data based on the corresponding element of each frame of data in the speech gain parameter matrix. Thereby achieving enhanced adjustment of the speech data as a whole, and different adjustments between the speech data parts.

[0043] The speech enhancement method provided in the embodiment of the present application obtains speech data, obtains the human voice detection result, signal-to-noise ratio detection result and speech gain parameter of the speech data based on the pre-trained speech enhancement model, adjusts the speech gain parameter based on the human voice detection result and the signal-to-noise ratio detection result, and performs enhancement processing on the speech data based on the adjusted speech gain parameter to obtain enhanced speech data. Dynamic adjustment of the gain parameter for the speech data is achieved, thereby adaptively enhancing the speech data, thereby avoiding suppressing the effective human voice in the speech data, suppressing the noise in the speech data, and improving the clarity and intelligibility of the speech data.

[0044] See also Figure 2 , Figure 2 A flow chart of a speech enhancement method provided by another embodiment of the present application is shown. Figure 2 The process shown is described in detail, and the speech enhancement method may specifically include the following steps:

[0045] Step S210: Acquire voice data.

[0046] Step S220: Based on a pre-trained speech enhancement model, a voice detection result, a signal-to-noise ratio detection result, and a speech gain parameter of the speech data are obtained, wherein the voice detection result is used to characterize the data corresponding to the voice part in the speech data, the signal-to-noise ratio detection result is used to characterize the signal-to-noise ratio corresponding to the speech data, and the speech gain parameter is used to enhance the speech data.

[0047] In the embodiment of the present application, step S210 and step S220 may refer to the contents of other embodiments and will not be described in detail here.

[0048] Step S230: Based on the human voice detection result and the signal-to-noise ratio detection result, determine the first data and the second data in the voice data, the first data being data in which the proportion of human voice is greater than a preset ratio, and the second data being data in the voice data other than the first data.

[0049] In the embodiment of the present application, after the computer device obtains the human voice detection result, the signal-to-noise ratio detection result and the voice gain parameter corresponding to the voice data through the voice enhancement model, it can be based on the human voice detection result and the signal-to-noise ratio detection result, and the data in the voice data other than the first data is used as the first data. It can be understood that the computer device divides the voice data into the first data and the second data, which is determined based on the proportion of human voice in the voice data. Wherein the proportion of human voice in the first data is greater than the preset proportion, the computer device can regard the first data as a valid human voice data segment. Of course, the first data can also include noise, but it is obvious that the human voice part is more. The second data can also include human voice, but the noise part contained therein is more. The computer device divides the voice data into the first data and the second data, and adjusts its voice gain parameters differently according to the difference of the main parts included in the first data and the second data, so as to realize the dynamic adjustment of the voice data and adapt to different usage scenarios.

[0050] In some embodiments, Figure 3 As shown, determining the first data and the second data in the voice data in step S230 may include the following steps:

[0051] Step S231: Based on the human voice detection result, determine data corresponding to the human voice part in the speech data as the third data.

[0052] In an embodiment of the present application, after obtaining the human voice detection result and the signal-to-noise ratio detection result corresponding to the voice data, the computer device can divide the voice data into a human voice part and a non-human voice part based on the human voice detection result, and use the data of the human voice part as the third data. It can be understood that the human voice detection result is used to characterize whether there is a human voice in the voice data, but because the voice data may be mixed with noise, and the strength of the noise cannot be determined, the voice gain parameter corresponding to the third data cannot be directly reduced. For example: there is a segment of data in the third data, and there is a human voice in the segment of data, but the corresponding signal-to-noise ratio is very low, indicating that although there is a human voice in the segment of data, the noise intensity is very large. At this time, the computer device needs to increase the voice gain parameter corresponding to the segment of data to suppress the noise in the segment of data.

[0053] In some embodiments, the human voice detection result output by the speech enhancement model can be a vector representing the probability of the existence of human voice in the speech data, and each element in the vector can represent the probability of the existence of human voice in the corresponding frame in the speech data. That is, within the probability range of "0-1", if the size of any element in the vector is "0.8", it means that there is a 0.8 probability of the existence of human voice in a frame of speech data corresponding to the element. After the computer device obtains the human voice detection result corresponding to the speech data, that is, the probability vector, it can divide the human voice detection result into human voice segments and non-human voice segments according to a pre-set probability standard, that is, the probability of the existence of human voice corresponding to each frame in the human voice segment is greater than the pre-set probability standard, and the probability of the existence of human voice corresponding to each frame in the non-human voice segment is less than or equal to the pre-set probability standard.

[0054] Step S232: Based on the signal-to-noise ratio detection result, determine data having a signal-to-noise ratio greater than a preset signal-to-noise ratio from the third data as the first data, and the preset signal-to-noise ratio is set based on the preset ratio.

[0055] In the embodiment of the present application, the computer device can appropriately reduce the speech gain parameter corresponding to the first data because there is valid human voice data in the first data, and the noise intensity in the first data is relatively low, that is, it can be considered that the voice data in the first data are relatively clear human voices, and the degree of suppression of the first data can be reduced at this time to avoid suppressing the valid human voice in the first data. The method for the computer device to obtain the first data can be obtained by selecting a part with weaker noise intensity in the third data. From the concept of signal-to-noise ratio, it can be known that the larger the signal-to-noise ratio, the weaker the noise intensity corresponding to the voice data, and vice versa. The noise intensity is stronger. Therefore, the computer device can determine the noise intensity in the voice data based on the signal-to-noise ratio detection result corresponding to the voice data, and then select the data with a signal-to-noise ratio greater than the preset signal-to-noise ratio in the third data as the first data. At this time, the first data not only includes valid human voice data, but also the corresponding signal-to-noise ratio is large, and the noise intensity is weak.

[0056] In some implementations, the signal-to-noise ratio detection result output by the speech enhancement model may be a prediction result of the corresponding signal-to-noise ratio in the speech data. The signal-to-noise ratio detection result may be a predicted signal-to-noise ratio vector, where each element of the vector corresponds to the signal-to-noise ratio corresponding to a frame of data in the speech data.

[0057] Step S233: Acquire data other than the first data from the voice data as the second data.

[0058] In the embodiment of the present application, after acquiring the first data, the computer device can use all the data in the voice data except the first data as the second data, so the second data may include some data segments with human voice data but strong noise intensity. Since the signal-to-noise ratio of this part of data is too low and the noise intensity is too strong, it is necessary to suppress it, that is, increase its corresponding voice gain parameter. The second data may also include some data with no human voice data but only noise. Since this part of data does not contain human voice data, its corresponding voice gain parameter can also be increased to further suppress the noise data contained therein.

[0059] Step S240: reducing the speech gain parameter corresponding to the first data, and increasing the speech gain parameter corresponding to the second data.

[0060] In the embodiment of the present application, it can be seen from the above analysis that the human voice in the first data is strong and the noise is weak, which is relatively clear voice data. Therefore, the computer device can appropriately reduce the voice gain parameter corresponding to the first data to avoid suppressing the effective human voice therein, and can also appropriately increase the voice gain parameter corresponding to the second data. Among them, the reduction range of the voice gain parameter corresponding to each frame of data in the first data, and the increase range of the voice gain parameter corresponding to each frame of data in the second data can be determined based on different human voice detection results and signal-to-noise ratio detection results corresponding to each frame of data, or can be manually adjusted based on human experience, which is not limited here.

[0061] Step S250: performing enhancement processing on the speech data based on the adjusted speech gain parameter to obtain enhanced speech data.

[0062] In the embodiment of the present application, step S250 can refer to the contents of other embodiments and will not be repeated here.

[0063] The speech enhancement method provided in the embodiment of the present application inputs speech data into a pre-trained speech enhancement model, determines the human voice part in the speech data as the third data through the human voice detection result output by the speech enhancement model; determines the data with a signal-to-noise ratio greater than a preset signal-to-noise ratio from the third data as the first data through the signal-to-noise ratio detection result output by the speech enhancement model; uses the data other than the first data in the speech data as the second data; reduces the speech gain parameter corresponding to the first data, increases the speech gain parameter corresponding to the second data, and then enhances the speech data based on the adjusted speech gain parameter to obtain enhanced speech data. By adjusting the different speech gain parameters between the effective human voice segment and the invalid noise segment in the speech data, dynamic adjustment of the speech data is achieved, and the clarity and intelligibility of the speech data are improved.

[0064] See also Figure 4 , Figure 4 A flow chart of a speech enhancement method provided by another embodiment of the present application is shown below. Figure 4 The process shown is described in detail, and the speech enhancement method may specifically include the following steps:

[0065] Step S310: Acquire the sample speech set, the sample speech set includes multiple sections of noisy speech, each section of the noisy speech is annotated with a voice label and a signal-to-noise ratio label, each section of the noisy speech is obtained by mixing a clean speech sample and a noise sample, the voice label is used to characterize data corresponding to the voice part in the noisy speech, and the signal-to-noise ratio label is used to characterize the signal-to-noise ratio corresponding to the noisy speech.

[0066] For the speech enhancement model in the aforementioned embodiment, the embodiment of the present application also includes a training method for the speech enhancement model. It is worth noting that the training of the speech enhancement model can be performed in advance based on the acquired training data set. Subsequently, each time the speech gain parameters corresponding to the speech data need to be obtained, the trained speech enhancement model can be used for training, without the need to train the speech enhancement model each time the speech gain parameters corresponding to the speech data need to be obtained. In the embodiment of the present application, the training process of the speech enhancement model will be described.

[0067] In an embodiment of the present application, the sample speech set may include multiple sections of mixed-noise speech, each of which includes valid human voice data and noise data, so as to train the initial enhancement model to obtain a speech enhancement model. Among them, the mixed-noise speech is marked with a human voice label to characterize the data corresponding to the human voice part and the data corresponding to the non-human voice part in the speech data, such as the human voice part data in the mixed-noise speech can be marked with a label of "1", and the non-human voice part data in the mixed-noise speech can be marked with a label of "0". It can be understood that the role of the label is only to distinguish the human voice part data from the non-human voice part data. The mixed-noise speech is also marked with a signal-to-noise ratio label to characterize the signal-to-noise ratio corresponding to the mixed-noise speech. The signal-to-noise ratio refers to the ratio between the signal and the noise intensity. The signal-to-noise ratio corresponding to the mixed-noise speech can represent the intensity of the noise in the mixed-noise speech. It can be understood that if the signal-to-noise ratio corresponding to the mixed-noise speech is lower, the noise included in the speech is stronger, and vice versa. Training the initial enhancement model based on the mixed-noise speech can improve the accuracy of the speech gain parameters generated by the model.

[0068] In some embodiments, when labeling the noisy speech, the labeling can be performed based on each frame of data in each segment of the noisy speech, that is, each frame of data corresponding to each segment of the noisy speech has its corresponding human voice label and signal-to-noise ratio label, which is used to mark whether there is human voice in the frame data and the signal-to-noise ratio corresponding to the frame data.

[0069] In some embodiments, Figure 5 As shown, step S310 may include the following steps:

[0070] Step S311: Acquire multiple clean speech samples and multiple noise samples.

[0071] In the implementation manner of the present application, the mixed-noise speech in the sample speech set can be obtained by mixing clean speech samples and noise samples. The mixed-noise speech obtained by mixing can not only make the human voice label and signal-to-noise ratio label corresponding to the mixed-noise speech more accurate, but also mix different proportions of speech data according to different model requirements, so that the initial enhancement model can be trained with different effects.

[0072] Step S312: Acquire vocal tags for marking vocal segments and non-vocal segments in the clean speech.

[0073] In the implementation mode of the present application, since the noisy speech used to train the initial enhancement model needs to have a human voice label and a signal-to-noise ratio label to make the model training accurate, the human voice label and the signal-to-noise ratio label in the noisy speech need to be accurately labeled to avoid errors in model training. In order to accurately label the human voice label in the noisy speech, the human voice segment and the non-human voice segment in the clean speech can be labeled. Since there is no noise in the clean speech, the division of the human voice segment and the non-human voice segment in the clean speech will be more accurate, thereby making the human voice label in the noisy speech obtained from the clean speech correspondingly accurate.

[0074] In some implementations, a voice activity detection (VAD) tool can be used to determine valid speech in clean speech, and the human voice segment in the clean speech can be labeled as "1", and the non-human voice segment in the clean speech can be labeled as "0". Thus, the noisy speech obtained based on the clean speech also carries corresponding labels for identifying the human voice segment and the non-human voice segment in the noisy speech.

[0075] Step S313: Mix the clean speech and the noise according to different signal-to-noise ratios to obtain multiple segments of the mixed-noise speech.

[0076] In the implementation mode of the present application, the multiple sections of mixed-noise speech in the sample speech set need to have different signal-to-noise ratios so that the trained model can be more widely used. When the initial enhanced model is trained based on the sample speech set, it is necessary to determine whether the training of the initial enhanced model is correct based on the signal-to-noise ratio corresponding to the mixed-noise speech. Therefore, the correctness of the signal-to-noise ratio corresponding to the mixed-noise speech directly affects the accuracy of the training of the initial enhanced model. Therefore, by mixing the clean speech and the noise data according to different preset signal-to-noise ratios, multiple sections of mixed-noise speech with controllable signal-to-noise ratios can be obtained, and the model trained based on these mixed-noise speech can also have higher accuracy and a wider range of applications.

[0077] Step S314: labeling each segment of the noisy speech with a signal-to-noise ratio label based on the signal-to-noise ratio used when mixing each segment of the noisy speech, and labeling each segment of the noisy speech with a voice label based on the voice label of the clean speech, to obtain the sample speech set.

[0078] In the implementation mode of the present application, when the clean speech is mixed with the noise, the signal-to-noise ratio during the mixing can be recorded and used as the signal-to-noise ratio label corresponding to the mixed-noise speech. At the same time, by marking the human voice segment and the non-human voice segment in the clean speech, that is, marking the human voice label in the clean speech, multiple mixed-noise speech segments obtained by mixing the clean speech segment with the noise can also have corresponding human voice labels.

[0079] In some implementations, after labeling each noisy speech segment with a signal-to-noise ratio label and a human voice label in step S314, the following steps may be performed:

[0080] Step S3141: performing reverberation processing on part of the multiple noisy speech segments, and adding the noisy speech after the reverberation processing to the sample speech set.

[0081] In an implementation manner of the present application, after obtaining noisy speech, part of the noisy speech can be selected from multiple noisy speech segments for reverberation processing, and the reverberated speech data can be added to the sample speech set again. Thus, the sample speech set includes not only the noisy speech, but also part of the reverberated speech, thereby expanding the speech data in the sample speech set for training the model. At the same time, the initial enhancement model is trained based on the sample speech set at this time, and the model can also have a certain ability to suppress reverberation.

[0082] In some implementations, a certain number of noisy speech segments may be randomly selected from multiple noisy speech segments according to a preset probability, and a certain degree of reverberation may be added to the noisy speech segments through a reverberation and aliasing tool, wherein the degree of reverberation added may be randomly selected.

[0083] Step S320: training the initial enhancement model according to the sample speech set, and the clean speech samples and noise samples corresponding to the mixed-noise speech, until the initial enhancement model meets the preset conditions, thereby obtaining a trained speech enhancement model.

[0084] In the implementation mode of the present application, after obtaining a sample speech set including multiple segments of mixed-noise speech, the initial enhancement model can be trained based on the mixed-noise speech to obtain a speech enhancement model after the training is completed. The preset conditions are used to determine whether the initial enhancement model can accurately obtain the speech gain parameters corresponding to the input speech data. If the initial enhancement model meets the preset conditions, it indicates that the model can accurately output the speech gain parameters corresponding to the speech data, so that the initial enhancement model can be used as a mature speech enhancement model.

[0085] In some embodiments, Figure 6 As shown, the initial enhancement model includes a first input module, a first output module, a second output module and a third output module. Step S320 may include the following steps:

[0086] Step S321: inputting the amplitude spectrum corresponding to the noisy speech into the first input module to obtain the hidden representation corresponding to the noisy speech and the initial speech gain parameter obtained based on the hidden feature.

[0087] In an embodiment of the present application, when the initial enhancement model is trained based on the noisy speech, the noisy speech can be subjected to a fast Fourier transform to obtain its complex spectrum, the amplitude in the complex spectrum is extracted as the amplitude spectrum corresponding to the noisy speech, and the amplitude spectrum is input into the first input module. The first input module can be a composite network composed of a convolutional neural network (CNN) and a long short-term memory network (LSTM). After the amplitude spectrum corresponding to the noisy speech is input into the first input module, the hidden representation corresponding to the noisy speech and the initial speech gain parameter can be obtained, wherein the acquisition of the initial speech gain parameter is based on the hidden representation.

[0088] Specifically, the first input module may include two parts. The first part may be a 7-layer CNN encoder structure, which can extract a hidden representation from the amplitude spectrum corresponding to the input noisy speech. The hidden representation can not only be used to input other modules in the initial enhancement model to obtain the corresponding parameters of the noisy speech, but also can be input into the second part of the first input module, i.e., a 2-layer LSTM network, to add timing information to the hidden representation, and finally restore the hidden representation with the timing information to the same size as the input amplitude spectrum as the initial speech gain parameter corresponding to the noisy speech.

[0089] Step S322: input the hidden representation to the first output module, the second output module and the third output module respectively, to obtain the initial human voice detection result output by the first output module, the initial signal-to-noise ratio detection result output by the second output module, and the initial noise amplitude spectrum output by the third output module.

[0090] In an embodiment of the present application, the initial enhancement model may further include a first output module, a second output module and a third output module, which may respectively output an initial human voice detection result, an initial signal-to-noise ratio detection result and an initial noise amplitude spectrum according to the hidden representation of the input.

[0091] In some embodiments, the first output module can be a 2-layer LSTM network, which can output a predicted human voice vector based on the input hidden representation, and use it as the initial human voice detection result. It can be understood that the initial human voice detection result output by the initial enhancement model is the prediction data of whether there is a human voice in the mixed and noisy speech, which is used to characterize the probability of the existence of a human voice in each frame of data in the mixed and noisy speech. After the initial enhancement model is continuously trained to meet the preset conditions, the human voice detection result output by the model can be close to the real data. The second output module can be a 2-layer fully connected layer, which can output a predicted signal-to-noise ratio as the initial signal-to-noise ratio detection result based on the input hidden representation. The third output module can include a 2-layer LSTM network to add timing information to the hidden representation, and can also include a 7-layer CNN decoder structure network to output a predicted noise amplitude spectrum based on the input hidden representation with timing information, and use it as the initial noise amplitude spectrum.

[0092] In step S323: based on the label data annotated by the noisy speech, the initial speech gain parameter, the initial human voice detection result, the initial signal-to-noise ratio detection result, the initial noise amplitude spectrum, and the clean speech samples and noise samples corresponding to the noisy speech, determine the total loss value of the initial enhancement model.

[0093] In an implementation manner of the present application, after obtaining the initial speech gain parameters, initial human voice detection structure, initial signal-to-noise ratio detection results, initial noise amplitude spectrum and other data output by the initial enhancement model, label data corresponding to the input noisy speech can also be obtained, including human voice labels and signal-to-noise ratio labels, as well as data such as clean speech samples and noise samples corresponding to the noisy speech. Based on the above data, the total loss value of the initial enhancement model can be determined to adjust the various parameters in the initial enhancement model based on the total loss value, so that the data corresponding to the noisy speech output by the initial enhancement model is more accurate, thereby realizing constraints on the training of the initial enhancement model.

[0094] In some embodiments, Figure 7 As shown, determining the total loss value of the initial enhancement model in step S323 may include the following steps:

[0095] Step S3231: determining a first loss value based on a difference between an enhanced amplitude spectrum and an amplitude spectrum corresponding to the clean speech sample, wherein the enhanced amplitude spectrum is obtained by enhancing the amplitude spectrum corresponding to the noisy speech based on the initial speech gain parameter.

[0096] In the implementation manner of the present application, the first loss value can be determined by the difference between the amplitude spectrum corresponding to the clean speech sample and the enhanced amplitude spectrum, the enhanced amplitude spectrum refers to the initial speech gain parameter obtained by the initial enhancement model, and the amplitude spectrum of the mixed and noisy speech is enhanced to obtain the enhanced amplitude spectrum. The first loss value can be calculated by the following formula:

[0097] loss gain =mse(mag clean ,mag noisy *out gain )

[0098] Among them, loss gain That is, the first loss value, mag clean Represents the amplitude spectrum corresponding to the clean speech sample, mag noisy *out gain It represents the enhanced amplitude spectrum. It can be seen from the formula that the enhanced amplitude spectrum is composed of the initial speech gain parameter out gain The amplitude spectrum mag corresponding to the mixed noise speech noisy The MSE function refers to the mean square error function, which is used to evaluate the degree of change of data. If the value of the MSE function, that is, the first loss value, is small, it can be shown that the model has good accuracy.

[0099] Step S3232: Determine a second loss value based on the difference between the initial human voice detection result and the human voice label.

[0100] In the implementation mode of the present application, the second loss value can be determined by the initial human voice detection result and the human voice label corresponding to the noisy speech, and the second loss value can be used to characterize whether the initial enhancement model can accurately identify the human voice part data and the non-human voice part data of the input noisy speech. Specifically, the size of the second loss value can be obtained by the following formula:

[0101] loss vad =softmax(label vad ,out vad )

[0102] Among them, loss vad Represents the second loss value, label vad Indicates the human voice label corresponding to the mixed-noise speech, out vad Represents the initial human voice detection result output by the initial enhancement model; the softmax function, that is, the normalized exponential function, can convert a K-dimensional vector containing any real number into another K-dimensional real vector so that the range of each element is between (0, 1) and the sum of all elements is 1.

[0103] Step S3233: Determine a third loss value based on the difference between the initial signal-to-noise ratio detection result and the signal-to-noise ratio label.

[0104] In the implementation mode of the present application, the third loss value can be determined by the initial signal-to-noise ratio detection result and the signal-to-noise ratio label corresponding to the noisy speech. The third loss value can be used to characterize whether the initial enhancement model can accurately identify the corresponding signal-to-noise ratio of the input noisy speech, that is, based on the difference between the initial signal-to-noise ratio detection result output by the initial enhancement model and the actual signal-to-noise ratio label corresponding to the noisy speech, the accuracy of the initial enhancement model in identifying the signal-to-noise ratio of the speech data can be determined. Specifically, the third loss value can be obtained by the formula:

[0105] loss snr =rmse(label snr ,out snr )

[0106] Among them, loss snr That is, the third loss value, label snr Used to indicate the signal-to-noise ratio label corresponding to the mixed-noise speech, out snrThe RMSE function is used to represent the initial signal-to-noise ratio test result output by the initial enhancement model. It is the root mean square error function, also known as the standard error. The function is the square root of the ratio of the square of the deviation between the predicted value and the true value to the number of observations. The root mean square error is used to measure the deviation between the observed value and the true value. The root mean square error function is very sensitive to very large or very small errors in a set of measurements, so it can well reflect the precision of the measurement.

[0107] Step S3234: Determine a fourth loss value based on the difference between the initial noise amplitude spectrum and the amplitude spectrum corresponding to the noise sample.

[0108] In the implementation manner of the present application, the fourth loss value can be determined by the initial noise amplitude spectrum output by the initial enhancement model and the amplitude spectrum corresponding to the noise sample, wherein the initial noise amplitude spectrum output in the initial enhancement model is only used to calculate the error of the initial enhancement model, and is not used to adjust the speech gain parameter output by the model. Specifically, the fourth loss value can be calculated by the following formula:

[0109] loss noise =mse(mag noise ,out noise )

[0110] Among them, loss noise Used to represent the fourth loss value, mag noise Used to represent the amplitude spectrum corresponding to the noise, out noise It is used to represent the initial noise amplitude spectrum output by the initial enhancement model. The MSE function is the mean square error function described in the above steps.

[0111] Step S3235: Determine the total loss value of the initial enhanced model based on the first loss value, the second loss value, the third loss value and the fourth loss value.

[0112] In the implementation of the present application, after obtaining the first loss value, the second loss value, the third loss value and the fourth loss value, the total loss value corresponding to the initial enhancement model can be determined based on the four loss values. Specifically, the sum of the four loss values ​​can be used as the total loss value of the initial enhancement model, as shown in the following formula:

[0113] loss total =loss gain +loss vad +loss snr +loss noise

[0114] Step S324: According to the total loss value, the initial enhancement model is iteratively trained until the initial enhancement model meets a preset condition.

[0115] In the implementation mode of the present application, after obtaining the total loss value corresponding to the initial enhancement model, the parameters in the initial enhancement model can be updated based on the back propagation algorithm until the total loss value meets the preset condition. The preset condition can be that the total loss value is less than the preset value, or that the total loss value does not change after approaching the minimum value, or that the number of training times for the initial enhancement model reaches the preset number of times.

[0116] Step S330: Acquire voice data.

[0117] Step S340: Based on a pre-trained speech enhancement model, a voice detection result, a signal-to-noise ratio detection result, and a speech gain parameter of the speech data are obtained, wherein the voice detection result is used to characterize the data corresponding to the voice part in the speech data, the signal-to-noise ratio detection result is used to characterize the signal-to-noise ratio corresponding to the speech data, and the speech gain parameter is used to enhance the speech data.

[0118] Step S350: adjusting the speech gain parameter based on the human voice detection result and the signal-to-noise ratio detection result.

[0119] Step S360: Performing enhancement processing on the speech data based on the adjusted speech gain parameter to obtain enhanced speech data.

[0120] In the implementation manner of the present application, the trained speech enhancement model can obtain the corresponding speech gain parameter based on the input speech data, and convolve the speech gain parameter with the amplitude spectrum corresponding to the speech data to obtain the enhanced amplitude spectrum, as shown in the following formula:

[0121] mag enhance =mag noisy *out gain

[0122] The enhanced amplitude spectrum can then be combined with the phase spectrum corresponding to the speech data, and the enhanced speech data can be finally obtained through inverse Fourier transformation.

[0123] In the embodiment of the present application, step S330 and step S360 can refer to the contents of other embodiments and will not be repeated here.

[0124] The speech enhancement method provided in the embodiment of the present application obtains multiple segments of noisy speech based on a mixture of clean speech samples and noise samples, so that the human voice label and signal-to-noise ratio label corresponding to the noisy speech are accurate. The initial enhancement model is trained based on a sample speech set consisting of multiple segments of noisy speech, and clean speech samples and noise samples corresponding to the noisy speech, until the loss value of the initial enhancement model meets the preset conditions, and the trained model is used as the speech enhancement model. The human voice detection result and the signal-to-noise ratio detection result corresponding to the speech data are obtained based on the trained speech enhancement model, the speech gain parameters corresponding to the speech data are adjusted, and the speech data is enhanced based on the adjusted speech gain parameters. Dynamic adjustment of speech data is achieved, and the clarity and intelligibility of speech data are improved.

[0125] See also Figure 8 , which shows a structural block diagram of a speech enhancement device 200 provided in an embodiment of the present application, and the speech enhancement device 200 includes: a speech acquisition module 210, a data acquisition module 220, a gain adjustment module 230 and a speech enhancement module 240. Among them, the speech acquisition module 210 is used to acquire speech data; the data acquisition module 220 is used to acquire the human voice detection result, the signal-to-noise ratio detection result and the speech gain parameter of the speech data based on the pre-trained speech enhancement model, the human voice detection result is used to characterize the data corresponding to the human voice part in the speech data, the signal-to-noise ratio detection result is used to characterize the signal-to-noise ratio corresponding to the speech data, and the speech gain parameter is used to enhance the speech data; the gain adjustment module 230 is used to adjust the speech gain parameter based on the human voice detection result and the signal-to-noise ratio detection result; the speech enhancement module 240 is used to enhance the speech data based on the adjusted speech gain parameter to obtain the enhanced speech data.

[0126] As a possible implementation, the gain adjustment module 230 includes a data determination unit and a data adjustment unit. The data determination unit is used to determine the first data and the second data in the speech data based on the human voice detection result and the signal-to-noise ratio detection result, the first data is the data with a human voice ratio greater than a preset ratio, and the second data is the data in the speech data other than the first data; the data adjustment unit is used to reduce the speech gain parameter corresponding to the first data, and to increase the speech gain parameter corresponding to the second data.

[0127] As a possible implementation, the data determination unit can be used to determine, based on the human voice detection result, data corresponding to the human voice part in the speech data as the third data; based on the signal-to-noise ratio detection result, determine from the third data data whose signal-to-noise ratio is greater than a preset signal-to-noise ratio as the first data, and the preset signal-to-noise ratio is set based on a preset ratio; and obtain data other than the first data from the speech data as the second data.

[0128] As a possible implementation, the speech enhancement device may also include: a sample acquisition module and a model training module. The sample acquisition module is used to acquire a sample speech set, the sample speech set includes multiple segments of noisy speech, each segment of noisy speech is annotated with a voice label and a signal-to-noise ratio label, each segment of noisy speech is obtained by mixing clean speech samples and noise samples, the voice label is used to characterize the data corresponding to the voice part of the noisy speech, and the signal-to-noise ratio label is used to characterize the signal-to-noise ratio corresponding to the noisy speech; the model training module is used to train the initial enhancement model according to the sample speech set, and the clean speech samples and noise samples corresponding to the noisy speech, until the initial enhancement model meets the preset conditions, and the trained speech enhancement model is obtained.

[0129] In this embodiment, the initial enhancement model includes a first input module, a first output module, a second output module and a third output module. The model training module can be specifically used to: input the amplitude spectrum corresponding to the noisy speech into the first input module to obtain the hidden representation corresponding to the noisy speech, and the initial speech gain parameter obtained based on the hidden feature; input the hidden feature into the first output module, the second output module and the third output module respectively, and obtain the initial human voice detection result output by the first output module, the initial signal-to-noise ratio detection result output by the second output module, and the initial noise amplitude spectrum output by the third output module; determine the total loss value of the initial enhancement model based on the label data annotated by the noisy speech, the initial speech gain parameter, the initial human voice detection result, the initial signal-to-noise ratio detection result, the initial noise amplitude spectrum, and the clean speech sample and noise sample corresponding to the noisy speech; according to the total loss value, iteratively train the initial speech enhancement model until the initial speech enhancement model meets the preset conditions.

[0130] Optionally, the model training module determines the total loss value of the initial enhancement model based on the label data of the noisy speech annotation, the initial speech gain parameter, the initial human voice detection result, the initial signal-to-noise ratio detection result, the initial noise amplitude spectrum, and the clean speech samples and noise samples corresponding to the noisy speech, including: determining a first loss value based on the difference between the enhanced amplitude spectrum and the amplitude spectrum corresponding to the clean speech sample, the enhanced amplitude spectrum being obtained by enhancing the amplitude spectrum corresponding to the noisy speech based on the initial speech gain parameter; determining a second loss value based on the difference between the initial human voice detection result and the human voice label; determining a third loss value based on the difference between the initial signal-to-noise ratio detection result and the signal-to-noise ratio label; determining a fourth loss value based on the difference between the initial noise amplitude spectrum and the amplitude spectrum corresponding to the noise sample; determining the total loss value of the initial enhancement model based on the first loss value, the second loss value, the third loss value and the fourth loss value.

[0131] In this embodiment, the sample acquisition module can be specifically used to: obtain multiple clean speech samples and multiple noise samples; obtain vocal labels for marking vocal segments and non-voice segments in the clean speech; mix the clean speech and noise according to different signal-to-noise ratios to obtain multiple mixed-noise speech segments; label each mixed-noise speech segment with a signal-to-noise ratio label based on the signal-to-noise ratio used when mixing each mixed-noise speech segment, and label each mixed-noise speech segment with a vocal label based on the vocal label of the clean speech to obtain a sample speech set.

[0132] Optionally, the sample acquisition module may also be specifically used to: perform reverberation processing on part of the noisy speech in the multiple noisy speech segments, and add the noisy speech after the reverberation processing to the sample speech set.

[0133] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0134] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0135] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0136] In summary, in the speech enhancement method provided by the present application, speech data is obtained; based on a pre-trained speech enhancement model, a human voice detection result, a signal-to-noise ratio detection result, and a speech gain parameter of the speech data are obtained, the human voice detection result is used to characterize the data corresponding to the human voice part in the speech data, the signal-to-noise ratio detection result is used to characterize the signal-to-noise ratio corresponding to the speech data, and the speech gain parameter is used to enhance the speech data; based on the human voice detection result and the signal-to-noise ratio detection result, the speech gain parameter is adjusted; based on the adjusted speech gain parameter, the speech data is enhanced to obtain enhanced speech data. This method can achieve dynamic suppression of speech data noise and improve the clarity and intelligibility of speech data by dynamically enhancing speech data.

[0137] Please refer to Fig. 9, which shows a structural block diagram of a computer device 100 provided in an embodiment of the present application. The computer device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more application programs, wherein the one or more application programs may be stored in the memory 120 and configured to be executed by one or more processors 120, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0138] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire computer device, and executes various functions of the computer device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0139] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the computer device during use.

[0140] Please refer to Fig.10, which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 800 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0141] The computer readable storage medium 800 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer readable storage medium 800 has storage space for program code 810 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 810 can be compressed, for example, in an appropriate form.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A speech enhancement method, characterized in that: The method comprises: Get voice data; Based on a pre-trained speech enhancement model, obtaining a human voice detection result, a signal-to-noise ratio detection result, and a speech gain parameter of the speech data, wherein the human voice detection result is used to characterize data corresponding to a human voice part in the speech data, the signal-to-noise ratio detection result is used to characterize a signal-to-noise ratio corresponding to the speech data, and the speech gain parameter is used to enhance the speech data; Adjusting the speech gain parameter based on the human voice detection result and the signal-to-noise ratio detection result; Performing enhancement processing on the speech data based on the adjusted speech gain parameter to obtain enhanced speech data; The speech enhancement model is obtained by iteratively training the initial enhancement model, the initial enhancement model includes a first input module, a first output module, a second output module and a third output module, and the speech enhancement model is trained by the following steps: Acquire a sample speech set, the sample speech set comprising a plurality of mixed-noise speech segments, each of the mixed-noise speech segments being annotated with a voice label and a signal-to-noise ratio label, each of the mixed-noise speech segments being obtained by mixing a clean speech sample and a noise sample, the voice label being used to characterize data corresponding to a voice part in the mixed-noise speech, and the signal-to-noise ratio label being used to characterize a signal-to-noise ratio corresponding to the mixed-noise speech; Inputting the amplitude spectrum corresponding to the noisy speech into the first input module to obtain the hidden representation corresponding to the noisy speech and the initial speech gain parameter obtained based on the hidden feature; Inputting the hidden representation to the first output module, the second output module and the third output module respectively, obtaining an initial human voice detection result output by the first output module, an initial signal-to-noise ratio detection result output by the second output module, and an initial noise amplitude spectrum output by the third output module; Determine the total loss value of the initial enhancement model based on the label data annotated by the noisy speech, the initial speech gain parameter, the initial human voice detection result, the initial signal-to-noise ratio detection result, the initial noise amplitude spectrum, and the clean speech samples and noise samples corresponding to the noisy speech; According to the total loss value, the initial enhancement model is iteratively trained until the initial enhancement model meets a preset condition, thereby obtaining a trained speech enhancement model.

2. The method according to claim 1, characterized in that The adjusting the speech gain parameter based on the human voice detection result and the signal-to-noise ratio detection result includes: Based on the human voice detection result and the signal-to-noise ratio detection result, determine first data and second data in the voice data, wherein the first data is data in which the proportion of human voice is greater than a preset proportion, and the second data is data in the voice data other than the first data; The speech gain parameter corresponding to the first data is reduced, and the speech gain parameter corresponding to the second data is increased.

3. The method according to claim 2, characterized in that The determining, based on the human voice detection result and the signal-to-noise ratio detection result, the first data and the second data in the speech data comprises: Based on the human voice detection result, determine data corresponding to the human voice part in the speech data as third data; Based on the signal-to-noise ratio detection result, determine data having a signal-to-noise ratio greater than a preset signal-to-noise ratio from the third data as the first data, wherein the preset signal-to-noise ratio is set based on the preset ratio; Data other than the first data is acquired from the voice data as the second data.

4. The method according to claim 1, characterized in that: The method of determining the total loss value of the initial enhancement model based on the label data annotated by the noisy speech, the initial speech gain parameter, the initial human voice detection result, the initial signal-to-noise ratio detection result, the initial noise amplitude spectrum, and the clean speech samples and noise samples corresponding to the noisy speech comprises: Determining a first loss value based on a difference between an enhanced amplitude spectrum and an amplitude spectrum corresponding to the clean speech sample, wherein the enhanced amplitude spectrum is obtained by enhancing the amplitude spectrum corresponding to the noisy speech based on the initial speech gain parameter; Determining a second loss value based on a difference between the initial human voice detection result and the human voice label; Determining a third loss value based on a difference between the initial signal-to-noise ratio detection result and the signal-to-noise ratio label; determining a fourth loss value based on a difference between the initial noise amplitude spectrum and the amplitude spectrum corresponding to the noise sample; Based on the first loss value, the second loss value, the third loss value and the fourth loss value, a total loss value of the initial enhancement model is determined.

5. The method according to claim 1, characterized in that The obtaining of the sample speech set comprises: Obtain multiple clean speech samples and multiple noise samples; Obtaining vocal tags for marking vocal segments and non-vocal segments in the clean speech; Mixing the clean speech and the noise according to different signal-to-noise ratios to obtain multiple segments of the mixed-noise speech; Based on the signal-to-noise ratio used when mixing each section of the noisy speech, a signal-to-noise ratio label is marked for each section of the noisy speech, and based on the voice label of the clean speech, a voice label is marked for each section of the noisy speech to obtain the sample speech set.

6. The method according to claim 5, characterized in that Before the initial enhancement model is trained according to the sample speech set and the clean speech samples and noise samples corresponding to the mixed-noise speech until the initial enhancement model meets the preset conditions and the trained speech enhancement model is obtained, the method further includes: Perform reverberation processing on some of the multiple noisy speech segments, and add the noisy speech after the reverberation processing to the sample speech set.

7. A speech enhancement device, characterized in that: The device comprises: a voice acquisition module, a data acquisition module, a gain adjustment module and a voice enhancement module, wherein: The voice acquisition module is used to acquire voice data; The data acquisition module is used to acquire a human voice detection result, a signal-to-noise ratio detection result, and a speech gain parameter of the speech data based on a pre-trained speech enhancement model, wherein the human voice detection result is used to characterize the data corresponding to the human voice part in the speech data, the signal-to-noise ratio detection result is used to characterize the signal-to-noise ratio corresponding to the speech data, and the speech gain parameter is used to enhance the speech data; The gain adjustment module is used to adjust the speech gain parameter based on the human voice detection result and the signal-to-noise ratio detection result; The speech enhancement module is used to perform enhancement processing on the speech data based on the adjusted speech gain parameter to obtain enhanced speech data; The speech enhancement model is obtained by iteratively training an initial enhancement model, the initial enhancement model includes a first input module, a first output module, a second output module and a third output module, and the speech enhancement model is trained by the following steps: obtaining a sample speech set, the sample speech set includes multiple sections of noisy speech, each section of the noisy speech is annotated with a voice label and a signal-to-noise ratio label, each section of the noisy speech is obtained by mixing clean speech samples and noise samples, the voice label is used to characterize the data corresponding to the voice part of the noisy speech, and the signal-to-noise ratio label is used to characterize the signal-to-noise ratio corresponding to the noisy speech; inputting the amplitude spectrum corresponding to the noisy speech into the first input module to obtain the hidden representation corresponding to the noisy speech, and based on the hidden feature The method comprises the following steps: inputting the hidden representation into the first output module, the second output module and the third output module respectively to obtain the initial human voice detection result output by the first output module, the initial signal-to-noise ratio detection result output by the second output module, and the initial noise amplitude spectrum output by the third output module; determining the total loss value of the initial enhancement model based on the label data annotated by the noisy speech, the initial speech gain parameter, the initial human voice detection result, the initial signal-to-noise ratio detection result, the initial noise amplitude spectrum, and the clean speech samples and noise samples corresponding to the noisy speech; iteratively training the initial enhancement model according to the total loss value until the initial enhancement model meets the preset conditions, thereby obtaining the trained speech enhancement model.

8. A computer device, characterized in that: The computer device comprises: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 6.

Images