Speech enhancement method and device

Abstract

The present disclosure relates to a speech enhancement method and device. The speech enhancement method comprises: obtaining a speech signal containing noise to be processed; dividing the speech amplitude spectrum of the speech signal containing noise to be processed into a broadband amplitude spectrum and a high-frequency amplitude spectrum; performing broadband enhancement processing on the broadband amplitude spectrum to obtain a broadband enhanced amplitude spectrum; performing high-frequency enhancement processing on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum to obtain a high-frequency enhanced amplitude spectrum; fusing the broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum, and obtaining a full-band enhanced speech signal of the speech signal containing noise to be processed according to the fusion result. The present disclosure solves the problem that full-band speech enhancement processing is not compatible with broadband speech enhancement processing.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a speech enhancement method and device. Background Art

[0002] With the development of 5G networks, the transmission rate and bandwidth of the network have been greatly improved, which also makes high-quality real-time audio and video communication possible. In order to meet users' pursuit of high sound quality, the audio sampling rate has been expanded from 16kHz (wideband) to 32kHz (ultra-wideband) and even 48kHz (full-band). Therefore, a full-band speech enhancement system is deployed for full-band speech in related technologies.

[0003] However, in the related technology, a full-band speech enhancement system is deployed for full-band speech, and the deployed full-band speech enhancement system is relatively independent of the deployed broadband speech enhancement system. Such a deployment method has the problem that the full-band speech enhancement processing is incompatible with the broadband speech enhancement processing. Summary of the invention

[0004] The present disclosure provides a speech enhancement method and apparatus to at least solve the problem that full-band speech enhancement processing is incompatible with broadband speech enhancement processing.

[0005] According to one aspect of an embodiment of the present disclosure, a speech enhancement method is provided, the speech enhancement method comprising:

[0006] Acquire a noisy speech signal to be processed;

[0007] Dividing the speech amplitude spectrum of the speech signal containing the noise into a broadband amplitude spectrum and a high-frequency amplitude spectrum;

[0008] Performing broadband enhancement processing on the broadband amplitude spectrum to obtain a broadband enhanced amplitude spectrum;

[0009] Performing high-frequency enhancement processing on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum to obtain a high-frequency enhanced amplitude spectrum;

[0010] The broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum are fused, and a full-band enhanced speech signal of the noisy speech signal to be processed is obtained according to the fusion result.

[0011] In one possible implementation,

[0012] The performing high-frequency enhancement processing on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum comprises:

[0013] Performing high-frequency enhancement processing on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum using a high-frequency enhancement model;

[0014] The training method of the high-frequency enhancement model includes:

[0015] Acquire first training samples of a plurality of speakers and corresponding sample reference high-frequency amplitude spectra, wherein the first training samples include a sample broadband enhanced amplitude spectrum and a sample high-frequency amplitude spectrum of each speaker; the sample broadband enhanced amplitude spectrum and the sample high-frequency amplitude spectrum are determined based on the full-band speech amplitude spectrum of the noisy speech signal of the corresponding speaker, and the sample reference high-frequency amplitude spectrum is determined based on the clean speech signal corresponding to each speaker;

[0016] Using the high-frequency enhancement model, high-frequency enhancement processing is performed on the sample high-frequency amplitude spectrum and the sample broadband enhanced amplitude spectrum of each speaker to obtain the sample high-frequency enhanced amplitude spectrum of each speaker;

[0017] The high frequency enhancement model is trained based on the difference between the sample high frequency enhancement amplitude spectrum and the sample reference high frequency amplitude spectrum of each speaker.

[0018] In one possible implementation,

[0019] The performing broadband enhancement processing on the broadband amplitude spectrum comprises:

[0020] Performing broadband enhancement processing on the broadband amplitude spectrum using a broadband enhancement model;

[0021] The training method of the broadband enhancement model includes:

[0022] Acquire second training samples of a plurality of speakers and corresponding sample reference broadband amplitude spectra, wherein the second training samples include a sample broadband amplitude spectrum of each speaker;

[0023] Performing broadband enhancement processing on the sample broadband amplitude spectrum using the broadband enhancement model to obtain a sample broadband enhanced amplitude spectrum of each of the speakers;

[0024] The broadband enhancement model is trained based on the difference between the sample reference broadband amplitude spectrum and the sample broadband enhancement amplitude spectrum of each speaker.

[0025] In one possible implementation,

[0026] The broadband enhancement model includes: a first frequency domain enhancement model, a second frequency domain enhancement model and a fusion model, wherein the first frequency domain enhancement model and the second frequency domain enhancement model are trained models; the first frequency domain enhancement model is established based on a temporal convolutional network; the second frequency domain enhancement model is established based on a convolutional recurrent network;

[0027] The step of performing broadband enhancement processing on the sample broadband amplitude spectrum by using the broadband enhancement model to obtain a sample broadband enhanced amplitude spectrum of each speaker includes:

[0028] Using the first frequency domain enhancement model and the second frequency domain enhancement model to perform frequency domain enhancement processing on the sample broadband amplitude spectrum respectively, to obtain a sample first frequency domain enhanced amplitude spectrum and a sample second frequency domain enhanced amplitude spectrum of each speaker;

[0029] The fusion model is used to perform fusion processing on the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum to obtain the sample broadband enhanced amplitude spectrum.

[0030] In one possible implementation,

[0031] The first frequency domain enhancement model is trained based on a first frequency domain enhancement loss function;

[0032] The first frequency domain enhancement loss function is determined based on the sample first frequency domain enhancement amplitude spectrum of each speaker, and the corresponding sample reference broadband amplitude spectrum, sample reference time domain signal and sample first enhanced time domain signal, wherein the sample first enhanced time domain signal is determined based on the sample first frequency domain enhancement amplitude spectrum, and the sample reference time domain signal is determined based on the corresponding clean speech signal;

[0033] The second frequency domain enhancement model is trained based on a second frequency domain enhancement loss function;

[0034] The second frequency domain enhancement loss function is determined based on the sample second frequency domain enhancement amplitude spectrum of each speaking object, and the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample second enhanced time domain signal; wherein the sample second enhanced time domain signal is determined based on the sample second frequency domain enhancement amplitude spectrum.

[0035] In one possible implementation,

[0036] The training of the broadband enhancement model based on the difference between the sample reference broadband amplitude spectrum and the sample broadband enhancement amplitude spectrum of each speaker includes:

[0037] determining a fusion enhancement loss function for characterizing the difference between the sample reference wideband magnitude spectrum and the sample wideband enhanced magnitude spectrum for each of the speaking objects;

[0038] Training the fusion model based on the fusion enhancement loss function;

[0039] Wherein, the fusion enhancement loss function is determined based on the sample broadband enhanced amplitude spectrum of each speaking object, and the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample third enhanced time domain signal; wherein, the sample third enhanced time domain signal is determined based on the sample broadband enhanced amplitude spectrum, and the sample reference time domain signal is determined based on the corresponding pure speech signal.

[0040] In one possible implementation,

[0041] The step of using the fusion model to fuse the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum to obtain the sample broadband enhanced amplitude spectrum includes:

[0042] Dividing the sample wideband amplitude spectrum, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum into a plurality of sub-bands;

[0043] Based on the sample wideband amplitude spectrum, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum, determine the subband weights of the subbands corresponding to each of the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum; wherein the sum of the subband weights of the same subband corresponding to the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum is 1;

[0044] Performing smoothing processing on each of the sub-band weights in the time dimension;

[0045] The sample broadband enhanced amplitude spectrum is determined based on the subband weight of each smoothed subband and the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum respectively corresponding to a plurality of the subbands.

[0046] In one possible implementation,

[0047] The step of fusing the broadband enhanced amplitude spectrum and the high frequency enhanced amplitude spectrum to obtain a full-band enhanced speech signal of the noisy speech signal to be processed according to the fusion result includes:

[0048] Acquire a sample potential speech broadband enhanced magnitude spectrum of a target speaker based on the broadband enhanced magnitude spectrum;

[0049] The latent speech wideband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum of the target speaker are fused, and a full-band enhanced speech signal of the noisy speech signal to be processed is obtained according to the fusion result.

[0050] In one possible implementation,

[0051] The method of obtaining a sample potential speech broadband enhanced amplitude spectrum of a target speaker based on the broadband enhanced amplitude spectrum to obtain calibration speech signals of a plurality of calibration speakers;

[0052] Determining the target speaking object among the plurality of calibrated speaking objects;

[0053] Determine the calibration voice signal of the target speaking object according to the calibration voice signal of the calibration speaking object determined as the target speaking object;

[0054] Amplitude spectrum extraction processing is performed on the calibrated speech signal and the broadband enhanced amplitude spectrum of the target speaker to obtain a sample potential speech broadband enhanced amplitude spectrum of the target speaker.

[0055] In one possible implementation,

[0056] The performing amplitude spectrum extraction processing on the calibration speech signal and the broadband enhanced amplitude spectrum of the target speaker comprises:

[0057] Performing amplitude spectrum extraction processing on the calibration speech signal and the broadband enhanced amplitude spectrum of the target speaker using a speaker amplitude spectrum extraction model;

[0058] The training method of the speaker amplitude spectrum extraction model includes:

[0059] Acquire a plurality of third training samples of the calibrated speaking objects and sample reference wideband amplitude spectra; wherein the third training samples include a sample wideband enhanced amplitude spectrum and a sample speech signal of each of the calibrated speaking objects;

[0060] Using the speaker amplitude spectrum extraction model, performing amplitude spectrum extraction processing on the sample speech signal and the sample broadband enhanced amplitude spectrum of each calibrated speaker, to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker;

[0061] The training of the speaking object amplitude spectrum extraction model is achieved based on the difference between the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaking object and the corresponding sample reference broadband amplitude spectrum.

[0062] In one possible implementation,

[0063] The speaker amplitude spectrum extraction model includes: a speaker voiceprint extraction model and a calibration speaker extraction model; the speaker voiceprint extraction model is a model that has been preliminarily trained;

[0064] The method of using the speaker amplitude spectrum extraction model to perform amplitude spectrum extraction processing on the sample speech signal and the sample broadband enhanced amplitude spectrum of each calibrated speaker to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker includes:

[0065] Using the speaker voiceprint extraction model, encoding the sample speech signal of each calibrated speaker to obtain the speaker voiceprint of each calibrated speaker;

[0066] The calibration speaking object extraction model is used to perform amplitude spectrum extraction processing on the speaking object voiceprint and the broadband enhanced amplitude spectrum of each calibration speaking object to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibration speaking object.

[0067] In one possible implementation,

[0068] The speaker voiceprint extraction model is trained based on a first cross entropy loss function;

[0069] Among them, the first cross entropy loss function is determined based on the identity prediction probability corresponding to each of the speaker's voiceprints and the label of the real identity of the speaker corresponding to each of the speaker's voiceprints, and the identity prediction probability is obtained by using a classifier to perform speaker identity discrimination processing on multiple speaker voiceprints of each of the calibrated speakers; the classifier is trained based on the first cross entropy loss function.

[0070] In one possible implementation,

[0071] The difference between the sample potential speech broadband enhanced amplitude spectrum of each of the calibrated speaking objects and the corresponding sample reference broadband amplitude spectrum is characterized by a second cross entropy loss function and a calibrated speaking object extraction loss function;

[0072] The calibration speaking object extraction loss function is determined based on the sample enhanced time domain signal, the sample reference time domain signal, the sample potential speech wideband enhanced amplitude spectrum and the corresponding sample reference wideband amplitude spectrum, the sample wideband amplitude spectrum, and the trained classifier; wherein the sample enhanced time domain signal is determined according to the sample potential speech wideband enhanced amplitude spectrum;

[0073] The second cross entropy loss function is determined based on the identity prediction probability corresponding to each of the speaker's voiceprints and the label of the speaker's true identity corresponding to each of the speaker's voiceprints. The identity prediction probability is obtained by using a classifier to perform speaker identity discrimination processing on multiple speaker voiceprints of each of the calibrated speakers.

[0074] According to another aspect of an embodiment of the present disclosure, a speech enhancement device is provided, the speech enhancement device comprising:

[0075] An acquisition unit, configured to acquire a speech signal containing noise to be processed;

[0076] A dividing unit, configured to divide the speech amplitude spectrum of the noisy speech signal to be processed into a broadband amplitude spectrum and a high-frequency amplitude spectrum;

[0077] A first processing unit is configured to perform broadband enhancement processing on the broadband amplitude spectrum to obtain a broadband enhanced amplitude spectrum;

[0078] A second processing unit is configured to perform high-frequency enhancement processing on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum to obtain a high-frequency enhanced amplitude spectrum;

[0079] The third processing unit is configured to perform fusion of the broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum, and obtain a full-band enhanced speech signal of the noisy speech signal to be processed according to the fusion result.

[0080] According to another aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0081] processor;

[0082] a memory for storing executable instructions for the processor;

[0083] The processor is configured to execute the executable instructions to implement the speech enhancement method described in any of the above embodiments.

[0084] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When at least one instruction in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device can implement the speech enhancement method described in any of the above embodiments.

[0085] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the speech enhancement method described in any of the above embodiments is implemented.

[0086] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0087] The speech amplitude spectrum of a speech signal containing noise is divided into a high-frequency amplitude spectrum and a wideband amplitude spectrum; the wideband amplitude spectrum is subjected to wideband enhancement processing to obtain a wideband enhanced amplitude spectrum; the obtained wideband enhanced amplitude spectrum and high-frequency amplitude spectrum are subjected to high-frequency enhancement processing to obtain a high-frequency enhanced amplitude spectrum, and then the high-frequency enhanced amplitude spectrum and the wideband enhanced amplitude spectrum are fused to obtain a full-band enhanced speech signal of the speech signal containing noise. In the disclosed embodiment, the acquisition process of the full-band enhanced speech signal includes the acquisition of the wideband enhanced speech signal, that is, it is compatible with the processing of wideband speech enhancement. The speech enhancement method solves the problem that the full-band speech enhancement processing is not compatible with the wideband speech enhancement processing.

[0088] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0090] Figure 1 is a schematic diagram of an implementation environment of a speech enhancement method according to an exemplary embodiment;

[0091] Figure 2 is a schematic diagram of a speech enhancement method according to an exemplary embodiment;

[0092] Figure 3 is a schematic diagram of a model-based speech enhancement method according to an exemplary embodiment;

[0093] Figure 4 is a schematic diagram of a speech enhancement model training structure according to an exemplary embodiment;

[0094] Figure 5 is a schematic diagram of a speech enhancement model structure according to an exemplary embodiment;

[0095] Figure 6 is a schematic diagram of a broadband enhancement training model structure according to an exemplary embodiment;

[0096] Figure 7 is a schematic diagram of a broadband enhancement model structure according to an exemplary embodiment;

[0097] Figure 8 is a schematic diagram of a personalized speech enhancement method according to an exemplary embodiment;

[0098] Fig. 9is a schematic diagram of a personalized speech enhancement method based on calibrating a speaking object according to an exemplary embodiment;

[0099] Fig.10 is a schematic diagram of a model-based personalized speech enhancement method according to an exemplary embodiment;

[0100] Fig.11 is a schematic diagram of a personalized speech enhancement model training structure according to an exemplary embodiment;

[0101] Fig.12 is a schematic diagram of a personalized speech enhancement model structure according to an exemplary embodiment;

[0102] Fig.13 is a schematic diagram of a training structure of a speaking object amplitude spectrum extraction model according to an exemplary embodiment;

[0103] Fig.14 is a schematic diagram of a structure of a speaker amplitude spectrum extraction model according to an exemplary embodiment;

[0104] Fig.15 is a schematic diagram of a personalized multi-model speech enhancement model training structure according to an exemplary embodiment;

[0105] Fig.16 is a schematic diagram of a personalized multi-model speech enhancement model structure according to an exemplary embodiment;

[0106] Fig.17 is a logical structure block diagram of a speech enhancement device according to an exemplary embodiment;

[0107] Fig.18 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0108] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0109] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0110] With the development of 5G networks, the transmission rate and bandwidth of the network have been greatly improved, which also makes high-quality real-time audio and video communication possible. In order to meet users' pursuit of high sound quality, the audio sampling rate has been expanded from 16kHz (wideband) to 32kHz (ultra-wideband) and even 48kHz (full-band). Therefore, a full-band speech enhancement system is deployed for full-band speech in related technologies.

[0111] The full-band speech enhancement system of related technologies mostly adopts the deep neural network method. In the training stage, the noisy speech and its corresponding clean speech can be obtained by the data simulation method of adding noise and reverberation. By adopting a supervised training method, we can obtain a trained deep neural network model. In the inference stage, the trained model takes the real-time collected speech data as input and filters out the reverberation and environmental noise.

[0112] Based on the above-mentioned implementation method of the full-band speech enhancement system, the full-band speech enhancement system and the broadband speech enhancement system are relatively independent. There is a problem that the full-band speech enhancement technology is not compatible with the related processing of the broadband speech backend, such as the broadband speech recognition system, and the full-band speech enhancement system cannot process broadband audio input. This results in a huge cost and price for maintaining multiple systems.

[0113] In view of this, a speech enhancement method is proposed in the embodiment of the present disclosure. Based on the frequency band processing technology, the noisy speech signal to be processed is divided into a high-frequency part and a broadband part, and the full-band speech signal is obtained by fusing the enhancement processing results of the two parts. The full-band speech enhancement processing is compatible with the broadband speech enhancement processing, that is, the processing of the broadband part; it solves the problem that the full-band speech enhancement system and the broadband speech enhancement system are relatively independent, and there is a problem that the full-band speech enhancement technology is not compatible with the related processing of the broadband speech backend.

[0114] And the implementation method of speech enhancement processing using the model is given. The broadband part is processed by the broadband enhancement model to obtain the broadband enhancement result, the broadband enhancement result and the high-frequency part are input into the high-frequency enhancement model to obtain the high-frequency enhancement result, and the broadband enhancement result and the high-frequency enhancement result are fused to finally realize full-band speech enhancement. The full-band speech enhancement system can be established on the basis of compatibility with the broadband speech enhancement system, thereby solving the problem that the full-band speech enhancement technology is not compatible with the related processing of the broadband speech back-end.

[0115] Figure 1 is a schematic diagram of an implementation environment of a speech enhancement method according to an exemplary embodiment. Figure 1 , the implementation environment may include a server 101 and a terminal 102, which will be described in detail below.

[0116] In an optional embodiment, the server 101 may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers. The terminal 102 may be one or more, and the terminal 102 is not limited to mobile phones, personal computers and other devices.

[0117] The terminal 102 may be used to send a speech signal containing noise to be processed to the server 101 .

[0118] In response to the noisy speech signal to be processed sent by the terminal 102, the server 101 starts to perform speech enhancement processing on the speech signal; during the processing, the speech amplitude spectrum of the noisy speech signal is first divided into a broadband amplitude spectrum and a high-frequency amplitude spectrum; the broadband amplitude spectrum is subjected to broadband enhancement processing to obtain a broadband enhanced amplitude spectrum; the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum are subjected to high-frequency enhancement processing to obtain a high-frequency enhanced amplitude spectrum; the broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum are fused, and the full-band enhanced speech signal of the noisy speech signal to be processed is obtained according to the fusion result.

[0119] While performing speech enhancement processing to obtain a full-band speech enhancement signal, the server 101 can also obtain a broadband enhancement amplitude spectrum. When responding to the full-band speech enhancement signal to the client 102, it can determine whether to respond to the broadband enhancement amplitude spectrum based on the client's needs. During specific use, it can be set according to the actual application scenario.

[0120] The local server 101 may be directly or indirectly connected to the server 102 via wired or wireless communication, which is not limited in the embodiments of the present disclosure.

[0121] Before processing the noisy speech signal to be processed, the server 101 may also train a broadband enhancement model, a high-frequency enhancement model, and a speaker amplitude spectrum extraction model to perform speech enhancement processing.

[0122] Figure 2 FIG. 1 is a schematic diagram of a speech enhancement method according to an exemplary embodiment. Figure 2 As shown, the speech enhancement method is applied to a computer device, and the following description is given by taking the computer device as a server as an example.

[0123] In step 201, a noisy speech signal to be processed is obtained.

[0124] In step 202, the speech amplitude spectrum of the noisy speech signal to be processed is divided into a broadband amplitude spectrum and a high-frequency amplitude spectrum.

[0125] In step 203, the broadband amplitude spectrum is subjected to broadband enhancement processing to obtain a broadband enhanced amplitude spectrum.

[0126] In step 204, high-frequency enhancement processing is performed on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum to obtain a high-frequency enhanced amplitude spectrum.

[0127] In step 205, the broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum are fused, and a full-band enhanced speech signal of the noisy speech signal to be processed is obtained according to the fusion result.

[0128] The broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum are fused, and a full-band enhanced speech signal of a noisy speech signal to be processed is obtained according to the fusion result, including:

[0129] The wideband enhanced amplitude spectrum and the high frequency enhanced amplitude spectrum are combined, post-processed and inverse Fourier transformed to obtain a full-band enhanced speech signal of the noisy speech signal to be processed.

[0130] The merging, post-processing and inverse Fourier transform processing of the broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum can be expressed by the following formula, but is not limited to the following implementation method:

[0131]

[0132] Among them, iSTFT represents inverse short time Fourier transform, post(·) represents post-processing, and concat(·) represents spectrum merging operation. is the broadband enhanced amplitude spectrum, is the high frequency enhanced amplitude spectrum.

[0133] In this embodiment, the speech amplitude spectrum of the noisy speech signal is divided into a high-frequency amplitude spectrum and a wideband amplitude spectrum; the wideband amplitude spectrum is subjected to wideband enhancement processing to obtain a wideband enhanced amplitude spectrum; the obtained wideband enhanced amplitude spectrum and high-frequency amplitude spectrum are subjected to high-frequency enhancement processing to obtain a high-frequency enhanced amplitude spectrum, and then the high-frequency enhanced amplitude spectrum and the wideband enhanced amplitude spectrum are fused to obtain a full-band enhanced speech signal of the noisy speech signal. In the disclosed embodiment, the acquisition process of the full-band enhanced speech signal includes the acquisition of the wideband enhanced speech signal, that is, it is compatible with the processing of wideband speech enhancement. This speech enhancement method solves the problem that the full-band speech enhancement processing is not compatible with the wideband speech enhancement processing.

[0134] Figure 3 is a schematic diagram of a model-based speech enhancement method according to an exemplary embodiment. Figure 3 As shown, the speech enhancement method is applied to a computer device, and the following description is given by taking the computer device as a server as an example.

[0135] In step 301, a noisy speech signal to be processed is obtained.

[0136] In step 302, the speech amplitude spectrum of the noisy speech signal to be processed is divided into a broadband amplitude spectrum and a high-frequency amplitude spectrum.

[0137] In step 303, a broadband enhancement model is used to perform broadband enhancement processing on the broadband amplitude spectrum to obtain a broadband enhanced amplitude spectrum.

[0138] In step 304, a high-frequency enhancement model is used to perform high-frequency enhancement processing on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum.

[0139] In step 305, the broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum are fused, and a full-band enhanced speech signal of the noisy speech signal to be processed is obtained according to the fusion result.

[0140] The broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum are fused, and a full-band enhanced speech signal of a noisy speech signal to be processed is obtained according to the fusion result, including:

[0141] The wideband enhanced amplitude spectrum and the high frequency enhanced amplitude spectrum are combined, post-processed and inverse Fourier transformed to obtain a full-band enhanced speech signal of the noisy speech signal to be processed.

[0142] The merging, post-processing and inverse Fourier transform processing of the broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum can be expressed by the following formula, but is not limited to the following implementation method:

[0143]

[0144] Among them, iSTFT represents inverse short time Fourier transform, post(·) represents post-processing, and concat(·) represents spectrum merging operation. is the broadband enhanced amplitude spectrum, is the high frequency enhanced amplitude spectrum.

[0145] In this embodiment, the speech amplitude spectrum of the noisy speech signal is divided into a high-frequency amplitude spectrum and a wideband amplitude spectrum; the wideband amplitude spectrum is subjected to wideband enhancement processing using a wideband enhancement model to obtain a wideband enhanced amplitude spectrum; the obtained wideband enhanced amplitude spectrum and high-frequency amplitude spectrum are subjected to high-frequency enhancement processing using a high-frequency enhancement model to obtain a high-frequency enhanced amplitude spectrum, and then the high-frequency enhanced amplitude spectrum and the wideband enhanced amplitude spectrum are fused to obtain a full-band enhanced speech signal of the noisy speech signal. In the disclosed embodiment, the process of obtaining the full-band enhanced speech signal includes obtaining the wideband enhanced speech signal using a wideband enhancement model, that is, it is compatible with the processing of wideband speech enhancement. The output and high-frequency part of the wideband enhancement model are used as the input of the high-frequency enhancement model, and finally full-band speech enhancement is achieved, which solves the problem that the full-band speech enhancement system and the wideband speech enhancement system are relatively independent, and there is a problem that the full-band speech enhancement technology is not compatible with the related processing of the wideband speech backend.

[0146] Among them, the training method of the high-frequency enhancement model used includes:

[0147] Acquire first training samples of a plurality of speakers and corresponding sample reference high-frequency amplitude spectra, wherein the first training samples include a sample broadband enhanced amplitude spectrum and a sample high-frequency amplitude spectrum of each speaker; the sample broadband enhanced amplitude spectrum and the sample high-frequency amplitude spectrum are determined based on the full-band speech amplitude spectrum of the noisy speech signal of the corresponding speaker, and the sample reference high-frequency amplitude spectrum is determined based on the pure speech signal corresponding to each speaker;

[0148] Using the high-frequency enhancement model, high-frequency enhancement processing is performed on the sample high-frequency amplitude spectrum and the sample broadband enhanced amplitude spectrum of each speaker to obtain the sample high-frequency enhanced amplitude spectrum of each speaker;

[0149] The high-frequency enhancement model is trained based on the difference between the sample high-frequency enhancement amplitude spectrum of each speaker and the sample reference high-frequency amplitude spectrum.

[0150] In some examples,

[0151] Based on the difference between the sample high-frequency enhancement amplitude spectrum of each speaker and the sample reference high-frequency amplitude spectrum, it is represented by a high-frequency enhancement loss function;

[0152] Among them, the specific calculation method of the high-frequency enhancement loss function can be:

[0153] determining a fourth magnitude spectrum loss between the sample high frequency enhanced magnitude spectrum and the sample reference high frequency magnitude spectrum;

[0154] Determining a fourth loss weight corresponding to a fourth amplitude spectrum loss according to a sample reference high-frequency amplitude spectrum and a corresponding sample high-frequency amplitude spectrum;

[0155] A value of a high frequency enhancement loss function is determined according to the fourth amplitude spectrum loss and the fourth loss weight.

[0156] The training method of the broadband enhancement model used includes:

[0157] Acquire second training samples of a plurality of speakers and corresponding sample reference broadband amplitude spectra, wherein the second training samples include a sample broadband amplitude spectrum of each speaker;

[0158] Performing broadband enhancement processing on the sample broadband amplitude spectrum using the broadband enhancement model to obtain the sample broadband enhanced amplitude spectrum of each speaker;

[0159] The broadband enhancement model is trained based on the difference between the sample reference broadband amplitude spectrum and the sample broadband enhancement amplitude spectrum of each speaker.

[0160] Figure 4 It is a schematic diagram of a speech enhancement model training structure according to an illustrative embodiment. Figure 4 The high-frequency enhancement model and the broadband enhancement model are shown in FIG. After the broadband enhancement model is trained, the high-frequency enhancement model is trained.

[0161] The first training sample used in training the broadband enhancement model and the second training sample used in training the high-frequency enhancement model may be the same training sample or different training samples. Figure 4 In the example, the same training samples are collectively referred to as training samples.

[0162] The specific training process is as follows:

[0163] The first step is to obtain training samples of multiple speakers, wherein the training samples include a sample wideband amplitude spectrum and a sample high-frequency amplitude spectrum of each speaker.

[0164] The sample wideband amplitude spectrum and the sample high-frequency amplitude spectrum are determined based on the full-band speech amplitude spectrum of the noisy speech signal of the corresponding speaker.

[0165] The second step is to use the broadband enhancement model to perform broadband speech enhancement processing on the sample broadband amplitude spectrum to obtain the sample broadband enhanced amplitude spectrum of each speaker.

[0166] The third step is to determine the value of the broadband enhancement loss function based on the sample reference broadband amplitude spectrum and the sample broadband enhancement amplitude spectrum of each speaker, wherein the sample reference broadband amplitude spectrum of each speaker is determined based on the pure speech signal of the corresponding speaker.

[0167] The fourth step is to adjust the parameters of the broadband enhancement model based on the value of the broadband enhancement loss function to achieve training of the broadband enhancement model.

[0168] The fifth step is to use the high-frequency enhancement model to perform high-frequency speech enhancement processing on the sample high-frequency amplitude spectrum of each speaker and the sample broadband enhanced amplitude spectrum obtained after the sample broadband speech enhancement model training is completed, so as to obtain the sample high-frequency enhanced amplitude spectrum of each speaker.

[0169] Step 6: Determine the value of the high-frequency enhancement loss function based on the sample high-frequency enhancement amplitude spectrum and the sample reference high-frequency amplitude spectrum of each speaker, wherein the sample reference high-frequency amplitude spectrum of each speaker is determined based on the pure speech signal of the corresponding speaker.

[0170] Step 7: Adjust the parameters of the high-frequency enhancement model based on the value of the high-frequency enhancement loss function to achieve training of the high-frequency enhancement model.

[0171] At this point, the training of the broadband enhancement model and the high-frequency enhancement model has been completed, and the trained speech enhancement model can output a high-frequency enhancement amplitude spectrum and a broadband enhancement amplitude spectrum.

[0172] Figure 5 The figure is a schematic diagram of a speech enhancement model structure according to an illustrative embodiment. Figure 5 is based on Figure 4 The trained model is based on Figure 5 The model shown can perform speech enhancement processing as follows:

[0173] Inputting the broadband amplitude spectrum of the noisy speech signal to be processed into the broadband enhancement model to obtain the broadband enhanced amplitude spectrum;

[0174] Inputting the high-frequency amplitude spectrum of the noisy speech signal to be processed and the obtained broadband enhanced amplitude spectrum into the high-frequency enhancement model to obtain the high-frequency enhanced amplitude spectrum;

[0175] The obtained high-frequency enhanced amplitude spectrum and wideband enhanced amplitude spectrum can be used for fusion processing to obtain a full-band speech enhancement signal.

[0176] The speech enhancement model in the embodiment of the present disclosure can simultaneously obtain a broadband enhanced amplitude spectrum and a high-frequency enhanced amplitude spectrum, thereby being compatible with a broadband enhancement model. During input, only a broadband amplitude spectrum may be input, and a broadband enhanced amplitude spectrum may also be obtained through the speech enhancement model.

[0177] In some examples,

[0178] The speech amplitude spectrum of the speech signal can be divided into a broadband amplitude spectrum and a high-frequency amplitude spectrum by using the frequency band division technology.

[0179] In some examples,

[0180] The broadband enhancement model includes: a first frequency domain enhancement model, a second frequency domain enhancement model and a fusion model, wherein the first frequency domain enhancement model and the second frequency domain enhancement model are trained models; the first frequency domain enhancement model is established based on a temporal convolutional network; the second frequency domain enhancement model is established based on a convolutional recurrent network;

[0181] The broadband enhancement model is used to perform broadband enhancement processing on the sample broadband amplitude spectrum to obtain the sample broadband enhanced amplitude spectrum of each speaker, including:

[0182] Using the first frequency domain enhancement model and the second frequency domain enhancement model, the sample broadband amplitude spectrum is subjected to frequency domain enhancement processing respectively, so as to obtain the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum of each speaker;

[0183] The sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum are fused by using a fusion model to obtain a sample broadband enhanced amplitude spectrum.

[0184] In some examples,

[0185] The first frequency domain enhancement model is trained based on the first frequency domain enhancement loss function;

[0186] The first frequency domain enhancement loss function is determined based on the sample first frequency domain enhancement amplitude spectrum of each speaker, as well as the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample first enhanced time domain signal, wherein the sample first enhanced time domain signal is determined based on the sample first frequency domain enhancement amplitude spectrum, and the sample reference time domain signal is determined based on the corresponding clean speech signal;

[0187] The second frequency domain enhancement model is trained based on the second frequency domain enhancement loss function;

[0188] The second frequency domain enhancement loss function is determined based on the sample second frequency domain enhancement amplitude spectrum of each speaker, as well as the corresponding sample reference broadband amplitude spectrum, sample reference time domain signal and sample second enhanced time domain signal; wherein the sample second enhanced time domain signal is determined based on the sample second frequency domain enhancement amplitude spectrum.

[0189] In some examples,

[0190] Based on the difference between the sample reference wideband amplitude spectrum and the sample wideband enhanced amplitude spectrum of each speaker, the wideband enhancement model is trained, including:

[0191] determining a fusion enhancement loss function for characterizing the difference between the sample reference wideband magnitude spectrum and the sample wideband enhanced magnitude spectrum for each speaker;

[0192] Train the fusion model based on the fusion enhancement loss function;

[0193] Among them, the fusion enhancement loss function is determined based on the sample broadband enhanced amplitude spectrum of each speaker, and the corresponding sample reference broadband amplitude spectrum, sample reference time domain signal and sample third enhanced time domain signal; wherein, the sample third enhanced time domain signal is determined based on the sample broadband enhanced amplitude spectrum, and the sample reference time domain signal is determined based on the corresponding pure speech signal.

[0194] Figure 6 It is a schematic diagram of a broadband enhancement training model structure according to an exemplary embodiment.

[0195] Figure 6 In the step of determining the first frequency domain enhancement loss function training based on the sample first frequency domain enhancement amplitude spectrum of each speaker, as well as the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample first enhanced time domain signal, the first frequency domain enhancement model is trained by the first frequency domain enhancement loss function;

[0196] Among them, the specific calculation method of the first frequency domain enhancement loss function is:

[0197] Determining a scale-invariant first signal-to-loss ratio between the sample first enhanced time domain signal and the sample reference time domain signal;

[0198] determining a first amplitude spectrum loss between a sample first frequency-domain enhanced amplitude spectrum and a sample reference wideband amplitude spectrum;

[0199] Determine a first loss weight corresponding to a first amplitude spectrum loss according to a sample reference broadband amplitude spectrum and a corresponding sample broadband amplitude spectrum;

[0200] A value of a first frequency domain enhancement loss function is determined according to the first signal-to-loss ratio, the first amplitude spectrum loss and the first loss weight.

[0201] Figure 6 In the method, a second frequency domain enhancement loss function is determined based on the sample second frequency domain enhancement amplitude spectrum of each speaker, and the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample second enhanced time domain signal, and the second frequency domain enhancement model is trained by the second frequency domain enhancement loss function;

[0202] Among them, the specific calculation method of the second frequency domain enhancement loss function is:

[0203] determining a scale-invariant second signal-to-loss ratio between the sample second enhanced time domain signal and the sample reference time domain signal;

[0204] determining a second magnitude spectrum loss between the sample second frequency-domain enhanced magnitude spectrum and the sample reference wideband magnitude spectrum;

[0205] Determine a second loss weight corresponding to a second amplitude spectrum loss according to the sample reference broadband amplitude spectrum and the corresponding sample broadband amplitude spectrum;

[0206] A value of a second frequency domain enhancement loss function is determined according to the second signal-to-loss ratio, the second amplitude spectrum loss, and the second loss weight.

[0207] Figure 6 In the method, a fusion enhancement loss function is determined based on the sample broadband enhanced amplitude spectrum of each speaker, the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample third enhanced time domain signal, and the fusion model is trained by the fusion enhancement loss function;

[0208] Among them, the specific calculation method of the fusion enhancement loss function is:

[0209] determining a scale-invariant third signal-to-loss ratio between the sample third enhanced time domain signal and the sample reference time domain signal;

[0210] determining a third magnitude spectrum loss between the sample wideband enhanced magnitude spectrum and the sample reference wideband magnitude spectrum;

[0211] Determine a third loss weight corresponding to a third amplitude spectrum loss according to the sample reference broadband amplitude spectrum and the corresponding sample broadband amplitude spectrum;

[0212] The value of the fusion enhancement loss function is determined according to the third signal-to-loss ratio, the third amplitude spectrum loss and the third loss weight.

[0213] In some examples,

[0214] The sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum are fused using the fusion model to obtain the sample broadband enhanced amplitude spectrum, including:

[0215] Dividing the sample wideband amplitude spectrum, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum into a plurality of sub-bands;

[0216] Based on the sample broadband amplitude spectrum, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum, determine the subband weights of the subbands corresponding to each sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum respectively; wherein the sum of the subband weights of the same subband corresponding to the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum is 1;

[0217] Smoothing each sub-band weight in the time dimension;

[0218] Based on the subband weight of each smoothed subband and the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum respectively corresponding to the multiple subbands, the sample broadband enhanced amplitude spectrum is determined.

[0219] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0220] Figure 7 It is a schematic diagram of a broadband enhancement model structure according to an exemplary embodiment. Figure 7 yes Figure 6 After the model training is completed, a broadband enhancement model is used for speech enhancement. The specific speech enhancement process is as follows:

[0221] Inputting the broadband amplitude spectrum corresponding to the speech amplitude spectrum of the noisy speech signal to be processed into the first frequency domain enhancement model and the second frequency domain enhancement model respectively, to obtain the first frequency domain enhancement amplitude spectrum and the second frequency domain enhancement amplitude spectrum respectively;

[0222] The first frequency domain enhanced amplitude spectrum and the second frequency domain enhanced amplitude spectrum are input into the fusion model to obtain the broadband enhanced amplitude spectrum.

[0223] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0224] Since environmental noise and interfering human voices are often collected during the audio collection process, audio and video communication systems often use non-personalized voice enhancement technology to remove the collected environmental noise, etc., retaining all voice parts, including interfering human voices. However, the presence of interfering human voices leads to a relatively poor real-time audio and video communication experience.

[0225] In order to remove the influence of interfering human voices, the personalized speech enhancement technology first learns the feature vector that describes the characteristics of the calibrated speaker from the pre-registered calibrated speaker voice, and then extracts the sound of the calibrated speaker from the noisy speech of multiple speakers with the help of the feature vector, thereby removing the interfering human voice. When the user can pre-record a speech as the calibrated speaker voice, the real-time audio and video system can use the personalized speech enhancement system to remove both environmental noise and interfering human voices. If the user cannot pre-record a speech, the real-time audio and video system can only use a non-personalized speech enhancement system.

[0226] The calibrated speaking object is a speech recorded for a single person with a high signal-to-noise ratio.

[0227] In order to meet the above requirements, the real-time audio and video system needs to maintain both the non-personalized and personalized speech enhancement systems, and the high-sampling rate speech enhancement system also needs to be compatible with the downstream speech processing system using a low sampling rate, such as speech recognition. However, the existing personalized full-band speech enhancement system is relatively independent of the personalized full-band speech enhancement system, and the full-band speech enhancement system is not compatible with the broadband speech recognition system. The cost and price of maintaining multiple independent systems will be very large.

[0228] Based on this, the embodiments of the present disclosure also provide a modular and scalable full-band speech enhancement model training method and a scalable speech enhancement method, which can meet the needs of non-personalized and personalized speech enhancement, and at the same time be compatible with related technologies for broadband speech related processing, such as speech recognition.

[0229] Figure 8 is a schematic diagram of a personalized speech enhancement method according to an exemplary embodiment. Figure 8 As shown, the speech enhancement method is applied to a computer device, and the following description is given by taking the computer device as a server as an example.

[0230] In step 801, a noisy speech signal to be processed is obtained.

[0231] In step 802, the speech amplitude spectrum of the noisy speech signal to be processed is divided into a broadband amplitude spectrum and a high-frequency amplitude spectrum.

[0232] In step 803, broadband enhancement processing is performed on the broadband amplitude spectrum to obtain a broadband enhanced amplitude spectrum.

[0233] In step 804, high-frequency enhancement processing is performed on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum to obtain a high-frequency enhanced amplitude spectrum. Step 806 is executed.

[0234] In step 805, a broadband enhanced magnitude spectrum of a sample potential speech of the target speaker is obtained based on the broadband enhanced magnitude spectrum.

[0235] In step 806, the broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum of the sample potential speech of the target speaker are fused, and a full-band enhanced speech signal of the noisy speech signal to be processed is obtained according to the fusion result.

[0236] The broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum of the sample potential speech of the target speaker are fused, and the full-band enhanced speech signal of the noisy speech signal to be processed is obtained according to the fusion result, including:

[0237] The wideband enhanced amplitude spectrum and high-frequency enhanced amplitude spectrum of the sample potential speech of the target speaker are combined, post-processed and inverse Fourier transformed to obtain a full-band enhanced speech signal of the noisy speech signal to be processed.

[0238] The merging, post-processing and inverse Fourier transform processing of the target speaker's sample potential speech broadband enhanced amplitude spectrum and high-frequency enhanced amplitude spectrum can be achieved by the following formula, but is not limited to the following formula:

[0239]

[0240] Among them, iSTFT represents inverse short time Fourier transform, post(·) represents post-processing, and concat(·) represents spectrum merging operation. is the target speaker’s sample latent speech broadband enhanced amplitude spectrum, is the high frequency enhanced amplitude spectrum.

[0241] In this embodiment, the speech amplitude spectrum of the noisy speech signal is divided into a high-frequency amplitude spectrum and a wideband amplitude spectrum; the wideband amplitude spectrum is subjected to wideband enhancement processing to obtain a wideband enhanced amplitude spectrum; the obtained wideband enhanced amplitude spectrum and high-frequency amplitude spectrum are subjected to high-frequency enhancement processing to obtain a high-frequency enhanced amplitude spectrum, and then the sample potential speech wideband enhanced amplitude spectrum of the target speaker is obtained based on the wideband enhanced amplitude spectrum; finally, the sample potential speech wideband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum of the target speaker are fused to obtain the full-band enhanced speech signal of the noisy speech signal of the target speaker. In the disclosed embodiment, the acquisition of the full-band enhanced speech signal includes the acquisition of the wideband enhanced speech signal, that is, the processing of the wideband speech enhancement is compatible. The speech enhancement method solves the problem that the full-band speech enhancement processing is not compatible with the wideband speech enhancement processing; and by adding the speech signal of the target speaker to further extract the wideband enhanced amplitude spectrum, the potential speech wideband enhanced amplitude spectrum is obtained and then fused, which can make personalized and non-personalized speech enhancement compatible.

[0242] Fig. 9 FIG. 1 is a schematic diagram of a personalized speech enhancement method based on calibrating a speaking object according to an exemplary embodiment. Fig. 9 As shown, the speech enhancement method is applied to a computer device, and the following description is given by taking the computer device as a server as an example.

[0243] In step 901, a noisy speech signal to be processed is obtained.

[0244] In step 902, the speech amplitude spectrum of the noisy speech signal to be processed is divided into a broadband amplitude spectrum and a high-frequency amplitude spectrum.

[0245] In step 903, the broadband enhancement process is performed on the broadband amplitude spectrum to obtain a broadband enhanced amplitude spectrum.

[0246] In step 904, high-frequency enhancement processing is performed on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum to obtain a high-frequency enhanced amplitude spectrum. Then, step 908 is executed.

[0247] In step 905, calibration speech signals of multiple calibration speaking objects are acquired, and a target speaking object is determined among the multiple calibration speaking objects.

[0248] In step 906, a calibration speech signal of the target speaking object is determined according to the calibration speech signal of the calibration speaking object determined as the target speaking object.

[0249] In step 907, the calibrated speech signal and the broadband enhanced amplitude spectrum of the target speaker are subjected to amplitude spectrum extraction processing to obtain the broadband enhanced amplitude spectrum of the sample potential speech of the target speaker.

[0250] In step 908, the broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum of the sample potential speech of the target speaker are fused, and a full-band enhanced speech signal of the noisy speech signal to be processed is obtained according to the fusion result.

[0251] In some examples,

[0252] In step 908, the broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum of the sample potential speech of the target speaker are fused, and a full-band enhanced speech signal of the noisy speech signal to be processed is obtained according to the fusion result, including:

[0253] The wideband enhanced amplitude spectrum and high-frequency enhanced amplitude spectrum of the sample potential speech of the target speaker are combined, post-processed and inverse Fourier transformed to obtain a full-band enhanced speech signal of the noisy speech signal to be processed.

[0254] In some examples,

[0255] In step 904, high-frequency enhancement processing is performed on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum, including:

[0256] The high-frequency enhancement model is used to perform high-frequency enhancement processing on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum;

[0257] Among them, the training method of the high-frequency enhancement model includes:

[0258] Acquire first training samples of a plurality of speakers and corresponding sample reference high-frequency amplitude spectra, wherein the first training samples include a sample broadband enhanced amplitude spectrum and a sample high-frequency amplitude spectrum of each speaker; the sample broadband enhanced amplitude spectrum and the sample high-frequency amplitude spectrum are determined based on the full-band speech amplitude spectrum of the noisy speech signal of the corresponding speaker, and the sample reference high-frequency amplitude spectrum is determined based on the pure speech signal corresponding to each speaker;

[0259] Using the high-frequency enhancement model, high-frequency enhancement processing is performed on the sample high-frequency amplitude spectrum and the sample broadband enhanced amplitude spectrum of each speaker to obtain the sample high-frequency enhanced amplitude spectrum of each speaker;

[0260] The high-frequency enhancement model is trained based on the difference between the sample high-frequency enhancement amplitude spectrum of each speaker and the sample reference high-frequency amplitude spectrum.

[0261] In some examples,

[0262] In step 903, broadband enhancement processing is performed on the broadband amplitude spectrum, including:

[0263] The broadband enhancement model is used to perform broadband enhancement processing on the broadband amplitude spectrum;

[0264] Among them, the training method of the broadband enhancement model includes:

[0265] Acquire second training samples of a plurality of speakers and corresponding sample reference broadband amplitude spectra, wherein the second training samples include a sample broadband amplitude spectrum of each speaker;

[0266] Performing broadband enhancement processing on the sample broadband amplitude spectrum using the broadband enhancement model to obtain the sample broadband enhanced amplitude spectrum of each speaker;

[0267] The broadband enhancement model is trained based on the difference between the sample reference broadband amplitude spectrum and the sample broadband enhancement amplitude spectrum of each speaker.

[0268] In some examples,

[0269] The broadband enhancement model includes: a first frequency domain enhancement model, a second frequency domain enhancement model and a fusion model, wherein the first frequency domain enhancement model and the second frequency domain enhancement model are trained models; the first frequency domain enhancement model is established based on a temporal convolutional network; the second frequency domain enhancement model is established based on a convolutional recurrent network;

[0270] The broadband enhancement model is used to perform broadband enhancement processing on the sample broadband amplitude spectrum to obtain the sample broadband enhanced amplitude spectrum of each speaker, including:

[0271] Using the first frequency domain enhancement model and the second frequency domain enhancement model, the sample broadband amplitude spectrum is subjected to frequency domain enhancement processing respectively, so as to obtain the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum of each speaker;

[0272] The sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum are fused by using a fusion model to obtain a sample broadband enhanced amplitude spectrum.

[0273] In some examples,

[0274] The first frequency domain enhancement model is trained based on the first frequency domain enhancement loss function;

[0275] The first frequency domain enhancement loss function is determined based on the sample first frequency domain enhancement amplitude spectrum of each speaker, as well as the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample first enhanced time domain signal, wherein the sample first enhanced time domain signal is determined based on the sample first frequency domain enhancement amplitude spectrum, and the sample reference time domain signal is determined based on the corresponding clean speech signal;

[0276] The second frequency domain enhancement model is trained based on the second frequency domain enhancement loss function;

[0277] The second frequency domain enhancement loss function is determined based on the sample second frequency domain enhancement amplitude spectrum of each speaker, as well as the corresponding sample reference broadband amplitude spectrum, sample reference time domain signal and sample second enhanced time domain signal; wherein the sample second enhanced time domain signal is determined based on the sample second frequency domain enhancement amplitude spectrum.

[0278] In some examples,

[0279] Based on the difference between the sample reference wideband amplitude spectrum and the sample wideband enhanced amplitude spectrum of each speaker, the wideband enhancement model is trained, including:

[0280] determining a fusion enhancement loss function for characterizing the difference between the sample reference wideband magnitude spectrum and the sample wideband enhanced magnitude spectrum for each speaker;

[0281] Train the fusion model based on the fusion enhancement loss function;

[0282] Among them, the fusion enhancement loss function is determined based on the sample broadband enhanced amplitude spectrum of each speaker, and the corresponding sample reference broadband amplitude spectrum, sample reference time domain signal and sample third enhanced time domain signal; wherein, the sample third enhanced time domain signal is determined based on the sample broadband enhanced amplitude spectrum, and the sample reference time domain signal is determined based on the corresponding pure speech signal.

[0283] In some examples,

[0284] The sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum are fused using the fusion model to obtain the sample broadband enhanced amplitude spectrum, including:

[0285] Dividing the sample wideband amplitude spectrum, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum into a plurality of sub-bands;

[0286] Based on the sample broadband amplitude spectrum, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum, determine the subband weights of the subbands corresponding to each sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum respectively; wherein the sum of the subband weights of the same subband corresponding to the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum is 1;

[0287] Smoothing each sub-band weight in the time dimension;

[0288] Based on the subband weight of each smoothed subband and the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum respectively corresponding to the multiple subbands, the sample broadband enhanced amplitude spectrum is determined.

[0289] In some examples,

[0290] In step 907, the calibrated speech signal and the broadband enhanced amplitude spectrum of the target speaker are subjected to amplitude spectrum extraction processing, including:

[0291] The amplitude spectrum extraction model of the speaker is used to extract the amplitude spectrum of the calibrated speech signal and the broadband enhanced amplitude spectrum of the target speaker;

[0292] The training method of the speaker amplitude spectrum extraction model includes:

[0293] Acquire a plurality of third training samples of the calibrated speaking objects and sample reference wideband amplitude spectra; wherein the third training samples include a sample wideband enhanced amplitude spectrum and a sample speech signal of each calibrated speaking object;

[0294] Using the speaker amplitude spectrum extraction model, the sample speech signal and the sample broadband enhanced amplitude spectrum of each calibrated speaker are subjected to amplitude spectrum extraction processing to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker;

[0295] The training of the speaker amplitude spectrum extraction model is achieved based on the difference between the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker and the corresponding sample reference broadband amplitude spectrum.

[0296] In some examples,

[0297] The speaker amplitude spectrum extraction model includes: a speaker voiceprint extraction model and a calibration speaker extraction model; the speaker voiceprint extraction model is a model that has been preliminarily trained;

[0298] The amplitude spectrum extraction model of the speaker is used to perform amplitude spectrum extraction processing on the sample speech signal and the sample broadband enhanced amplitude spectrum of each calibrated speaker to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker, including:

[0299] Using a speaker voiceprint extraction model to encode the sample speech signal of each calibrated speaker to obtain a speaker voiceprint of each calibrated speaker;

[0300] The calibrated speaker extraction model is used to perform amplitude spectrum extraction processing on the speaker voiceprint and broadband enhanced amplitude spectrum of each calibrated speaker to obtain the broadband enhanced amplitude spectrum of the sample potential speech of each calibrated speaker.

[0301] In some examples,

[0302] The speaker voiceprint extraction model is trained based on the first cross entropy loss function;

[0303] Among them, the first cross entropy loss function is determined based on the identity prediction probability corresponding to each speaker voiceprint and the label of the real identity of the speaker corresponding to each speaker voiceprint. The identity prediction probability is obtained by using a classifier to perform speaker identity discrimination processing on multiple speaker voiceprints of each calibrated speaker; the classifier is trained based on the first cross entropy loss function.

[0304] In some examples,

[0305] The difference between the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker and the corresponding sample reference broadband amplitude spectrum is characterized by a second cross entropy loss function and a calibrated speaker extraction loss function;

[0306] The calibration speaking object extraction loss function is determined based on the sample enhanced time domain signal, the sample reference time domain signal, the sample potential speech wideband enhanced amplitude spectrum and the corresponding sample reference wideband amplitude spectrum, the sample wideband amplitude spectrum, and the trained classifier; wherein the sample enhanced time domain signal is determined based on the sample potential speech wideband enhanced amplitude spectrum;

[0307] The second cross entropy loss function is determined based on the identity prediction probability corresponding to each speaker voiceprint and the label of the real identity of the speaker corresponding to each speaker voiceprint. The identity prediction probability is obtained by using a classifier to perform speaker identity discrimination processing on multiple speaker voiceprints of each calibrated speaker.

[0308] Fig.10 FIG. 1 is a schematic diagram of a personalized speech enhancement method based on a model according to an exemplary embodiment. Fig.10 As shown, the speech enhancement method is applied to a computer device, and the following description is given by taking the computer device as a server as an example.

[0309] In step 1001, a noisy speech signal to be processed is obtained.

[0310] In step 1002, the speech amplitude spectrum of the noisy speech signal to be processed is divided into a broadband amplitude spectrum and a high-frequency amplitude spectrum.

[0311] In step 1003, a broadband enhancement model is used to perform broadband enhancement processing on the broadband amplitude spectrum to obtain a broadband enhanced amplitude spectrum.

[0312] In step 1004, high-frequency enhancement processing is performed on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum using a high-frequency enhancement model. Step 1008 is executed.

[0313] In step 1005, calibration speech signals of a plurality of calibration speaking objects are acquired, and a target speaking object is determined among the plurality of calibration speaking objects.

[0314] In step 1006, a calibration speech signal of the target speaker is determined according to the calibration speech signal of the calibration speaker determined as the target speaker;

[0315] In step 1007, the speaker amplitude spectrum extraction model is used to perform amplitude spectrum extraction processing on the calibrated speech signal and the broadband enhanced amplitude spectrum of the target speaker to obtain the broadband enhanced amplitude spectrum of the sample potential speech of the target speaker.

[0316] In step 1008, the broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum of the sample potential speech of the target speaker are fused, and a full-band enhanced speech signal of the noisy speech signal to be processed is obtained according to the fusion result.

[0317] In some examples,

[0318] The speech amplitude spectrum of the speech signal can be divided into a broadband amplitude spectrum and a high-frequency amplitude spectrum by using the frequency band division technology.

[0319] In some examples,

[0320] The training method of the high-frequency enhancement model includes:

[0321] Acquire first training samples of a plurality of speakers and corresponding sample reference high-frequency amplitude spectra, wherein the first training samples include a sample broadband enhanced amplitude spectrum and a sample high-frequency amplitude spectrum of each speaker; the sample broadband enhanced amplitude spectrum and the sample high-frequency amplitude spectrum are determined based on the full-band speech amplitude spectrum of the noisy speech signal of the corresponding speaker, and the sample reference high-frequency amplitude spectrum is determined based on the pure speech signal corresponding to each speaker;

[0322] Using the high-frequency enhancement model, high-frequency enhancement processing is performed on the sample high-frequency amplitude spectrum and the sample broadband enhanced amplitude spectrum of each speaker to obtain the sample high-frequency enhanced amplitude spectrum of each speaker;

[0323] The high-frequency enhancement model is trained based on the difference between the sample high-frequency enhancement amplitude spectrum of each speaker and the sample reference high-frequency amplitude spectrum.

[0324] In some examples,

[0325] Based on the difference between the sample high-frequency enhancement amplitude spectrum of each speaker and the sample reference high-frequency amplitude spectrum, it is represented by a high-frequency enhancement loss function;

[0326] Among them, the specific calculation method of the high-frequency enhancement loss function can be:

[0327] determining a fourth magnitude spectrum loss between the sample high frequency enhanced magnitude spectrum and the sample reference high frequency magnitude spectrum;

[0328] Determining a fourth loss weight corresponding to a fourth amplitude spectrum loss according to a sample reference high-frequency amplitude spectrum and a corresponding sample high-frequency amplitude spectrum;

[0329] A value of a high frequency enhancement loss function is determined according to the fourth amplitude spectrum loss and the fourth loss weight.

[0330] In some examples,

[0331] The training method of the broadband enhancement model includes:

[0332] Acquire second training samples of a plurality of speakers and corresponding sample reference broadband amplitude spectra, wherein the second training samples include a sample broadband amplitude spectrum of each speaker;

[0333] Performing broadband enhancement processing on the sample broadband amplitude spectrum using the broadband enhancement model to obtain the sample broadband enhanced amplitude spectrum of each speaker;

[0334] The broadband enhancement model is trained based on the difference between the sample reference broadband amplitude spectrum and the sample broadband enhancement amplitude spectrum of each speaker.

[0335] In some examples,

[0336] The training method of the speaker amplitude spectrum extraction model includes:

[0337] Acquire a plurality of third training samples of the calibrated speaking objects and sample reference wideband amplitude spectra; wherein the third training samples include a sample wideband enhanced amplitude spectrum and a sample speech signal of each calibrated speaking object;

[0338] Using the speaker amplitude spectrum extraction model, the sample speech signal and the sample broadband enhanced amplitude spectrum of each calibrated speaker are subjected to amplitude spectrum extraction processing to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker;

[0339] The training of the speaker amplitude spectrum extraction model is achieved based on the difference between the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker and the corresponding sample reference broadband amplitude spectrum.

[0340] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0341] Fig.11 It is a schematic diagram of a personalized speech enhancement model training structure according to an illustrative embodiment. Fig.11 A high-frequency enhancement model, a broadband enhancement model and a speaking object amplitude spectrum extraction model are shown. After the broadband enhancement model is trained, the high-frequency enhancement model and the speaking object amplitude spectrum extraction model are trained again. The training of the high-frequency enhancement model and the speaking object amplitude spectrum extraction model is not in any particular order.

[0342] The first training sample used in training the broadband enhancement model and the second training sample used in training the high-frequency enhancement model may be the same training sample or different training samples. Fig.11 In the example, the same training samples are collectively referred to as training samples.

[0343] The specific training process is as follows:

[0344] The first step is to obtain training samples of multiple speakers, wherein the training samples include a sample wideband amplitude spectrum and a sample high-frequency amplitude spectrum of each speaker.

[0345] The sample wideband amplitude spectrum and the sample high-frequency amplitude spectrum are determined based on the full-band speech amplitude spectrum of the noisy speech signal of the corresponding speaker.

[0346] In order to calculate the loss function, it is also necessary to obtain the clean speech signal corresponding to the noisy speech signal of the calibrated speaker. The noisy speech signal of each speaker and the corresponding clean speech signal form a signal pair.

[0347] The second step is to use the broadband enhancement model to perform broadband speech enhancement processing on the sample broadband amplitude spectrum to obtain the sample broadband enhanced amplitude spectrum of each speaker.

[0348] The third step is to determine the value of the broadband enhancement loss function based on the sample reference broadband amplitude spectrum and the sample broadband enhancement amplitude spectrum of each speaker, wherein the sample reference broadband amplitude spectrum of each speaker is determined based on the pure speech signal of the corresponding speaker.

[0349] The fourth step is to adjust the parameters of the broadband enhancement model based on the value of the broadband enhancement loss function to achieve training of the broadband enhancement model.

[0350] The fifth step is to use the high-frequency enhancement model to perform high-frequency speech enhancement processing on the sample high-frequency amplitude spectrum of each speaker and the sample broadband enhanced amplitude spectrum obtained after the sample broadband speech enhancement model training is completed, so as to obtain the sample high-frequency enhanced amplitude spectrum of each speaker.

[0351] Step 6: Determine the value of the high-frequency enhancement loss function based on the sample high-frequency enhancement amplitude spectrum and the sample reference high-frequency amplitude spectrum of each speaker, wherein the sample reference high-frequency amplitude spectrum of each speaker is determined based on the pure speech signal of the corresponding speaker.

[0352] Step 7: Adjust the parameters of the high-frequency enhancement model based on the value of the high-frequency enhancement loss function to achieve training of the high-frequency enhancement model.

[0353] At this point, the training of the full-band speech enhancement model including the broadband enhancement model and the high-frequency enhancement model has been completed. The trained speech enhancement model can output the high-frequency enhancement amplitude spectrum and the broadband enhancement amplitude spectrum. If only the sample broadband amplitude spectrum is input, only the broadband enhancement amplitude spectrum will be output.

[0354] The eighth step is to obtain sample speech signals of multiple calibrated speaking objects.

[0355] The sample speech signal of the calibration speaker is a single person speech signal of the calibration speaker with a high signal-to-noise ratio.

[0356] The clean speech signal is the clean speech of the calibrated speaker corresponding to the sample speech signal of the calibrated speaker in the noisy speech signal.

[0357] The ninth step is to use the speaker amplitude spectrum extraction model to perform amplitude spectrum extraction processing on the sample speech signal of each calibrated speaker, the label of the calibrated speaker and the broadband enhanced amplitude spectrum to obtain the broadband enhanced amplitude spectrum of the sample potential speech of each calibrated speaker.

[0358] Step 10: Determine the value of the speaking object amplitude spectrum extraction loss function based on the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaking object and the corresponding reference broadband amplitude spectrum.

[0359] Step 11: adjust the parameters of the speaking object amplitude spectrum extraction model according to the value of the speaking object amplitude spectrum extraction loss function to realize the training of the speaking object amplitude spectrum extraction model.

[0360] The implementation sequence of step 8 and step 5.

[0361] When the training of the speaker amplitude spectrum extraction model and the high-frequency enhancement model is completed, the training of the non-personalized personalized speech enhancement model is completed.

[0362] In some examples, the broadband enhancement model includes: a first frequency domain enhancement model, a second frequency domain enhancement model, and a fusion model.

[0363] In the second step, the broadband enhancement model is used to perform broadband speech enhancement processing on the sample broadband amplitude spectrum to obtain the sample broadband enhanced amplitude spectrum of each speaker, including:

[0364] Using the first frequency domain enhancement model and the second frequency domain enhancement model, the sample broadband amplitude spectrum is subjected to frequency domain speech enhancement processing respectively, so as to obtain the sample first frequency domain enhancement amplitude spectrum and the sample second frequency domain enhancement amplitude spectrum of each speaker;

[0365] Using the fusion model, a first frequency domain enhanced amplitude spectrum of samples obtained by the trained first frequency domain enhancement model and a second frequency domain enhanced amplitude spectrum of samples obtained by the trained second frequency domain enhancement model are fused to obtain a broadband enhanced amplitude spectrum of samples of each speaker;

[0366] The high-frequency enhancement model is used to perform high-frequency speech enhancement processing on the sample high-frequency amplitude spectrum of each speaker and the sample broadband enhanced amplitude spectrum obtained after the broadband speech enhancement model training is completed, and the sample high-frequency enhanced amplitude spectrum of each speaker is obtained, including:

[0367] The high-frequency enhancement model is used to perform high-frequency speech enhancement processing on the sample high-frequency amplitude spectrum of each speaker and the sample broadband enhanced amplitude spectrum obtained after the fusion model training is completed, so as to obtain the sample high-frequency enhanced amplitude spectrum of each speaker.

[0368] In some examples, the broadband enhancement loss function includes: a first frequency domain enhancement loss function, a second frequency domain enhancement loss function, and a fusion enhancement loss function;

[0369] In the third step, based on the sample reference broadband amplitude spectrum and the sample broadband enhancement amplitude spectrum of each speaker, the value of the broadband enhancement loss function is determined, including:

[0370] Determine the value of the first frequency domain enhancement loss function based on the sample first frequency domain enhancement amplitude spectrum of each speaker, as well as the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample first enhanced time domain signal; wherein the sample first enhanced time domain signal is determined based on the sample first frequency domain enhancement amplitude spectrum, and the sample reference time domain signal is determined based on the corresponding clean speech signal;

[0371] Determine the value of the second frequency domain enhancement loss function based on the sample second frequency domain enhancement amplitude spectrum of each speaker, and the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample second enhanced time domain signal; wherein the sample second enhanced time domain signal is determined based on the sample second frequency domain enhancement amplitude spectrum;

[0372] The value of the fusion enhancement loss function is determined based on the sample broadband enhanced amplitude spectrum of each speaker, and the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample third enhanced time domain signal; wherein the sample third enhanced time domain signal is determined based on the broadband enhanced amplitude spectrum.

[0373] In some examples,

[0374] The parameters of the broadband enhancement model are adjusted based on the value of the broadband enhancement loss function to implement the training of the broadband enhancement model, including:

[0375] Adjusting the parameters of the first frequency domain enhancement model based on the value of the first frequency domain enhancement loss function to implement training of the first frequency domain enhancement model;

[0376] Adjusting the parameters of the second frequency domain enhancement model based on the value of the second frequency domain enhancement loss function to implement training of the second frequency domain enhancement model;

[0377] The parameters of the fusion enhancement model are adjusted based on the value of the fusion enhancement loss function to achieve training of the fusion enhancement model.

[0378] The value of the first frequency domain enhancement loss function is determined based on the sample first frequency domain enhanced amplitude spectrum of each speaker, and the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample first enhanced time domain signal, including:

[0379] Determining a scale-invariant first signal-to-loss ratio between the sample first enhanced time domain signal and the sample reference time domain signal;

[0380] determining a first amplitude spectrum loss between a sample first frequency-domain enhanced amplitude spectrum and a sample reference wideband amplitude spectrum;

[0381] Determine a first loss weight corresponding to a first amplitude spectrum loss according to a sample reference broadband amplitude spectrum and a corresponding sample broadband amplitude spectrum;

[0382] Determining a value of a first frequency domain enhancement loss function according to the first signal-to-loss ratio, the first amplitude spectrum loss, and the first loss weight;

[0383] Determining the value of the second frequency domain enhancement loss function based on the sample second frequency domain enhancement amplitude spectrum of each speaker, and the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal, and the sample second enhanced time domain signal, includes:

[0384] determining a scale-invariant second signal-to-loss ratio between the sample second enhanced time domain signal and the sample reference time domain signal;

[0385] determining a second magnitude spectrum loss between the sample second frequency-domain enhanced magnitude spectrum and the sample reference wideband magnitude spectrum;

[0386] Determine a second loss weight corresponding to a second amplitude spectrum loss according to the sample reference broadband amplitude spectrum and the corresponding sample broadband amplitude spectrum;

[0387] Determine a value of a second frequency domain enhancement loss function according to the second signal-to-loss ratio, the second amplitude spectrum loss, and the second loss weight;

[0388] Determining the value of the fusion enhancement loss function based on the sample broadband enhanced amplitude spectrum of each speaker, and the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal, and the sample third enhanced time domain signal, including:

[0389] determining a scale-invariant third signal-to-loss ratio between the sample third enhanced time domain signal and the sample reference time domain signal;

[0390] determining a third magnitude spectrum loss between the sample wideband enhanced magnitude spectrum and the sample reference wideband magnitude spectrum;

[0391] Determine a third loss weight corresponding to a third amplitude spectrum loss according to the sample reference broadband amplitude spectrum and the corresponding sample broadband amplitude spectrum;

[0392] The value of the fusion enhancement loss function is determined according to the third signal-to-loss ratio, the third amplitude spectrum loss and the third loss weight.

[0393] In some examples,

[0394] In the fourth step, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum are fused using the fusion model to obtain the sample broadband enhanced amplitude spectrum, including:

[0395] Dividing the sample wideband amplitude spectrum, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum into a plurality of sub-bands;

[0396] Determine the subband weight of the subband corresponding to each sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum based on the sample broadband amplitude spectrum, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum; wherein the sum of the subband weights of the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum of each subband is 1;

[0397] The subband weights are smoothed in the time dimension using the exponential moving average method;

[0398] Based on the subband weight of each smoothed subband and the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum corresponding to the multiple subbands, the sample broadband enhanced amplitude spectrum is determined.

[0399] In some examples,

[0400] In the sixth step, based on the sample high-frequency enhancement amplitude spectrum of each speaker and the sample reference high-frequency amplitude spectrum, the value of the high-frequency enhancement loss function is determined, including:

[0401] determining a fourth magnitude spectrum loss between the sample high frequency enhanced magnitude spectrum and the sample reference high frequency magnitude spectrum;

[0402] Determining a fourth loss weight corresponding to a fourth amplitude spectrum loss according to a sample reference high-frequency amplitude spectrum and a corresponding sample high-frequency amplitude spectrum;

[0403] A value of a high frequency enhancement loss function is determined according to the fourth amplitude spectrum loss and the fourth loss weight.

[0404] In some examples, the speaking object amplitude spectrum extraction model includes: a speaking object voiceprint extraction model and a calibration speaking object extraction model;

[0405] In the ninth step, the amplitude spectrum extraction model of the speaker is used to perform amplitude spectrum extraction processing on the sample speech signal of each calibrated speaker, the label of the calibrated speaker, and the sample broadband enhanced amplitude spectrum to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker, including:

[0406] Using a speaker voiceprint extraction model to encode the sample speech signal of each calibrated speaker to obtain a speaker voiceprint of each calibrated speaker;

[0407] The calibrated speaking object extraction model is used to perform amplitude spectrum extraction processing on the speaking object voiceprint of each calibrated speaking object, the label of the calibrated speaking object and the sample broadband enhanced amplitude spectrum to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaking object.

[0408] In some examples, the speaking object amplitude spectrum extraction model further includes: a classifier;

[0409] The loss function for extracting the amplitude spectrum of a speaking object includes: a first cross entropy loss function and a calibration speaking object extraction loss function;

[0410] In the tenth step, the value of the speaking object amplitude spectrum extraction loss function is determined according to the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaking object and the corresponding sample reference broadband amplitude spectrum, including:

[0411] Using a classifier to perform speaker identity discrimination processing on multiple speaker voiceprints of each calibrated speaker, and obtain an estimated identity prediction probability corresponding to each speaker voiceprint; determining the value of a first cross entropy loss function according to the identity prediction probability corresponding to each speaker voiceprint and the label of the real identity of the speaker corresponding to each speaker voiceprint;

[0412] Determine a scale-invariant fourth signal-to-loss ratio between a sample fourth enhanced time domain signal and a sample reference time domain signal; wherein the fourth enhanced time domain signal is determined based on a potential speech broadband enhanced amplitude spectrum;

[0413] determining a fifth magnitude spectrum loss between a potential speech wideband enhancement magnitude spectrum and a corresponding sample reference wideband magnitude spectrum;

[0414] determining a fifth loss weight corresponding to a fifth amplitude spectrum loss according to the sample reference broadband amplitude spectrum and the corresponding sample broadband amplitude spectrum;

[0415] Determining a value of a second cross entropy loss function according to the initially trained speaker voiceprint extraction model and the trained classifier;

[0416] Determining a value of a calibration speaking object extraction loss function according to a fourth signal-to-loss ratio, a fifth amplitude spectrum loss, a fifth loss weight, and a second cross entropy loss function;

[0417] The parameters of the speaking object amplitude spectrum extraction model are adjusted according to the value of the speaking object amplitude spectrum extraction loss function to implement the training of the speaking object amplitude spectrum extraction model, including:

[0418] Adjusting parameters of the classifier and the speaker voiceprint extraction model according to the value of the first cross entropy loss function to achieve preliminary training of the speaker voiceprint extraction model and training of the classifier;

[0419] The parameters of the speaking object voiceprint extraction model and the calibrated speaking object extraction model are adjusted according to the value of the calibrated speaking object extraction loss function to achieve the training of the speaking object voiceprint extraction model and the calibrated speaking object extraction model.

[0420] In some examples,

[0421] The speaker amplitude spectrum extraction model includes: a speaker voiceprint extraction model and a calibration speaker extraction model; the speaker voiceprint extraction model is a model that has been preliminarily trained;

[0422] The amplitude spectrum extraction model of the speaker is used to perform amplitude spectrum extraction processing on the sample speech signal and the sample broadband enhanced amplitude spectrum of each calibrated speaker to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker, including:

[0423] Using a speaker voiceprint extraction model to encode the sample speech signal of each calibrated speaker to obtain a speaker voiceprint of each calibrated speaker;

[0424] The calibrated speaker extraction model is used to perform amplitude spectrum extraction processing on the speaker voiceprint and sample broadband enhanced amplitude spectrum of each calibrated speaker to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker.

[0425] In some examples,

[0426] The speaker voiceprint extraction model is trained based on the first cross entropy loss function;

[0427] Among them, the first cross entropy loss function is determined based on the identity prediction probability corresponding to each speaker voiceprint and the label of the real identity of the speaker corresponding to each speaker voiceprint. The identity prediction probability is obtained by using a classifier to perform speaker identity discrimination processing on multiple speaker voiceprints of each calibrated speaker; the classifier is trained based on the first cross entropy loss function.

[0428] In some examples,

[0429] The difference between the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker and the corresponding sample reference broadband amplitude spectrum is characterized by a second cross entropy loss function and a calibrated speaker extraction loss function;

[0430] The calibration speaking object extraction loss function is determined based on the sample enhanced time domain signal, the sample reference time domain signal, the sample potential speech wideband enhanced amplitude spectrum and the corresponding sample reference wideband amplitude spectrum, the sample wideband amplitude spectrum, and the trained classifier; wherein the sample enhanced time domain signal is determined based on the sample potential speech wideband enhanced amplitude spectrum;

[0431] The second cross entropy loss function is determined based on the identity prediction probability corresponding to each speaker voiceprint and the label of the real identity of the speaker corresponding to each speaker voiceprint. The identity prediction probability is obtained by using a classifier to perform speaker identity discrimination processing on multiple speaker voiceprints of each calibrated speaker.

[0432] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0433] Fig.12 The figure is a schematic diagram of a personalized speech enhancement model structure according to an exemplary embodiment. Fig.12 yes Fig.11 The trained personalized speech enhancement model can be used for both personalized speech enhancement models and non-personalized speech enhancement models.

[0434] See also Fig.12 As shown, the broadband amplitude spectrum corresponding to the speech amplitude spectrum of the noisy speech signal to be processed is input into the broadband enhancement model to obtain the broadband enhanced amplitude spectrum;

[0435] Inputting the high-frequency enhancement amplitude spectrum corresponding to the speech amplitude spectrum of the noisy speech signal to be processed and the obtained broadband enhancement amplitude spectrum into the high-frequency enhancement model to obtain the high-frequency enhancement amplitude spectrum;

[0436] The broadband enhanced amplitude spectrum and the speech signal of the target speaker are input into the speaker amplitude spectrum extraction model to obtain the broadband enhanced amplitude spectrum of the sample potential speech of the target speaker.

[0437] Fig.12 The personalized speech enhancement model shown can obtain a broadband enhancement amplitude spectrum, a high-frequency enhancement amplitude spectrum, and a broadband enhancement amplitude spectrum of a sample potential speech of a target speaker, and can obtain the required amplitude spectrum as needed for speech enhancement.

[0438] In the embodiment of the present disclosure, the full-band speech amplitude spectrum of the noisy speech signal is divided into a sample high-frequency amplitude spectrum and a sample wideband amplitude spectrum; the wideband enhancement model is trained based on the wideband enhancement amplitude spectrum corresponding to the sample wideband amplitude spectrum obtained by the wideband enhancement model, and the reference wideband amplitude spectrum; after the wideband enhancement model training is completed, the high-frequency enhancement model is trained using the wideband enhancement amplitude spectrum obtained by the trained wideband enhancement model, the sample high-frequency enhancement amplitude spectrum and the reference high-frequency enhancement amplitude spectrum; and the speaking object amplitude spectrum extraction model is trained based on the sample speech signal and the wideband enhancement amplitude spectrum of each calibrated speaking object. The full-band speech enhancement model in the embodiment of the present disclosure is compatible with wideband speech enhancement processing, as well as personalized and non-personalized full-band speech enhancement models, and the wideband amplitude spectrum corresponding to the wideband speech can also be input to obtain the wideband enhancement amplitude spectrum. The speech enhancement model training method can solve the problem that the full-band speech enhancement processing is not compatible with the wideband speech enhancement processing, and the full-band speech enhancement model cannot process the wideband speech input, and also solves the problem that the personalized speech enhancement model and the non-personalized speech enhancement model are not compatible.

[0439] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0440] Fig.13 FIG. 1 is a schematic diagram of a training structure of a speaker amplitude spectrum extraction model according to an exemplary embodiment. Fig.13 As shown, the training process of each model in the amplitude spectrum extraction model of the speaking object is as follows:

[0441] The first step is to obtain sample speech signals of multiple calibrated speaking objects.

[0442] The second step is to use the speaker voiceprint extraction model to encode the sample speech signal of each calibrated speaker to obtain the speaker voiceprint of each calibrated speaker.

[0443] The third step is to use the classifier to perform speaker identity discrimination processing on the speaker voiceprint of each calibrated speaker to obtain the estimated identity prediction probability corresponding to each speaker voiceprint; determine the value of the first cross entropy loss function based on the identity prediction probability corresponding to each speaker voiceprint and the label of the speaker's true identity corresponding to each speaker voiceprint.

[0444] That is, the first cross entropy loss function is determined based on the identity prediction probability corresponding to each speaker voiceprint and the label of the real identity of the speaker corresponding to each speaker voiceprint, and the identity prediction probability is obtained by using a classifier to perform speaker identity discrimination processing on multiple speaker voiceprints of each calibrated speaker; the classifier is trained based on the first cross entropy loss function.

[0445] The fourth step is to adjust the parameters of the classifier and the speaker voiceprint extraction model according to the value of the first cross entropy loss function to achieve preliminary training of the speaker voiceprint extraction model and training of the classifier.

[0446] Step 5: Use the calibrated speaker extraction model to perform amplitude spectrum extraction processing on the speaker voiceprint and sample broadband enhanced amplitude spectrum of each calibrated speaker to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker.

[0447] Step 6: Determine the difference between the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker and the corresponding sample reference broadband amplitude spectrum, and the difference is characterized by the second cross entropy loss function and the calibrated speaker extraction loss function.

[0448] The calibration speaking object extraction loss function is determined based on the sample enhanced time domain signal, the sample reference time domain signal, the sample potential speech wideband enhanced amplitude spectrum and the corresponding sample reference wideband amplitude spectrum, the sample wideband amplitude spectrum, and the trained classifier; wherein the sample enhanced time domain signal is determined based on the sample potential speech wideband enhanced amplitude spectrum;

[0449] The second cross entropy loss function is determined based on the identity prediction probability corresponding to each speaker voiceprint and the label of the real identity of the speaker corresponding to each speaker voiceprint. The identity prediction probability is obtained by using a classifier to perform speaker identity discrimination processing on multiple speaker voiceprints of each calibrated speaker.

[0450] The specific calculation method of the value of the second cross entropy loss function and the value of the calibration speaking object extraction loss function is as follows:

[0451] Determine a scale-invariant fourth signal-to-loss ratio between a sample fourth enhanced time domain signal and a sample reference time domain signal; wherein the sample fourth enhanced time domain signal is determined based on a potential speech broadband enhanced amplitude spectrum;

[0452] determining a fifth magnitude spectrum loss between a sample potential speech wideband enhancement magnitude spectrum and a corresponding sample reference wideband magnitude spectrum;

[0453] determining a fifth loss weight corresponding to a fifth amplitude spectrum loss according to the sample reference broadband amplitude spectrum and the corresponding sample broadband amplitude spectrum;

[0454] Determining a value of a second cross entropy loss function according to the initially trained speaker voiceprint extraction model and the trained classifier;

[0455] The value of the calibration speaking object extraction loss function is determined according to the values ​​of the fourth signal-to-loss ratio, the fifth amplitude spectrum loss, the fifth loss weight and the second cross entropy loss function.

[0456] Step 7: Adjust the parameters of the speaking object voiceprint extraction model and the calibrated speaking object extraction model according to the value of the calibrated speaking object extraction loss function to implement the training of the speaking object voiceprint extraction model and the calibrated speaking object extraction model.

[0457] Fig.14 The figure is a schematic diagram of a model structure for extracting amplitude spectrum of a speaking object according to an exemplary embodiment. Fig.14 yes Fig.13 The trained speaker amplitude spectrum extraction model is used to extract the potential speech broadband enhanced amplitude spectrum. The specific implementation process is as follows:

[0458] Inputting the speech signal of the target speaker into the speaker voiceprint extraction model to obtain the speaker voiceprint of the target speaker;

[0459] The obtained speaking object voiceprint and broadband enhanced amplitude spectrum of the target speaking object are input into the calibrated speaking object extraction model to obtain the broadband enhanced amplitude spectrum of the sample potential speech of the target speaking object.

[0460] In the embodiment of the present disclosure, the full-band speech amplitude spectrum of the noisy speech signal is divided into a sample high-frequency amplitude spectrum and a sample wideband amplitude spectrum; the wideband enhancement model is trained based on the wideband enhancement amplitude spectrum corresponding to the sample wideband amplitude spectrum obtained by the wideband enhancement model, and the reference wideband amplitude spectrum; after the wideband enhancement model training is completed, the high-frequency enhancement model is trained using the wideband enhancement amplitude spectrum obtained by the trained wideband enhancement model, the sample high-frequency enhancement amplitude spectrum and the reference high-frequency enhancement amplitude spectrum; and the speaking object amplitude spectrum extraction model is trained based on the sample speech signal and the wideband enhancement amplitude spectrum of each calibrated speaking object. The full-band speech enhancement model in the embodiment of the present disclosure is compatible with wideband speech enhancement processing, as well as personalized and non-personalized full-band speech enhancement models, and the wideband amplitude spectrum corresponding to the wideband speech can also be input to obtain the wideband enhancement amplitude spectrum. The speech enhancement model training method can solve the problem that the full-band speech enhancement processing is not compatible with the wideband speech enhancement processing, and the full-band speech enhancement model cannot process the wideband speech input, and also solves the problem that the personalized speech enhancement model and the non-personalized speech enhancement model are not compatible.

[0461] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0462] Fig.15 It is a schematic diagram of a personalized multi-model speech enhancement model training structure according to an illustrative embodiment. Fig.15 In the present invention, for the broadband enhancement model, the second frequency domain enhancement model, the first frequency domain enhancement model and the fusion model are used to jointly realize broadband enhancement. For the speaking object amplitude spectrum extraction model, the speaking object voiceprint extraction model, the calibrated speaking object extraction model and the classifier are used to jointly realize the extraction of the calibrated speaking object amplitude spectrum.

[0463] Combine the following Fig.15 The training process of a multi-model speech enhancement model that is compatible with both non-personalized and personalized models is given:

[0464] The first step is to obtain training samples of multiple speakers and sample speech signals of multiple calibrated speakers.

[0465] The training samples include a sample wideband amplitude spectrum and a sample high-frequency amplitude spectrum of each speaker;

[0466] The sample wideband amplitude spectrum and the sample high frequency amplitude spectrum are determined based on the full-band speech amplitude spectrum of the noisy speech signal of the corresponding speaker.

[0467] During the training process, the clean speech signal corresponding to the noisy speech signal of the speaker is also used to calculate the value of some loss functions; the sample speech signal of the calibrated speaker is the speech of a single person of the calibrated speaker with a high signal-to-noise ratio;

[0468] The clean speech signal is the clean speech of the calibrated speaker corresponding to the sample speech signal of the calibrated speaker in the noisy speech signal.

[0469] The process of obtaining training samples is given by taking a specific example:

[0470] Data simulation is performed using multiple voices of tens of thousands of people. Hundreds of data pairs (y(t), s(t)) are constructed for each speaker. In the case of compatible personalized speech enhancement, it is also necessary to add a segment of speech pre-registered by the calibrated speaker and the speaker's label. The multiple data pairs established are a segment of speech pre-registered by the speaker and the speaker's label. Therefore, the data pairs corresponding to the samples obtained for the personalized and non-personalized speech enhancement training model are (x(t), y(t), s(t), label), where x(t) is the speech signal of a single speaker with a high signal-to-noise ratio (SNR). The y(t) corresponding to each speaker includes both the speech of a single speaker and the noisy speech signals of multiple speakers with different SNRs and different RIRs. s(t) is the clean speech signal of the speaker corresponding to x(t) in y(t). label is the label corresponding to the speaker.

[0471] For y(t) in the data pair, a short-time Fourier transform is required: Taking the processing of a single voice y(t) as an example, the original audio signal is converted to the time-frequency domain through a short-time Fourier transform. If the original audio signal of length T is y(t) in the time domain, where t represents time, 0<t≤T, then after the short-time Fourier transform, y(t) can be expressed in the time-frequency domain as:

[0472] Y(n,k)=STFT(y(t)) (1)

[0473] Where n is the frame sequence, 0<n≤N, (N is the total number of frames); k is the center frequency sequence 0<k≤K; (K is the total number of frequency points). For convenience, the subscripts n and k are omitted below.

[0474] The amplitude spectrum |Y| in the time-frequency domain is used as an input feature, and the phase value ∠Y is used in the inverse short-time Fourier transform as the phase of the predicted speech. In the embodiment disclosed herein, in order to maintain the compatibility of the back-end wideband speech recognition system, we adopt a frequency division processing method to divide the full-band speech amplitude spectrum into a wideband amplitude spectrum |Y| 16 and high frequency amplitude spectrum |Y| 16-48 .

[0475] In the second step, the first frequency domain enhancement model and the second frequency domain enhancement model are used to perform frequency domain speech enhancement processing on the sample broadband amplitude spectrum respectively, so as to obtain the sample first frequency domain enhancement amplitude spectrum and the sample second frequency domain enhancement amplitude spectrum of each speaker.

[0476] Taking the sample broadband amplitude spectrum |Y| 16 As input, correlation processing is performed based on the first frequency domain enhancement model and the second frequency domain enhancement model to obtain the sample first frequency domain enhancement amplitude spectrum and the sample second frequency domain enhancement amplitude spectrum of each speaker respectively.

[0477] The first frequency domain enhancement model is based on the broadband speech enhancement function f of the temporal convolutional network (TCN). TCN (·), and then obtain the mask corresponding to the broadband amplitude spectrum Finally, the enhanced sample first frequency domain enhanced amplitude spectrum is obtained and output

[0478] The second frequency domain enhancement model is based on the broadband speech enhancement function f of the Convolutional Recurrent Network (CRN). CRN (·), and then obtain the mask corresponding to the broadband amplitude spectrum Finally, the enhanced sample second frequency domain enhanced amplitude spectrum is obtained and output

[0479] in, and It can be expressed by the following formula:

[0480]

[0481]

[0482] in, Represents dot product.

[0483] The two frequency domain enhancement models in the embodiments of the present disclosure are implemented by using a convolutional recurrent network and a sequential convolutional network as an example, but are not limited to the implementation of these two networks. The two frequency domain enhancement models can be implemented using different networks.

[0484] The third step is to determine the value of the first frequency domain enhancement loss function based on the sample first frequency domain enhanced amplitude spectrum of each speaker, as well as the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample first enhanced time domain signal, wherein the sample first enhanced time domain signal is determined based on the sample first frequency domain enhanced amplitude spectrum, and the sample reference time domain signal is determined based on the corresponding pure speech signal.

[0485] The third step determines the value of the first frequency domain enhancement loss function based on the sample first frequency domain enhancement amplitude spectrum of each speaker, and the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample first enhanced time domain signal, specifically including:

[0486] Step A: determining a scale-invariant first signal-to-loss ratio between a sample first enhanced time domain signal and a sample reference time domain signal.

[0487] Step B: determining a first amplitude spectrum loss between the sample first frequency domain enhanced amplitude spectrum and the sample reference broadband amplitude spectrum.

[0488] Step C: determining a first loss weight corresponding to the first amplitude spectrum loss according to the sample reference wideband amplitude spectrum and the corresponding sample wideband amplitude spectrum.

[0489] Step D: Determine the value of the first frequency domain enhancement loss function according to the first signal-to-loss ratio, the first amplitude spectrum loss and the first loss weight.

[0490] In this disclosed embodiment, we use the frequency domain loss function J IAM-MALE And the time domain loss function J SI-SDR To train f TCN (·) The parameters of the network, the specific calculation formula of the first frequency domain enhancement loss function can be: IAM-MALE +J SI-SDR ;

[0491] in,

[0492]

[0493]

[0494] but,

[0495]

[0496] In formula (5), M = (|S| / |Y|) γ , M is the ideal amplitude mask (IAM). In the first frequency domain enhancement loss function, M is used to balance the importance of speech and noise. The smaller the value of M, the greater the corresponding loss weight e.(a / (b+M)) a, b and γ are adjustable weights, such as 2, 1 and 1, respectively, but there is no restriction on this.

[0497] in, Sample reference broadband amplitude spectrum |S| = |S| 16 , the sample reference time domain signal s is determined based on the corresponding clean speech signal, that is, s(t) in the data pair, and iSTFT represents inverse Fourier transform.

[0498] J IAM-MALE The first frequency domain enhanced amplitude spectrum of the sample is calculated The dot product of the first amplitude spectrum loss and the first loss weight between the corresponding sample reference broadband amplitude spectrum |S|, the sample first frequency domain enhanced amplitude spectrum The smaller the difference between the corresponding sample reference wideband amplitude spectrum |S|, the closer the enhanced speech amplitude spectrum is to the reference speech. In order to reduce the range of amplitude spectrum values, we compress the amplitude spectrum in the logarithmic domain, i.e., ln(·).

[0499] J SI-SDR The first time domain enhanced signal of the sample is calculated and the scale-invariant signal to distortion ratio (SI-SDR) between the sample reference time domain signal s.

[0500] When the scale-invariant signal-to-loss ratio is larger, it means that the enhanced signal is closer to the reference time domain signal. Since the loss function is minimized in the embodiment of the present disclosure, J SI-SDR The negative of the scale-invariant first signal-to-loss ratio is taken as the loss.

[0501] Step 4: Adjust the parameters of the first frequency domain enhancement model based on the value of the first frequency domain enhancement loss function to implement the training of the first frequency domain enhancement model. Execute step 7.

[0502] When training the first frequency domain enhancement model, the training of the first frequency domain enhancement model is terminated until the value of the first frequency domain enhancement loss function no longer decreases in two consecutive iterations in the sample set, or the value of the first frequency domain enhancement loss function is less than a first preset threshold.

[0503] In the embodiment of the present disclosure, the specific implementation process of adjusting the parameters of the first frequency domain enhancement model based on the value of the first frequency domain enhancement loss function is not limited.

[0504] In the fifth step, the value of the second frequency domain enhancement loss function is determined based on the sample second frequency domain enhanced amplitude spectrum of each speaker, and the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample second enhanced time domain signal.

[0505] In the fifth step, the value of the second frequency domain enhancement loss function is determined based on the sample second frequency domain enhanced amplitude spectrum of each speaker, and the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample second enhanced time domain signal, including:

[0506] determining a scale-invariant second signal-to-loss ratio between the sample second enhanced time domain signal and the sample reference time domain signal;

[0507] determining a second magnitude spectrum loss between the sample second frequency-domain enhanced magnitude spectrum and the sample reference wideband magnitude spectrum;

[0508] Determine a second loss weight corresponding to a second amplitude spectrum loss according to the sample reference broadband amplitude spectrum and the corresponding sample broadband amplitude spectrum;

[0509] A value of a second frequency domain enhancement loss function is determined according to the second signal-to-loss ratio, the second amplitude spectrum loss, and the second loss weight.

[0510] The expression of the second loss function is the same as formula (5), but the content corresponding to each character used for calculation is different, where: Reference broadband amplitude spectrum |S|=|S| 16 ,

[0511] In the sixth step, the parameters of the second frequency domain enhancement model are adjusted based on the value of the second frequency domain enhancement loss function to implement the training of the second frequency domain enhancement model.

[0512] When training the second frequency domain enhancement model, the training of the second frequency domain enhancement model is terminated until the value of the second frequency domain enhancement loss function no longer decreases for two consecutive iterations in the sample set, or the value of the second frequency domain enhancement loss function is less than a second preset threshold.

[0513] In specific implementation, the training of the first frequency domain enhancement model (the third and fourth steps) and the second frequency domain enhancement model (the fifth and sixth steps) can be performed simultaneously, or they can be performed in sequence, which is not limited in the embodiments of the present disclosure.

[0514] Step 7: Use the fusion model to fuse the sample first frequency domain enhanced amplitude spectrum obtained by the trained first frequency domain enhancement model and the sample second frequency domain enhanced amplitude spectrum obtained by the trained second frequency domain enhancement model to obtain the sample broadband enhanced amplitude spectrum of each speaker.

[0515] In the seventh step, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum are fused using the fusion model to obtain the sample broadband enhanced amplitude spectrum, including:

[0516] Dividing the sample wideband amplitude spectrum, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum into a plurality of sub-bands having the same number;

[0517] Determine the subband weight of the subband corresponding to each sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum based on the sample broadband amplitude spectrum, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum; wherein the sum of the subband weights of the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum of each subband is 1;

[0518] The subband weights are smoothed in the time dimension using the exponential moving average method;

[0519] Based on the subband weight of each smoothed subband and the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum corresponding to the multiple subbands, the sample broadband enhanced amplitude spectrum is determined.

[0520] In specific implementation, the method can be implemented by dividing into 16 sub-bands, and is not limited to the implementation method of 16 sub-bands. Taking the division into 16 sub-bands as an example:

[0521] In order to utilize the complementary information between the first frequency domain broadband enhanced amplitude spectrum of the sample and the second frequency domain broadband enhanced amplitude spectrum of the sample obtained by the first frequency domain enhancement model and the second frequency domain enhancement model, the predicted first frequency domain enhanced amplitude spectrum of the sample is fused by estimating the weights of the neural network in the fusion model. And the second frequency domain enhanced amplitude spectrum of the samples

[0522] The sample broadband amplitude spectrum, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum are divided into 16 sub-bands, and a weight is estimated for each sub-band. and The sum of the weights corresponding to each identical subband (e.g., the first subband, the 16th subband, etc. are identical subbands in the 16 subbands) is equal to 1. The fusion model is based on the sample broadband amplitude spectrum |Y| 16 , and is input and 16 weight coefficients are predicted.

[0523] therefore, Fig.15 The wideband amplitude spectrum of the sample |Y| 16 The input fusion enhancement model is only for estimating weights and is not used for fusion.

[0524] At the same time, in order to avoid weight jumps, the weights are processed as follows in the time domain:

[0525] The exponential moving average method is used to smooth the weights in the time dimension, and the decay coefficient can be 0.95, but is not limited to this.

[0526] Dividing into 16 sub-bands means processing the weights in the frequency domain, and smoothing in the time dimension means processing the weights in the time domain.

[0527] By putting and The corresponding sub-bands are multiplied by the corresponding weights and added to obtain the fused sample broadband enhanced amplitude spectrum.

[0528] Step 8: Determine the value of the fusion enhancement loss function based on the sample wideband enhanced amplitude spectrum of each speaker, and the corresponding sample reference wideband amplitude spectrum, the sample reference time domain signal and the sample third enhanced time domain signal.

[0529] In the eighth step, the value of the fusion enhancement loss function is determined based on the sample broadband enhanced amplitude spectrum of each speaker, and the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample third enhanced time domain signal, including:

[0530] determining a scale-invariant third signal-to-loss ratio between the sample third enhanced time domain signal and the sample reference time domain signal;

[0531] determining a third magnitude spectrum loss between the sample wideband enhanced magnitude spectrum and the sample reference wideband magnitude spectrum;

[0532] Determine a third loss weight corresponding to a third amplitude spectrum loss according to the sample reference broadband amplitude spectrum and the corresponding sample broadband amplitude spectrum;

[0533] The value of the fusion enhancement loss function is determined according to the third signal-to-loss ratio, the third amplitude spectrum loss and the third loss weight.

[0534] The expression of the fusion enhancement loss function is the same as formula (5), but the content corresponding to each character used for calculation is different, where: Sample reference broadband amplitude spectrum |S| = |S| 16 , the sample reference time domain signal s is determined based on the corresponding clean speech signal, that is, s(t) in the data pair.

[0535] in, is the sample broadband enhanced magnitude spectrum, |S| 16 is the sample reference broadband amplitude spectrum, s is the sample reference time domain signal, is the third enhanced time domain signal of the sample.

[0536] Step 9. Adjust the parameters of the fusion enhancement model based on the value of the fusion enhancement loss function to achieve training of the fusion enhancement model.

[0537] When the fusion enhancement model is trained, the training of the fusion enhancement model is terminated until the value of the fusion loss function no longer decreases in two consecutive iterations in the sample set, or the value of the fusion enhancement loss function is less than a third preset threshold.

[0538] In the embodiments of the present disclosure, there is no limitation on the specific implementation process of adjusting the parameters of the fusion enhancement model based on the value of the fusion enhancement loss function.

[0539] After the fusion enhancement model training is completed, the high-frequency enhancement model and the speaking object amplitude spectrum extraction model can be trained simultaneously, or one can be trained first and then the other. This is not limited in the embodiments of the present disclosure.

[0540] The training processes of the two models are given below, wherein steps 10 to 12 are the training of the high-frequency enhancement model, and steps 13 to 18 are the training processes of the speaker amplitude spectrum extraction model.

[0541] Step 10: Use the high-frequency enhancement model to perform high-frequency speech enhancement processing on the sample high-frequency amplitude spectrum of each speaker and the sample broadband enhanced amplitude spectrum obtained after the fusion model training is completed, so as to obtain the sample high-frequency enhanced amplitude spectrum of each speaker.

[0542] The high-frequency enhancement model uses two parallel branches of convolutional neural networks to enhance the amplitude spectrum of the input sample broadband. and the sample high frequency amplitude spectrum |Y| 16-48 The broadband speech features and high-frequency speech features are learned, and then the broadband and high-frequency speech features are combined using recurrent neural networks and forward neural networks to predict the high-frequency speech spectrum corresponding to the speech.

[0543] Step 11: Determine the value of the high-frequency enhancement loss function based on the sample high-frequency enhancement amplitude spectrum and the sample reference high-frequency amplitude spectrum of each speaker, wherein the sample reference high-frequency amplitude spectrum of each speaker is determined based on the pure speech signal of the corresponding speaker.

[0544] We use formula (4) as the loss function to train the high-frequency enhancement model. and |S| 16-48 is the sample reference high frequency amplitude spectrum.

[0545] Step 12: Adjust the parameters of the high-frequency enhancement model based on the value of the high-frequency enhancement loss function to achieve training of the high-frequency enhancement model. End this process.

[0546] When training the high-frequency enhancement model, the training of the high-frequency enhancement model is terminated until the value of the high-frequency enhancement loss function no longer decreases in two consecutive iterations in the sample set, or the value of the high-frequency enhancement loss function is less than the fourth preset threshold.

[0547] In the embodiments of the present disclosure, the specific implementation process of adjusting the parameters of the high-frequency enhancement model based on the value of the high-frequency enhancement loss function is not limited.

[0548] Step 13: Use the speaker voiceprint extraction model to encode the sample speech signal of each calibrated speaker to obtain the speaker voiceprint of each calibrated speaker.

[0549] The speaker voiceprint extraction model is used to encode x(t) to obtain the speaker voiceprint e that can describe the speaker's voiceprint vector. The specific implementation process is as follows:

[0550] The Mel-domain Mel-cepstrum features are used as input, and the frame-based Mel-cepstrum features are processed by a multi-layer network g(·) to learn the frame-level features R(n,i), where i is the number of nodes in the neural network layer.

[0551] R(n,i)=g(Mel(|STFT(x(t))|)) (7)

[0552] After obtaining the frame-level features, attentive statistical pooling is used to obtain the speaker's voiceprint (a vector) of the speaker at the sentence level.

[0553] e=statisticpooing(R(n,i)) (8)

[0554] Step 14: Use a classifier to perform speaker identity discrimination processing on multiple speaker voiceprints of each calibrated speaker to obtain an estimated identity prediction probability corresponding to each speaker voiceprint; determine the value of the first cross entropy loss function based on the identity prediction probability corresponding to each speaker voiceprint and the label of the speaker's true identity corresponding to each speaker voiceprint.

[0555] The specific expression of the first cross entropy loss function is as follows:

[0556]

[0557] Wherein, C is the total number of speakers in the training data. When the speaker voiceprint e is the speaker c, P is equal to 1, otherwise it is equal to 0. p(c|e) is the probability that the speech is predicted to be the speaker c.

[0558] Step 15: Adjust the parameters of the classifier and the speaker voiceprint extraction model according to the value of the first cross entropy loss function to achieve preliminary training of the speaker voiceprint extraction model and training of the classifier.

[0559] When training the voiceprint extraction model and classifier of the speaker, the training of the voiceprint extraction model and classifier of the speaker is terminated until the value of the first cross entropy loss function no longer decreases for two consecutive iterations in the sample set, or the value of the first cross entropy loss function is less than the fifth preset threshold.

[0560] Step 16: Use the calibrated speaking object extraction model to perform amplitude spectrum extraction processing on the speaking object voiceprint, the calibrated speaking object label and the sample broadband enhanced amplitude spectrum of each calibrated speaking object to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaking object.

[0561] In the specific implementation, a feedforward neural network, GRU and a dilated convolutional network h(·) are used to calibrate the speaker’s voiceprint information e as reference information, and the broadband enhanced amplitude spectrum Extracting the wideband enhanced amplitude spectrum of the sample potential speech of the calibrated speaker The specific calculation formula is:

[0562]

[0563] Step 17, determine the scale-invariant fourth signal-to-loss ratio between the sample fourth enhanced time domain signal and the sample reference time domain signal; wherein the sample fourth enhanced time domain signal is determined based on the sample potential speech wideband enhanced amplitude spectrum; determine the fifth amplitude spectrum loss between the sample potential speech wideband enhanced amplitude spectrum and the corresponding sample reference wideband amplitude spectrum; determine the fifth loss weight corresponding to the fifth amplitude spectrum loss based on the sample reference wideband amplitude spectrum and the corresponding sample wideband amplitude spectrum; determine the value of the second cross-entropy loss function based on the initially trained speaking object voiceprint extraction model and the trained classifier; determine the value of the calibrated speaking object extraction loss function based on the fourth signal-to-loss ratio, the fifth amplitude spectrum loss, the fifth loss weight and the value of the second cross-entropy loss function.

[0564] Among them, the loss function for calibrating the speaking object extraction is expressed as follows:

[0565]

[0566] In formula (11), M = (|S| / |Y|) γ , M is the ideal amplitude mask (IAM). In the first frequency domain enhancement loss function, M is used to balance the importance of speech and noise. The smaller the value of M, the greater the corresponding loss weight e. (a / (b+M))a, b and γ are adjustable weights, such as 2, 1 and 1, respectively, but there is no restriction on this.

[0567] in, is the sample potential speech wideband enhanced magnitude spectrum; |S| = |S| 16 ; Sample fourth enhanced time domain signal iSTFT stands for inverse Fourier transform; the sample reference time domain signal s is determined based on the corresponding pure speech signal, i.e., in the data pair, and p(c|e) is determined based on the trained classifier.

[0568] The sample potential speech wideband enhancement amplitude spectrum is calculated The dot product of the fifth amplitude spectrum loss and the fifth loss weight between the corresponding sample reference broadband amplitude spectrum |S|, the sample potential speech broadband enhanced amplitude spectrum The smaller the difference between the corresponding sample reference wideband amplitude spectrum |S|, the closer the enhanced speech amplitude spectrum is to the reference speech. In order to reduce the range of amplitude spectrum values, we compress the amplitude spectrum in the logarithmic domain, i.e., ln(·).

[0569] The fourth time domain enhanced signal is calculated and the scale-invariant fourth signal-to-loss ratio between the reference time domain signal s.

[0570] The larger the scale-invariant signal-to-loss ratio is, the closer the enhanced signal is to the reference time domain signal. Since the loss function is minimized in the embodiment of the present disclosure, the negative of the fourth scale-invariant signal-to-loss ratio is taken as the loss.

[0571] Step 18: Adjust the parameters of the speaking object voiceprint extraction model and the calibrated speaking object extraction model according to the value of the calibrated speaking object extraction loss function to achieve the training of the speaking object voiceprint extraction model and the calibrated speaking object extraction model.

[0572] When training the speaking object voiceprint extraction model and the calibrated speaking object extraction model, the training of the speaking object voiceprint extraction model and the calibrated speaking object extraction model is terminated until the value of the speaking object amplitude spectrum extraction loss function no longer decreases in the sample set for two consecutive iterations, or the value of the calibrated speaking object extraction loss function is less than the sixth preset threshold.

[0573] In the embodiments of the present disclosure, there is no limitation on the specific implementation process of adjusting the speaking object voiceprint extraction model and calibrating the parameters of the speaking object extraction model based on the value of the speaking object amplitude spectrum extraction loss function.

[0574] Completed Fig.15Training of the speech enhancement model shown.

[0575] After the model training is completed, speech enhancement processing can be performed in actual scenarios.

[0576] After the personalized speech training model is trained, you can set whether to use the speech object amplitude spectrum extraction model according to the specific application scenario. The setting can be configured by the user. You can also pre-configure the model to input the broadband enhanced amplitude spectrum into the speaking object amplitude spectrum extraction model to output the potential speech broadband enhanced amplitude spectrum of the calibrated speaking object when there is a speech signal input of the calibrated speaking object; otherwise, directly output the broadband enhanced amplitude spectrum.

[0577] Fig.16 It is a schematic diagram of a personalized multi-model speech enhancement model structure according to an illustrative embodiment. Fig.16 for Fig.15 After training, the speech enhancement model is based on Fig.16 The model structure shown involves the following speech enhancement processing:

[0578] Inputting the broadband amplitude spectrum corresponding to the speech amplitude spectrum of the noisy speech signal to be processed into the first frequency domain enhancement model and the second frequency domain enhancement model respectively, to obtain the first frequency domain enhancement amplitude spectrum and the second frequency domain enhancement amplitude spectrum respectively;

[0579] Inputting the first frequency domain enhanced amplitude spectrum and the second frequency domain enhanced amplitude spectrum into a fusion model to obtain a broadband enhanced amplitude spectrum;

[0580] Inputting the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum corresponding to the speech amplitude spectrum of the to-be-processed noisy speech signal into the high-frequency enhancement model to obtain the high-frequency enhanced amplitude spectrum;

[0581] Inputting the speech signal of the target speaker into the speaker voiceprint extraction model to obtain the speaker voiceprint of the target speaker;

[0582] The obtained speaking object voiceprint of the target speaking object and the obtained broadband enhanced amplitude spectrum are input into the calibrated speaking object extraction model to obtain the sample potential speech broadband enhanced amplitude spectrum of the target speaking object.

[0583] At this point, the high-frequency enhanced amplitude spectrum of the noisy speech signal to be processed and the broadband enhanced amplitude spectrum of the target speaker's sample potential speech are obtained, which can be used for fusion processing to obtain the full-band enhanced speech signal corresponding to the noisy speech signal to be processed.

[0584] In the embodiment of the present disclosure, the full-band speech amplitude spectrum of the noisy speech signal is divided into a sample high-frequency amplitude spectrum and a sample wideband amplitude spectrum; the wideband enhancement model is trained based on the wideband enhancement amplitude spectrum corresponding to the sample wideband amplitude spectrum obtained by the wideband enhancement model, and the reference wideband amplitude spectrum; after the wideband enhancement model training is completed, the high-frequency enhancement model is trained using the wideband enhancement amplitude spectrum obtained by the trained wideband enhancement model, the sample high-frequency enhancement amplitude spectrum and the reference high-frequency enhancement amplitude spectrum; and the speaking object amplitude spectrum extraction model is trained based on the sample speech signal and the wideband enhancement amplitude spectrum of each calibrated speaking object. The full-band speech enhancement model in the embodiment of the present disclosure is compatible with wideband speech enhancement processing, as well as personalized and non-personalized full-band speech enhancement models, and the wideband amplitude spectrum corresponding to the wideband speech can also be input to obtain the wideband enhancement amplitude spectrum. The speech enhancement model training method can solve the problem that the full-band speech enhancement processing is not compatible with the wideband speech enhancement processing, and the full-band speech enhancement model cannot process the wideband speech input, and also solves the problem that the personalized speech enhancement model and the non-personalized speech enhancement model are not compatible.

[0585] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0586] In specific implementation, the voice enhancement model can be set through the user's configuration information to determine whether to perform full-band voice enhancement or personalized voice enhancement. After the judgment is completed, the function of the corresponding model can be turned off or on to achieve a voice enhancement model that meets the user's needs. It is also possible to add judgments within the model to determine the output content based on the input content, which is not limited in the embodiments of the present application.

[0587] In specific implementation, all functions of speech enhancement processing can be encapsulated into a speech enhancement processing model. As long as the noisy speech signal to be processed is input, the full-band speech enhancement signal can be output. It can also be set whether to output the broadband enhancement amplitude spectrum according to actual needs. Such a speech enhancement processing model can be directly established in a compatible manner with the broadband enhancement model.

[0588] It is also possible to encapsulate a personalized and non-personalized compatible speech enhancement processing model, in which the full-band speech enhancement signal of the target object can be obtained by inputting the noisy speech signal to be processed and the corresponding speech signal of the target speaking object; when only the noisy speech signal to be processed is input, a non-personalized full-band speech enhancement signal is output.

[0589] In the specific implementation, a setting page can also be provided for the model, and the combination of various functional modules can be realized through page setting to meet various practical needs.

[0590] If the user configures to perform full-band and personalized speech enhancement processing, the speech enhancement model is set to execute the speaker amplitude spectrum extraction model and the high-frequency enhancement model;

[0591] When the user configures to perform full-band and non-personalized speech enhancement processing, the speech enhancement model is set to perform a high-frequency enhancement model, and the speaker amplitude spectrum extraction model is not performed;

[0592] When the user configures to perform broadband enhancement processing and personalized speech enhancement processing, the speech enhancement model is set to execute the speaker amplitude spectrum extraction model, and the high-frequency enhancement model is not executed;

[0593] When the user configures to perform broadband and non-personalized speech enhancement processing, the speech enhancement model is set not to execute the speaking object amplitude spectrum extraction model and not to execute the high-frequency enhancement model.

[0594] The speech enhancement model in the embodiment of the present disclosure can simultaneously obtain a broadband enhanced amplitude spectrum and a high-frequency enhanced amplitude spectrum, and is thus compatible with a broadband enhancement network. During input, only the broadband amplitude spectrum can be input, and the broadband enhanced amplitude spectrum can also be obtained through the speech enhancement model. Similarly, when the speech signal of the calibrated speaking object is input, a personalized broadband enhanced amplitude spectrum can be obtained. Therefore, the speech enhancement model in the embodiment of the present disclosure makes the full-band and broadband processing images compatible, and makes personalization and non-personalization compatible, and various incompatibility problems can be solved through a speech enhancement processing model.

[0595] Fig.17 FIG. 1 is a logical structure diagram of a speech enhancement device according to an exemplary embodiment. Fig.17 As shown, the speech enhancement device comprises:

[0596] An acquisition unit 1701 is configured to acquire a speech signal containing noise to be processed;

[0597] The dividing unit 1702 is configured to divide the speech amplitude spectrum of the noisy speech signal to be processed into a broadband amplitude spectrum and a high-frequency amplitude spectrum;

[0598] The first processing unit 1703 is configured to perform broadband enhancement processing on the broadband amplitude spectrum to obtain a broadband enhanced amplitude spectrum;

[0599] The second processing unit 1704 is configured to perform high-frequency enhancement processing on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum to obtain a high-frequency enhanced amplitude spectrum;

[0600] The third processing unit 1705 is configured to perform fusion of the wideband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum, and obtain a full-band enhanced speech signal of the noisy speech signal to be processed according to the fusion result.

[0601] In one possible implementation,

[0602] The first processing unit 1703 is configured to perform high-frequency enhancement processing on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum by using a high-frequency enhancement model when performing high-frequency enhancement processing on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum;

[0603] The training method of the high-frequency enhancement model includes:

[0604] Acquire first training samples of a plurality of speakers and corresponding sample reference high-frequency amplitude spectra, wherein the first training samples include a sample broadband enhanced amplitude spectrum and a sample high-frequency amplitude spectrum of each speaker; the sample broadband enhanced amplitude spectrum and the sample high-frequency amplitude spectrum are determined based on the full-band speech amplitude spectrum of the noisy speech signal of the corresponding speaker, and the sample reference high-frequency amplitude spectrum is determined based on the clean speech signal corresponding to each speaker;

[0605] Using the high-frequency enhancement model, high-frequency enhancement processing is performed on the sample high-frequency amplitude spectrum and the sample broadband enhanced amplitude spectrum of each speaker to obtain the sample high-frequency enhanced amplitude spectrum of each speaker;

[0606] The high frequency enhancement model is trained based on the difference between the sample high frequency enhancement amplitude spectrum and the sample reference high frequency amplitude spectrum of each speaker.

[0607] In one possible implementation,

[0608] The second processing unit 1704 is configured to perform broadband enhancement processing on the broadband amplitude spectrum by using a broadband enhancement model when performing broadband enhancement processing on the broadband amplitude spectrum;

[0609] The training method of the broadband enhancement model includes:

[0610] Acquire second training samples of a plurality of speakers and corresponding sample reference broadband amplitude spectra, wherein the second training samples include a sample broadband amplitude spectrum of each speaker;

[0611] Performing broadband enhancement processing on the sample broadband amplitude spectrum using the broadband enhancement model to obtain a sample broadband enhanced amplitude spectrum of each of the speakers;

[0612] The broadband enhancement model is trained based on the difference between the sample reference broadband amplitude spectrum and the sample broadband enhancement amplitude spectrum of each speaker.

[0613] In one possible implementation,

[0614] The broadband enhancement model includes: a first frequency domain enhancement model, a second frequency domain enhancement model and a fusion model, wherein the first frequency domain enhancement model and the second frequency domain enhancement model are trained models; the first frequency domain enhancement model is established based on a temporal convolutional network; the second frequency domain enhancement model is established based on a convolutional recurrent network;

[0615] The step of performing broadband enhancement processing on the sample broadband amplitude spectrum by using the broadband enhancement model to obtain a sample broadband enhanced amplitude spectrum of each speaker includes:

[0616] Using the first frequency domain enhancement model and the second frequency domain enhancement model to perform frequency domain enhancement processing on the sample broadband amplitude spectrum respectively, to obtain a sample first frequency domain enhanced amplitude spectrum and a sample second frequency domain enhanced amplitude spectrum of each speaker;

[0617] The fusion model is used to perform fusion processing on the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum to obtain the sample broadband enhanced amplitude spectrum.

[0618] In one possible implementation,

[0619] The first frequency domain enhancement model is trained based on a first frequency domain enhancement loss function;

[0620] The first frequency domain enhancement loss function is determined based on the sample first frequency domain enhancement amplitude spectrum of each speaker, and the corresponding sample reference broadband amplitude spectrum, sample reference time domain signal and sample first enhanced time domain signal, wherein the sample first enhanced time domain signal is determined based on the sample first frequency domain enhancement amplitude spectrum, and the sample reference time domain signal is determined based on the corresponding clean speech signal;

[0621] The second frequency domain enhancement model is trained based on a second frequency domain enhancement loss function;

[0622] The second frequency domain enhancement loss function is determined based on the sample second frequency domain enhancement amplitude spectrum of each speaking object, and the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample second enhanced time domain signal; wherein the sample second enhanced time domain signal is determined based on the sample second frequency domain enhancement amplitude spectrum.

[0623] In one possible implementation,

[0624] The training of the broadband enhancement model based on the difference between the sample reference broadband amplitude spectrum and the sample broadband enhancement amplitude spectrum of each speaker includes:

[0625] determining a fusion enhancement loss function for characterizing the difference between the sample reference wideband magnitude spectrum and the sample wideband enhanced magnitude spectrum for each of the speaking objects;

[0626] Training the fusion model based on the fusion enhancement loss function;

[0627] Wherein, the fusion enhancement loss function is determined based on the sample broadband enhanced amplitude spectrum of each speaking object, and the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample third enhanced time domain signal; wherein, the sample third enhanced time domain signal is determined based on the sample broadband enhanced amplitude spectrum, and the sample reference time domain signal is determined based on the corresponding pure speech signal.

[0628] In one possible implementation,

[0629] The step of using the fusion model to fuse the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum to obtain the sample broadband enhanced amplitude spectrum includes:

[0630] Dividing the sample wideband amplitude spectrum, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum into a plurality of sub-bands;

[0631] Based on the sample wideband amplitude spectrum, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum, determine the subband weights of the subbands corresponding to each of the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum; wherein the sum of the subband weights of the same subband corresponding to the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum is 1;

[0632] Performing smoothing processing on each of the sub-band weights in the time dimension;

[0633] The sample broadband enhanced amplitude spectrum is determined based on the subband weight of each smoothed subband and the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum respectively corresponding to a plurality of the subbands.

[0634] In one possible implementation,

[0635] The third processing unit 1705 is configured to perform fusion of the broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum, and when the full-band enhanced speech signal of the noisy speech signal to be processed is obtained according to the fusion result, obtain the sample potential speech broadband enhanced amplitude spectrum of the target speaking object based on the broadband enhanced amplitude spectrum; fuse the potential speech broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum of the target speaking object, and obtain the full-band enhanced speech signal of the noisy speech signal to be processed according to the fusion result.

[0636] In one possible implementation,

[0637] The third processing unit 1705 is configured to obtain calibration speech signals of multiple calibration speaking objects when obtaining the sample potential speech broadband enhancement amplitude spectrum of the target speaking object based on the broadband enhancement amplitude spectrum; determine the target speaking object from the multiple calibration speaking objects; determine the calibration speech signal of the target speaking object according to the calibration speech signal of the calibration speaking object determined as the target speaking object;

[0638] Amplitude spectrum extraction processing is performed on the calibrated speech signal and the broadband enhanced amplitude spectrum of the target speaker to obtain a sample potential speech broadband enhanced amplitude spectrum of the target speaker.

[0639] The third processing unit 1705 is configured to perform amplitude spectrum extraction processing on the calibration speech signal and the broadband enhanced amplitude spectrum of the target speaker, including:

[0640] Performing amplitude spectrum extraction processing on the calibration speech signal and the broadband enhanced amplitude spectrum of the target speaker using a speaker amplitude spectrum extraction model;

[0641] The training method of the speaker amplitude spectrum extraction model includes:

[0642] Acquire a plurality of third training samples of the calibrated speaking objects and sample reference wideband amplitude spectra; wherein the third training samples include a sample wideband enhanced amplitude spectrum and a sample speech signal of each of the calibrated speaking objects;

[0643] Using the speaker amplitude spectrum extraction model, performing amplitude spectrum extraction processing on the sample speech signal and the sample broadband enhanced amplitude spectrum of each calibrated speaker, to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker;

[0644] The training of the speaking object amplitude spectrum extraction model is achieved based on the difference between the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaking object and the corresponding sample reference broadband amplitude spectrum.

[0645] In one possible implementation,

[0646] The speaker amplitude spectrum extraction model includes: a speaker voiceprint extraction model and a calibration speaker extraction model; the speaker voiceprint extraction model is a model that has been preliminarily trained;

[0647] The method of using the speaker amplitude spectrum extraction model to perform amplitude spectrum extraction processing on the sample speech signal and the sample broadband enhanced amplitude spectrum of each calibrated speaker to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker includes:

[0648] Using the speaker voiceprint extraction model, encoding the sample speech signal of each calibrated speaker to obtain the speaker voiceprint of each calibrated speaker;

[0649] The calibration speaking object extraction model is used to perform amplitude spectrum extraction processing on the speaking object voiceprint and the broadband enhanced amplitude spectrum of each calibration speaking object to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibration speaking object.

[0650] In one possible implementation,

[0651] The speaker voiceprint extraction model is trained based on a first cross entropy loss function;

[0652] Among them, the first cross entropy loss function is determined based on the identity prediction probability corresponding to each of the speaker's voiceprints and the label of the speaker's real identity corresponding to each of the speaker's voiceprints, and the identity prediction probability is obtained by using a classifier to perform speaker identity discrimination processing on multiple speaker voiceprints of each of the calibrated speakers; the classifier is trained based on the first cross entropy loss function.

[0653] In one possible implementation,

[0654] The difference between the sample potential speech broadband enhanced amplitude spectrum of each of the calibrated speaking objects and the corresponding sample reference broadband amplitude spectrum is characterized by a second cross entropy loss function and a calibrated speaking object extraction loss function;

[0655] The calibration speaking object extraction loss function is determined based on the sample enhanced time domain signal, the sample reference time domain signal, the sample potential speech wideband enhanced amplitude spectrum and the corresponding sample reference wideband amplitude spectrum, the sample wideband amplitude spectrum, and the trained classifier; wherein the sample enhanced time domain signal is determined according to the sample potential speech wideband enhanced amplitude spectrum;

[0656] The second cross entropy loss function is determined based on the identity prediction probability corresponding to each speaker voiceprint and the label of the real identity of the speaker corresponding to each speaker voiceprint, and the identity prediction probability is obtained by using a classifier to perform speaker identity discrimination processing on multiple speaker voiceprints of each calibrated speaker.

[0657] Regarding the speech enhancement training device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the speech enhancement model training method, and will not be elaborated here.

[0658] It should be noted that: the above embodiment only illustrates the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0659] Fig.18 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. In some embodiments, the electronic device is a local server. The electronic device 1800 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1801 and one or more memories 1802, wherein at least one program code is stored in the memory 1802, and the at least one program code is loaded and executed by the processor 1801 to implement the speech enhancement model training method or speech enhancement method provided in the above-mentioned various embodiments. Of course, the electronic device 1800 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 1800 may also include other components for implementing device functions, which will not be repeated here.

[0660] In an exemplary embodiment, a computer-readable storage medium including at least one instruction is also provided, such as a memory including at least one instruction. The at least one instruction can be executed by a processor in a computer device to implement the speech enhancement method in the above embodiment.

[0661] Optionally, the above-mentioned computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may include ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk and optical data storage device, etc.

[0662] In an exemplary embodiment, a computer program product is also provided, including one or more instructions, which can be executed by a processor of a computer device to implement the speech enhancement method provided in the above-mentioned various embodiments.

[0663] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0664] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A speech enhancement method, characterized in that: The speech enhancement method comprises: Acquire a noisy speech signal to be processed; Dividing the speech amplitude spectrum of the noisy speech signal to be processed into a broadband amplitude spectrum and a high-frequency amplitude spectrum; Performing broadband enhancement processing on the broadband amplitude spectrum to obtain a broadband enhanced amplitude spectrum; Performing high-frequency enhancement processing on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum to obtain a high-frequency enhanced amplitude spectrum; The broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum are fused, and a full-band enhanced speech signal of the noisy speech signal to be processed is obtained according to the fusion result; The step of fusing the broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum to obtain the full-band enhanced speech signal of the noisy speech signal to be processed according to the fusion result includes: Acquire a potential speech broadband enhanced magnitude spectrum of a target speaker based on the broadband enhanced magnitude spectrum; The latent speech wideband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum of the target speaker are fused, and a full-band enhanced speech signal of the noisy speech signal to be processed is obtained according to the fusion result.

2. The speech enhancement method according to claim 1, characterized in that: The performing high-frequency enhancement processing on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum comprises: Performing high-frequency enhancement processing on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum using a high-frequency enhancement model; The training method of the high-frequency enhancement model includes: Acquire first training samples of a plurality of speakers and corresponding sample reference high-frequency amplitude spectra, wherein the first training samples include a sample broadband enhanced amplitude spectrum and a sample high-frequency amplitude spectrum of each speaker; the sample broadband enhanced amplitude spectrum and the sample high-frequency amplitude spectrum are determined based on the full-band speech amplitude spectrum of the noisy speech signal of the corresponding speaker, and the sample reference high-frequency amplitude spectrum is determined based on the clean speech signal corresponding to each speaker; Using the high-frequency enhancement model, high-frequency enhancement processing is performed on the sample high-frequency amplitude spectrum and the sample broadband enhanced amplitude spectrum of each speaker to obtain the sample high-frequency enhanced amplitude spectrum of each speaker; The high frequency enhancement model is trained based on the difference between the sample high frequency enhancement amplitude spectrum and the sample reference high frequency amplitude spectrum of each speaker.

3. The speech enhancement method according to claim 1, characterized in that: The performing broadband enhancement processing on the broadband amplitude spectrum comprises: Performing broadband enhancement processing on the broadband amplitude spectrum using a broadband enhancement model; The training method of the broadband enhancement model includes: Acquire second training samples of a plurality of speakers and corresponding sample reference broadband amplitude spectra, wherein the second training samples include a sample broadband amplitude spectrum of each speaker; Performing broadband enhancement processing on the sample broadband amplitude spectrum using the broadband enhancement model to obtain a sample broadband enhanced amplitude spectrum of each of the speakers; The broadband enhancement model is trained based on the difference between the sample reference broadband amplitude spectrum and the sample broadband enhancement amplitude spectrum of each speaker.

4. The speech enhancement method according to claim 3, characterized in that: The broadband enhancement model includes: a first frequency domain enhancement model, a second frequency domain enhancement model and a fusion model, wherein the first frequency domain enhancement model and the second frequency domain enhancement model are trained models; the first frequency domain enhancement model is established based on a temporal convolutional network; the second frequency domain enhancement model is established based on a convolutional recurrent network; The step of performing broadband enhancement processing on the sample broadband amplitude spectrum by using the broadband enhancement model to obtain a sample broadband enhanced amplitude spectrum of each speaker includes: Using the first frequency domain enhancement model and the second frequency domain enhancement model to perform frequency domain enhancement processing on the sample broadband amplitude spectrum respectively, to obtain a sample first frequency domain enhanced amplitude spectrum and a sample second frequency domain enhanced amplitude spectrum of each speaker; The fusion model is used to perform fusion processing on the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum to obtain the sample broadband enhanced amplitude spectrum.

5. The speech enhancement method according to claim 4, characterized in that: The first frequency domain enhancement model is trained based on a first frequency domain enhancement loss function; The first frequency domain enhancement loss function is determined based on the sample first frequency domain enhancement amplitude spectrum of each speaker, and the corresponding sample reference broadband amplitude spectrum, sample reference time domain signal and sample first enhanced time domain signal, wherein the sample first enhanced time domain signal is determined based on the sample first frequency domain enhancement amplitude spectrum, and the sample reference time domain signal is determined based on the corresponding clean speech signal; The second frequency domain enhancement model is trained based on a second frequency domain enhancement loss function; The second frequency domain enhancement loss function is determined based on the sample second frequency domain enhancement amplitude spectrum of each speaking object, and the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample second enhanced time domain signal; wherein the sample second enhanced time domain signal is determined based on the sample second frequency domain enhancement amplitude spectrum.

6. The speech enhancement method according to claim 4, characterized in that: The training of the broadband enhancement model based on the difference between the sample reference broadband amplitude spectrum and the sample broadband enhancement amplitude spectrum of each speaker includes: determining a fusion enhancement loss function for characterizing the difference between the sample reference wideband magnitude spectrum and the sample wideband enhanced magnitude spectrum for each of the speaking objects; Training the fusion model based on the fusion enhancement loss function; Wherein, the fusion enhancement loss function is determined based on the sample broadband enhanced amplitude spectrum, the corresponding sample reference broadband amplitude spectrum, the sample reference time domain signal and the sample third enhanced time domain signal of each speaking object; wherein, the sample third enhanced time domain signal is determined based on the sample broadband enhanced amplitude spectrum, and the sample reference time domain signal is determined based on the corresponding pure speech signal.

7. The speech enhancement method according to claim 4, characterized in that: The step of using the fusion model to fuse the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum to obtain the sample broadband enhanced amplitude spectrum includes: Dividing the sample wideband amplitude spectrum, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum into a plurality of sub-bands; Based on the sample wideband amplitude spectrum, the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum, determine the subband weights of the subbands corresponding to each of the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum; wherein the sum of the subband weights of the same subband corresponding to the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum is 1; Performing smoothing processing on each of the sub-band weights in the time dimension; The sample broadband enhanced amplitude spectrum is determined based on the subband weight of each smoothed subband and the sample first frequency domain enhanced amplitude spectrum and the sample second frequency domain enhanced amplitude spectrum respectively corresponding to a plurality of the subbands.

8. The speech enhancement method according to claim 1, characterized in that: The step of obtaining a sample potential speech broadband enhanced amplitude spectrum of a target speaker based on the broadband enhanced amplitude spectrum comprises: Acquire calibrated speech signals of multiple calibrated speaking objects; Determining the target speaking object among the plurality of calibrated speaking objects; Determine the calibration voice signal of the target speaking object according to the calibration voice signal of the calibration speaking object determined as the target speaking object; Amplitude spectrum extraction processing is performed on the calibrated speech signal and the broadband enhanced amplitude spectrum of the target speaker to obtain a sample potential speech broadband enhanced amplitude spectrum of the target speaker.

9. The speech enhancement method according to claim 8, characterized in that: The performing amplitude spectrum extraction processing on the calibration speech signal and the broadband enhanced amplitude spectrum of the target speaker comprises: Performing amplitude spectrum extraction processing on the calibration speech signal and the broadband enhanced amplitude spectrum of the target speaker using a speaker amplitude spectrum extraction model; The training method of the speaker amplitude spectrum extraction model includes: Acquire a plurality of third training samples of the calibrated speaking objects and sample reference wideband amplitude spectra; wherein the third training samples include a sample wideband enhanced amplitude spectrum and a sample speech signal of each of the calibrated speaking objects; Using the speaker amplitude spectrum extraction model, the sample speech signal and the sample broadband enhanced amplitude spectrum of each calibrated speaker are subjected to amplitude spectrum extraction processing to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker; The training of the speaking object amplitude spectrum extraction model is achieved based on the difference between the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaking object and the corresponding sample reference broadband amplitude spectrum.

10. The speech enhancement method according to claim 9, characterized in that: The speaker amplitude spectrum extraction model includes: a speaker voiceprint extraction model and a calibration speaker extraction model; the speaker voiceprint extraction model is a model that has been preliminarily trained; The method of using the speaker amplitude spectrum extraction model to perform amplitude spectrum extraction processing on the sample speech signal and the sample broadband enhanced amplitude spectrum of each calibrated speaker to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibrated speaker includes: Using the speaker voiceprint extraction model, encoding the sample speech signal of each calibrated speaker to obtain the speaker voiceprint of each calibrated speaker; The calibration speaking object extraction model is used to perform amplitude spectrum extraction processing on the speaking object voiceprint and the broadband enhanced amplitude spectrum of each calibration speaking object to obtain the sample potential speech broadband enhanced amplitude spectrum of each calibration speaking object.

11. The speech enhancement method according to claim 10, characterized in that: The speaker voiceprint extraction model is trained based on a first cross entropy loss function; Among them, the first cross entropy loss function is determined based on the identity prediction probability corresponding to each of the speaker's voiceprints and the label of the speaker's real identity corresponding to each of the speaker's voiceprints, and the identity prediction probability is obtained by using a classifier to perform speaker identity discrimination processing on multiple speaker voiceprints of each of the calibrated speakers; the classifier is trained based on the first cross entropy loss function.

12. The speech enhancement method according to claim 10, characterized in that: The difference between the sample potential speech broadband enhanced amplitude spectrum of each of the calibrated speaking objects and the corresponding sample reference broadband amplitude spectrum is characterized by a second cross entropy loss function and a calibrated speaking object extraction loss function; The calibration speaking object extraction loss function is determined based on the sample enhanced time domain signal, the sample reference time domain signal, the sample potential speech wideband enhanced amplitude spectrum and the corresponding sample reference wideband amplitude spectrum, the sample wideband amplitude spectrum, and the trained classifier; wherein the sample enhanced time domain signal is determined according to the sample potential speech wideband enhanced amplitude spectrum; The second cross entropy loss function is determined based on the identity prediction probability corresponding to each of the speaker's voiceprints and the label of the speaker's true identity corresponding to each of the speaker's voiceprints. The identity prediction probability is obtained by using a classifier to perform speaker identity discrimination processing on multiple speaker voiceprints of each of the calibrated speakers.

13. A speech enhancement device, characterized in that: The speech enhancement device comprises: An acquisition unit, configured to acquire a speech signal containing noise to be processed; A dividing unit, configured to divide the speech amplitude spectrum of the noisy speech signal to be processed into a broadband amplitude spectrum and a high-frequency amplitude spectrum; A first processing unit is configured to perform broadband enhancement processing on the broadband amplitude spectrum to obtain a broadband enhanced amplitude spectrum; A second processing unit is configured to perform high-frequency enhancement processing on the broadband enhanced amplitude spectrum and the high-frequency amplitude spectrum to obtain a high-frequency enhanced amplitude spectrum; a third processing unit, configured to perform fusion of the broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum, and obtain a full-band enhanced speech signal of the noisy speech signal to be processed according to the fusion result; in, The third processing unit is specifically configured to perform fusion of the broadband enhanced amplitude spectrum and the high-frequency enhanced amplitude spectrum, and when the full-band enhanced speech signal of the noisy speech signal to be processed is obtained according to the fusion result, obtain the potential broadband enhanced amplitude spectrum of the target speaking object based on the broadband enhanced amplitude spectrum; fuse the potential broadband enhanced amplitude spectrum of the target speaking object and the high-frequency enhanced amplitude spectrum, and obtain the full-band enhanced speech signal of the noisy speech signal to be processed according to the fusion result.

14. An electronic device, characterized in that: include: processor; a memory for storing executable instructions for the processor; The processor is configured to execute the executable instructions to implement the speech enhancement method as described in any one of claims 1-12.

15. A computer-readable storage medium, characterized in that: When at least one instruction in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to implement the speech enhancement method as described in any one of claims 1 to 12.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the speech enhancement method according to any one of claims 1 to 12 is implemented.

Images