Voice enhancement model training method and device, storage medium and electronic device

Abstract

This application discloses a training method, apparatus, storage medium, and electronic device for a speech enhancement model, relating to the field of smart home technology. The training method for the speech enhancement model includes: acquiring a first encoding result of a preset encoder for a first speech data, and a second encoding result of the preset encoder for a second speech data; wherein the first speech data is speech data after processing the original audio by the speech enhancement model, and the second speech data is clean speech data; calculating the first encoding result and the second encoding result according to the target algorithm corresponding to the loss function to determine the difference data between the first encoding result and the second encoding result; training the speech enhancement model at least using the difference data; the above technical solution solves the problem that existing speech enhancement models in related technologies only consider speech distortion, resulting in low accuracy of the output results of the speech enhancement model.

Description

Technical Field

[0001] This application relates to the field of smart home technology, and more specifically, to a training method and apparatus for a voice enhancement model, a storage medium, and an electronic device. Background Technology

[0002] In related technologies, most speech enhancement methods aim to improve speech intelligibility, and subjective and objective evaluation methods for speech enhancement methods mainly revolve around speech quality. After speech enhancement, in addition to making the speech sound clearer, the speech recognition system also needs to maintain a certain level of robustness in complex noisy environments. Therefore, the evaluation metrics of speech recognition systems—word error rate and character error rate—also indirectly reflect the performance of speech enhancement in noisy environments. Currently, the more advanced speech enhancement solutions mainly use neural networks and deep learning methods. Deep learning models primarily use minimum mean square error, taking the signal-to-noise ratio and signal distortion ratio as inputs to the loss function. This method only considers the degree of signal distortion, but speech distortion and speech recognition rate are not always strongly correlated. Therefore, in practical applications, even if the audio achieves good scores in both subjective and objective evaluation criteria, the speech recognition rate may not necessarily improve, and may even decrease.

[0003] To address the issue of insufficient improvement in speech recognition rates, current engineering applications often compensate for this gap by adding speech-enhanced audio to the training set of the speech recognition model. However, since speech recognition models are often large and have long training cycles, this approach is relatively time-consuming, labor-intensive, and impractical.

[0004] In response to the problem that existing speech enhancement models only consider speech distortion, resulting in low accuracy of the output results, no effective solution has yet been proposed.

[0005] Therefore, it is necessary to improve the relevant technology to overcome the aforementioned defects. Summary of the Invention

[0006] This invention provides a training method and apparatus, storage medium and electronic device for a speech enhancement model, to at least solve the problem that existing speech enhancement models in the related art only consider speech distortion, resulting in low accuracy of the output results of the speech enhancement model.

[0007] According to one aspect of the present invention, a method for training a speech enhancement model is provided, comprising: acquiring a first encoding result of a preset encoder for first speech data, and a second encoding result of the preset encoder for second speech data; wherein the first speech data is speech data after processing original audio by a speech enhancement model, the second speech data is clean speech data, and the first speech data is second speech data with added noise; calculating the first encoding result and the second encoding result to determine a first difference data between the first encoding result and the second encoding result; and training the speech enhancement model at least using the first difference data to obtain a trained speech enhancement model.

[0008] In an exemplary embodiment, before obtaining the first encoding result of the preset encoder on the first speech data, the method further includes: obtaining the first time-frequency mask output by the speech enhancement model after processing the original audio; and using the multiplication result of the first time-frequency mask and the original audio as the first speech data.

[0009] In an exemplary embodiment, before obtaining the first encoding result of the preset encoder on the first speech data, the method further includes: performing feature extraction on the original audio to obtain frequency features after feature extraction; inputting the frequency features into the speech enhancement model to obtain a second time-frequency mask corresponding to the original audio; and using the multiplication result of the second time-frequency mask and the original audio as the first speech data.

[0010] In an exemplary embodiment, calculating the first encoding result and the second encoding result to determine the first difference data between the first encoding result and the second encoding result includes: determining a target algorithm corresponding to a loss function, wherein the target algorithm includes at least a minimum mean square error algorithm; inputting the first encoding result and the second encoding result into the target algorithm to determine the first difference data between the first encoding result and the second encoding result according to the target algorithm.

[0011] In an exemplary embodiment, after training the speech enhancement model using at least the first difference data to obtain a trained speech enhancement model, the method further includes: if the first difference data is greater than a preset threshold, performing the following training steps: inputting the original audio into the trained speech enhancement model for processing, and inputting the obtained third speech data into a preset encoder for encoding to obtain a third encoding result; calculating the third encoding result and the second encoding result to determine the second difference data between the third encoding result and the second encoding result; and training the trained speech enhancement model using the second difference data to obtain a retrained speech enhancement model.

[0012] In an exemplary embodiment, after retraining the speech enhancement model using the second difference data to obtain a retrained speech enhancement model, the method further includes: if the second difference data is greater than the preset threshold, performing the training step again; if the second difference data is less than the preset threshold, determining that the retrained speech enhancement model is a trained speech enhancement model.

[0013] In an exemplary embodiment, after determining that the retrained speech enhancement model is a trained speech enhancement model, the method further includes: upon receiving the original audio again, processing the received original audio according to the trained speech enhancement model to obtain enhanced speech of the original audio.

[0014] In an exemplary embodiment, after determining that the retrained speech enhancement model is a trained speech enhancement model, the method further includes: upon receiving the original audio again, processing the received original audio according to the trained speech enhancement model to obtain estimated speech data; mapping the estimated speech data to the speech domain of the original audio to obtain mapped speech features; and performing an inverse Fourier transform on the mapped speech features to obtain enhanced speech of the original audio.

[0015] In an exemplary embodiment, obtaining a first encoding result of a preset encoder on first speech data and a second encoding result of the preset encoder on second speech data includes: obtaining a first encoding result of an encoder in an end-to-end speech recognition framework on first speech data and a second encoding result of an encoder in an end-to-end speech recognition framework on second speech data.

[0016] According to another aspect of the present invention, a training apparatus for a speech enhancement model is also provided, comprising: a first acquisition module, configured to acquire a first encoding result of a preset encoder for first speech data, and a second encoding result of the preset encoder for second speech data; wherein the first speech data is speech data after processing original audio by a speech enhancement model, the second speech data is clean speech data, and the first speech data is second speech data with added noise; configured to calculate the first encoding result and the second encoding result to determine a first difference data between the first encoding result and the second encoding result; and a training module, configured to train the speech enhancement model at least using the first difference data to obtain a trained speech enhancement model.

[0017] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, wherein the computer program is configured to execute the training method of the above-described speech enhancement model at runtime.

[0018] According to another aspect of the present invention, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the training method of the speech enhancement model through the computer program.

[0019] This invention obtains a first encoding result of a preset encoder on a first speech data, and a second encoding result of the preset encoder on a second speech data; wherein the first speech data is speech data after processing the original audio by a speech enhancement model, the second speech data is clean speech data, and the first speech data is second speech data with added noise; the first encoding result and the second encoding result are calculated to determine a first difference data between the first encoding result and the second encoding result; the speech enhancement model is trained at least using the first difference data to obtain a trained speech enhancement model; by adopting the above technical solution, the problem that existing speech enhancement models in related technologies only consider speech distortion, resulting in low accuracy of the output results of the speech enhancement model is solved; the technical effect of improving the accuracy of the output results of the speech enhancement model is achieved. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the hardware environment for an optional training method of a speech enhancement model according to an embodiment of this application;

[0023] Figure 2 This is a flowchart of an optional training method for a speech enhancement model according to an embodiment of this application;

[0024] Figure 3 This is a flowchart illustrating an optional training method for a speech enhancement model according to an embodiment of this application;

[0025] Figure 4 This is a structural block diagram of a training device for an optional speech enhancement model according to an embodiment of the present invention;

[0026] Figure 5 This is a structural block diagram of a training device for another optional speech enhancement model according to an embodiment of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] According to one aspect of the embodiments of this application, a method for training a voice enhancement model is provided. This method for training the voice enhancement model is widely applicable to whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligencehouse ecosystems. Optionally, in this embodiment, the above-mentioned method for training the voice enhancement model can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1 As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0030] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0031] This embodiment provides a training method for a speech enhancement model, including but not limited to applications on a server. Figure 2 This is a flowchart of a training method for a speech enhancement model according to an embodiment of the present invention, the process including the following steps:

[0032] Step S202: Obtain the first encoding result of the preset encoder on the first speech data, and the second encoding result of the preset encoder on the second speech data; wherein, the first speech data is the speech data after the original audio has been processed by the speech enhancement model, the second speech data is the clean speech data, and the first speech data is the second speech data after adding noise;

[0033] Step S204: Calculate the first encoding result and the second encoding result to determine the first difference data between the first encoding result and the second encoding result;

[0034] Step S206: Train the speech enhancement model using at least the first difference data to obtain the trained speech enhancement model.

[0035] Through the above steps, a first encoding result of the preset encoder on the first speech data and a second encoding result of the preset encoder on the second speech data are obtained; wherein, the first speech data is the speech data after the original audio has been processed by the speech enhancement model, the second speech data is the clean speech data, and the first speech data is the second speech data after noise has been added; the first encoding result and the second encoding result are calculated to determine the first difference data between the first encoding result and the second encoding result; the difference data is input into the speech enhancement model to train the speech enhancement model at least through the first difference data to obtain the trained speech enhancement model; by adopting the above technical solution, the problem that the existing speech enhancement models in related technologies only consider speech distortion, resulting in low accuracy of the output results of the speech enhancement models is solved; the technical effect of improving the accuracy of the output results of the speech enhancement model is achieved.

[0036] In an exemplary embodiment, before obtaining the first encoding result of the preset encoder on the first speech data, the method further includes: obtaining the first time-frequency mask output by the speech enhancement model after processing the original audio; and using the multiplication result of the first time-frequency mask and the original audio as the first speech data.

[0037] Before obtaining the first encoding result of the preset encoder on the first speech data, the original audio needs to be input into the speech enhancement model for processing. The speech enhancement model outputs the first time-frequency mask. The obtained first time-frequency mask and the original audio are subjected to matrix multiplication. The result obtained after the operation is used as the first speech data for subsequent processing.

[0038] It should be noted that the above speech enhancement model can be a recurrent neural network consisting of four layers of gated recurrent units, or it can be other types of neural networks. This application does not limit this.

[0039] Optionally, before performing step S202 above: obtaining the first encoding result of the preset encoder on the first speech data, the method further includes: extracting features from the original audio to obtain frequency features after feature extraction; inputting the frequency features into the speech enhancement model to obtain a second time-frequency mask corresponding to the original audio; and using the multiplication result of the second time-frequency mask and the original audio as the first speech data.

[0040] Optionally, the method for obtaining the first speech data may further include: first performing a short-time Fourier transform on the original audio, then performing feature filtering and extraction through a Mel filter to obtain the extracted frequency features; then inputting the obtained frequency features into a speech enhancement model to obtain a second time-frequency mask corresponding to the original audio; performing matrix multiplication on the obtained second time-frequency mask and the original audio; and using the result obtained after the operation as the first speech data for subsequent processing.

[0041] Optionally, step S204 above: calculating the first encoding result and the second encoding result to determine the first difference data between the first encoding result and the second encoding result can be implemented by the following scheme, specifically including: determining the target algorithm corresponding to the loss function, wherein the target algorithm includes at least the minimum mean square error algorithm; inputting the first encoding result and the second encoding result into the target algorithm to determine the first difference data between the first encoding result and the second encoding result according to the target algorithm.

[0042] After obtaining the first and second encoding results, it is necessary to first determine the target algorithm corresponding to the loss function used to calculate the loss. If the target algorithm includes the minimum mean square error algorithm, the first and second encoding results are calculated according to the minimum mean square error algorithm corresponding to the loss function to obtain the difference data between the first and second encoding results. The speech enhancement model is then retrained based on the obtained difference data.

[0043] Based on the above steps, after training the speech enhancement model using at least the first difference data to obtain the trained speech enhancement model, the method further includes: when the first difference data is greater than a preset threshold, performing the following training steps: inputting the original audio into the trained speech enhancement model for processing, and inputting the obtained third speech data into a preset encoder for encoding to obtain a third encoding result; calculating the third encoding result and the second encoding result to determine the second difference data between the third encoding result and the second encoding result; and training the trained speech enhancement model using the second difference data to obtain a retrained speech enhancement model.

[0044] After training the speech enhancement model using the first difference data, in order to achieve the expected effect after training, if the first difference data is greater than a preset threshold, the following training steps are performed: inputting the original audio into the trained speech enhancement model for processing, and inputting the obtained third speech data into a preset encoder for encoding to obtain a third encoding result, calculating the third encoding result and the second encoding result using a target algorithm to determine the second difference data between the third encoding result and the second encoding result, and training the trained speech enhancement model using the second difference data to obtain a retrained speech enhancement model.

[0045] Based on the above training steps, after training the speech enhancement model again using the second difference data to obtain the retrained speech enhancement model, the method further includes: if the second difference data is greater than the preset threshold, performing the training steps again; if the second difference data is less than the preset threshold, determining that the retrained speech enhancement model is a completed speech enhancement model.

[0046] After retraining, it is necessary to determine whether the second difference data obtained is greater than a preset threshold. If the second difference data is still greater than the preset threshold, it means that the speech enhancement model has not achieved the expected training effect. In this case, the above training steps need to be repeated, that is, the original audio is input into the retrained speech enhancement model for training. After the above processing steps, a new difference data is obtained until the new difference data is less than the preset threshold. If the second difference data is less than the preset threshold, the retrained speech enhancement model is determined to be a successfully trained speech enhancement model.

[0047] Furthermore, after determining that the retrained speech enhancement model is a trained speech enhancement model, the method further includes: upon receiving the original audio again, processing the received original audio according to the trained speech enhancement model to obtain enhanced speech of the original audio.

[0048] Once the trained speech enhancement model is obtained, the original audio is enhanced using the trained speech enhancement model. Then, the original audio is processed again using the trained speech enhancement model to obtain the enhanced speech of the original audio.

[0049] Optionally, after determining that the retrained speech enhancement model is the trained speech enhancement model, the method further includes: upon receiving the original audio again, processing the received original audio according to the trained speech enhancement model to obtain estimated speech data; mapping the estimated speech data to the speech domain of the original audio to obtain mapped speech features; and performing an inverse Fourier transform on the mapped speech features to obtain the enhanced speech of the original audio.

[0050] Given a trained speech enhancement model, another possible approach to obtaining enhanced speech from the original audio is to first process the received original audio using the trained speech enhancement model to obtain estimated speech data, perform feature mapping on the estimated speech data to map it back to the original speech domain to obtain mapped speech features, and then perform an inverse Fourier transform on the mapped speech features to obtain the enhanced speech from the original audio.

[0051] The above step S202 can also be implemented through the following steps: obtaining the first encoding result of the preset encoder on the first speech data and the second encoding result of the preset encoder on the second speech data, including: obtaining the first encoding result of the encoder in the end-to-end speech recognition framework on the first speech data and the second encoding result of the encoder in the end-to-end speech recognition framework on the second speech data.

[0052] Optionally, the aforementioned preset encoder can be an encoder in an end-to-end speech recognition framework, that is, to obtain the first encoding result of the encoder in the end-to-end speech recognition framework for the first speech data, and to obtain the second encoding result of the encoder in the end-to-end speech recognition framework for the second speech data.

[0053] Obviously, the embodiments described above are merely some embodiments of the present invention, and not all embodiments. To better understand the training method of the above-described speech enhancement model, the process is explained below with reference to embodiments, but this is not intended to limit the technical solutions of the embodiments of the present invention. Specifically:

[0054] In an optional embodiment, Figure 3 This is a flowchart illustrating an optional training method for a speech enhancement model according to an embodiment of this application, as shown below. Figure 3 As shown, the steps are as follows:

[0055] The original audio undergoes a short-time Fourier transform and then undergoes a Mel filter for feature extraction to obtain Mel frequency features. These Mel frequency features are then input into a neural network, specifically a recurrent neural network consisting of four layers of gated recurrent units. After processing, the neural network outputs a time-frequency mask. Multiplying the original audio and the time-frequency mask yields estimated clean speech features. These estimated clean speech features are then input into the ESPNet encoder for encoding, resulting in an encoded clean speech. Simultaneously, the clean speech is also input into the ESPNet encoder for encoding, resulting in an encoded clean speech. The encoded results of the estimated features and the encoded clean speech are then input into a minimum mean square error loss function to calculate the neural network loss. This loss result is fed back into the neural network for retraining. The trained neural network is used to obtain estimated clean speech features, which are then mapped back to the original speech domain after feature mapping. Finally, a short-time inverse Fourier transform is performed to obtain the enhanced speech.

[0056] Through the above steps, an end-to-end speech recognition framework is introduced into the speech enhancement process to train the speech enhancement model and obtain more accurate speech results. By adopting the above steps, the problem that existing speech enhancement models in related technologies only consider speech distortion, resulting in low accuracy of the output results of the speech enhancement model is solved; the technical effect of improving the accuracy of the output results of the speech enhancement model is achieved.

[0057] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0058] This embodiment also provides a training device for a speech enhancement model, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0059] Figure 4 This is a structural block diagram of an optional speech enhancement model training device according to an embodiment of the present invention, the device comprising:

[0060] The first acquisition module 42 is used to acquire the first encoding result of the preset encoder on the first speech data, and the second encoding result of the preset encoder on the second speech data; wherein, the first speech data is the speech data after the original audio has been processed by the speech enhancement model, the second speech data is the clean speech data, and the first speech data is the second speech data after adding noise.

[0061] The first calculation module 44 calculates the first encoding result and the second encoding result to determine the first difference data between the first encoding result and the second encoding result;

[0062] Training module 46 is used to train the speech enhancement model using at least the first difference data to obtain the trained speech enhancement model.

[0063] The above-described device acquires a first encoding result of a preset encoder on first speech data and a second encoding result of the preset encoder on second speech data. The first speech data is the original audio processed by a speech enhancement model, the second speech data is clean speech data, and the first speech data is the second speech data with added noise. The first encoding result and the second encoding result are calculated to determine a first difference data between them. The speech enhancement model is trained using at least the first difference data to obtain a trained speech enhancement model. This technical solution solves the problem in related technologies where existing speech enhancement models only consider speech distortion, resulting in low accuracy of the output results. It achieves the technical effect of improving the accuracy of the speech enhancement model's output results.

[0064] In an exemplary embodiment, the first acquisition module 42 is further configured to acquire the first time-frequency mask output by the speech enhancement model after processing the original audio; and use the multiplication result of the first time-frequency mask and the original audio as the first speech data.

[0065] Before obtaining the first encoding result of the preset encoder on the first speech data, the original audio needs to be input into the speech enhancement model for processing. The speech enhancement model outputs the first time-frequency mask. The obtained first time-frequency mask and the original audio are subjected to matrix multiplication. The result obtained after the operation is used as the first speech data for subsequent processing.

[0066] Optionally, the first acquisition module 42 is further configured to perform feature extraction on the original audio to obtain frequency features after feature extraction; input the frequency features into the speech enhancement model to obtain a second time-frequency mask corresponding to the original audio; and use the multiplication result of the second time-frequency mask and the original audio as the first speech data.

[0067] Optionally, the method for obtaining the first speech data may further include: first performing a short-time Fourier transform on the original audio, then performing feature filtering and extraction through a Mel filter to obtain the extracted frequency features; then inputting the obtained frequency features into a speech enhancement model to obtain a second time-frequency mask corresponding to the original audio; performing matrix multiplication on the obtained second time-frequency mask and the original audio; and using the result obtained after the operation as the first speech data for subsequent processing.

[0068] Based on the above steps, the first calculation module 44 is further used to determine the target algorithm corresponding to the loss function, wherein the target algorithm includes at least: the minimum mean square error algorithm; inputting the first encoding result and the second encoding result into the target algorithm to determine the first difference data between the first encoding result and the second encoding result according to the target algorithm.

[0069] After obtaining the first and second encoding results, it is necessary to first determine the target algorithm corresponding to the loss function used to calculate the loss. If the target algorithm includes the minimum mean square error algorithm, the first and second encoding results are calculated according to the minimum mean square error algorithm corresponding to the loss function to obtain the difference data between the first and second encoding results. The speech enhancement model is then retrained based on the obtained difference data.

[0070] Based on the above steps, the training module 46 is further configured to perform the following training steps when the first difference data is greater than a preset threshold: inputting the original audio into the trained speech enhancement model for processing, and inputting the obtained third speech data into a preset encoder for encoding to obtain a third encoding result; calculating the third encoding result and the second encoding result to determine the second difference data between the third encoding result and the second encoding result; and training the trained speech enhancement model using the second difference data to obtain a retrained speech enhancement model.

[0071] After training the speech enhancement model using the first difference data, in order to achieve the expected effect after training, if the first difference data is greater than a preset threshold, the following training steps are performed: inputting the original audio into the trained speech enhancement model for processing, and inputting the obtained third speech data into a preset encoder for encoding to obtain a third encoding result, calculating the third encoding result and the second encoding result using a target algorithm to determine the second difference data between the third encoding result and the second encoding result, and training the trained speech enhancement model using the second difference data to obtain a retrained speech enhancement model.

[0072] Based on the above steps, the training module 46 is further configured to execute the training step again when the second difference data is greater than the preset threshold; and to determine the retrained speech enhancement model as the completed speech enhancement model when the second difference data is less than the preset threshold.

[0073] After retraining, it is necessary to determine whether the second difference data obtained is greater than a preset threshold. If the second difference data is still greater than the preset threshold, it means that the speech enhancement model has not achieved the expected training effect. In this case, the above training steps need to be repeated, that is, the original audio is input into the retrained speech enhancement model for training. After the above processing steps, a new difference data is obtained until the new difference data is less than the preset threshold. If the second difference data is less than the preset threshold, the retrained speech enhancement model is determined to be a successfully trained speech enhancement model.

[0074] In addition, the training module 46 is also used to process the received original audio again according to the trained speech enhancement model when the original audio is received again, so as to obtain the enhanced speech of the original audio.

[0075] Once the trained speech enhancement model is obtained, the original audio is enhanced using the trained speech enhancement model. Then, the original audio is processed again using the trained speech enhancement model to obtain the enhanced speech of the original audio.

[0076] Optionally, the training module 46 is further configured to, upon receiving the original audio again, process the received original audio according to the trained speech enhancement model to obtain estimated speech data; map the estimated speech data to the speech domain of the original audio to obtain mapped speech features; and perform inverse Fourier transform on the mapped speech features to obtain enhanced speech of the original audio.

[0077] Given a trained speech enhancement model, another possible approach to obtaining enhanced speech from the original audio is to first process the received original audio using the trained speech enhancement model to obtain estimated speech data, perform feature mapping on the estimated speech data to map it back to the original speech domain to obtain mapped speech features, and then perform an inverse Fourier transform on the mapped speech features to obtain the enhanced speech from the original audio.

[0078] Optionally, the first acquisition module 42 is further configured to acquire the first encoding result of the encoder in the end-to-end speech recognition framework for the first speech data, and the second encoding result of the encoder in the end-to-end speech recognition framework for the second speech data.

[0079] Optionally, the aforementioned preset encoder can be an encoder in an end-to-end speech recognition framework, that is, to obtain the first encoding result of the encoder in the end-to-end speech recognition framework for the first speech data, and to obtain the second encoding result of the encoder in the end-to-end speech recognition framework for the second speech data.

[0080] The above process will be described below with reference to another embodiment, but it is not intended to limit the technical solution of the embodiments of the present invention. For details, see [link to specific embodiments]. Figure 5 , Figure 5 This is a structural block diagram of a training device for another optional speech enhancement model according to an embodiment of the present invention, specifically including:

[0081] Second acquisition module 52: used to acquire the first time-frequency mask output by the speech enhancement model after processing the original audio;

[0082] Second calculation module 54: The result of multiplying the first time-frequency mask and the original audio is used as the first speech data.

[0083] Before obtaining the first encoding result of the preset encoder on the first speech data, the original audio needs to be input into the speech enhancement model for processing. The speech enhancement model outputs a first time-frequency mask. A matrix multiplication operation is performed between the obtained first time-frequency mask and the original audio, and the result is used as the first speech data for subsequent processing. Using the above scheme, the original audio is preprocessed to obtain the first speech data, which is then used for training the subsequent speech enhancement model.

[0084] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0085] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0086] S1, obtain the first encoding result of the preset encoder on the first speech data, and the second encoding result of the preset encoder on the second speech data; wherein, the first speech data is the speech data after the original audio has been processed by the speech enhancement model, and the second speech data is the clean speech data.

[0087] S2, calculate the first encoding result and the second encoding result according to the target algorithm corresponding to the loss function to determine the difference data between the first encoding result and the second encoding result;

[0088] S3, the speech enhancement model is trained at least using the difference data.

[0089] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0090] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0091] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0092] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0093] S1, obtain the first encoding result of the preset encoder on the first speech data, and the second encoding result of the preset encoder on the second speech data; wherein, the first speech data is the speech data after the original audio has been processed by the speech enhancement model, and the second speech data is the clean speech data.

[0094] S2, calculate the first encoding result and the second encoding result according to the target algorithm corresponding to the loss function to determine the difference data between the first encoding result and the second encoding result;

[0095] S3, the speech enhancement model is trained at least using the difference data.

[0096] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0097] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0098] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0099] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A training method for a speech enhancement model, characterized in that, include: Obtain the first encoding result of the preset encoder on the first speech data, and the second encoding result of the preset encoder on the second speech data; wherein, the first speech data is the speech data after the original audio has been processed by the speech enhancement model, the second speech data is the clean speech data, and the first speech data is the second speech data after adding noise; Calculate the first encoding result and the second encoding result to determine the first difference data between the first encoding result and the second encoding result; The speech enhancement model is trained at least using the first difference data to obtain the trained speech enhancement model; The first voice data is obtained through the following methods: Obtain the first time-frequency mask output by the speech enhancement model after processing the original audio, and use the product of the first time-frequency mask and the original audio as the first speech data; or Feature extraction is performed on the original audio to obtain frequency features. The frequency features are then input into the speech enhancement model to obtain a second time-frequency mask corresponding to the original audio. The product of the second time-frequency mask and the original audio is used as the first speech data. Wherein, after training the speech enhancement model using at least the first difference data to obtain the trained speech enhancement model, the method further includes: If the first difference data is greater than a preset threshold, the following training steps are performed: The original audio is input into the trained speech enhancement model for processing, and the resulting third speech data is input into a preset encoder for encoding to obtain a third encoding result; The third encoding result and the second encoding result are calculated to determine the second difference data between the third encoding result and the second encoding result; The trained speech enhancement model is trained using the second difference data to obtain a retrained speech enhancement model.

2. The training method for the speech enhancement model according to claim 1, characterized in that, Calculations are performed on the first encoding result and the second encoding result to determine the first difference data between the first encoding result and the second encoding result, including: Determine the target algorithm corresponding to the loss function, wherein the target algorithm includes at least: the minimum mean square error algorithm; The first encoding result and the second encoding result are input into the target algorithm to determine the first difference data between the first encoding result and the second encoding result according to the target algorithm.

3. The training method for the speech enhancement model according to claim 1, characterized in that, After retraining the speech enhancement model using the second difference data to obtain a retrained speech enhancement model, the method further includes: If the second difference data is greater than the preset threshold, the training step is executed again; If the second difference data is less than the preset threshold, the retrained speech enhancement model is determined to be a trained speech enhancement model.

4. The training method for the speech enhancement model according to claim 3, characterized in that, After determining that the retrained speech enhancement model is the successfully trained speech enhancement model, the method further includes: Upon receiving the original audio again, the received original audio is processed according to the trained speech enhancement model to obtain the enhanced speech of the original audio.

5. The training method for the speech enhancement model according to claim 3, characterized in that, After determining that the retrained speech enhancement model is the successfully trained speech enhancement model, the method further includes: Upon receiving the original audio again, the original audio is processed according to the trained speech enhancement model to obtain estimated speech data. The estimated speech data is mapped to the speech domain of the original audio to obtain the mapped speech features; The mapped speech features are subjected to inverse Fourier transform to obtain the enhanced speech of the original audio.

6. The training method for the speech enhancement model according to any one of claims 1 to 5, characterized in that, Obtaining the first encoding result of the preset encoder for the first speech data, and the second encoding result of the preset encoder for the second speech data, includes: Obtain the first encoding result of the encoder in the end-to-end speech recognition framework for the first speech data, and the second encoding result of the encoder in the end-to-end speech recognition framework for the second speech data.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 6.

8. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 6 through the computer program.

Images