Speech quality enhancement method, device and system based on deep learning
Through the speech sound quality enhancement method based on deep learning, the speech reconstruction neural network is used to extract and reconstruct low-quality speech, solving the problem of improving the sound quality of low-quality code rate voice, and achieving the popularization of high-definition voice communication.
Patent Information
- Application Number
- CN202111069826.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2018-06-05
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2038-06-05
AI Technical Summary
The prior art is difficult to effectively improve the sound quality of low-quality bit rate voice. Traditional methods such as filling or interpolation data methods cannot restore high-quality voice and are not suitable for all users to enjoy high-definition voice communication.
The voice quality enhancement method based on deep learning is adopted to extract features by obtaining the to-processed voice data, and reconstructing it into high-quality voice data using the trained voice reconstruction neural network to achieve sound quality improvement.
It has achieved significant improvement in low-quality voice sound quality, and can provide high-quality voice experience without increasing communication bandwidth, which is suitable for various voice communication scenarios.
Smart Images

Figure CN113870872B_ABST
Abstract
Description
[0001] This application is a divisional application of Chinese patent application No. 201810583123.0 filed on June 5, 2018. Technical Field
[0002] The present invention relates to the technical field of sound quality optimization, and more specifically to a method, device and system for speech quality enhancement based on deep learning. Background Art
[0003] In recent years, voice wireless communication has developed rapidly and is now widely used in various civil and industrial fields. Wireless communication is limited by bandwidth, which requires voice coding compression to reduce the sampling frequency and bit rate of voice as much as possible. Although voice coding reduces voice quality, it also greatly saves resources. Early digital voice communication coding, such as Global Mobile Communication System-Half Rate (GMS-HR), has a bit rate of about 6.5kbps and a sampling frequency of 8kHz. The actual bandwidth is less than 4k, which loses a lot of high-frequency information, making the human voice lack of recognition and can only meet basic voice communication needs.
[0004] As people's demand for sound quality increases, low-bitrate and low-quality voice can no longer meet the demand. As network bandwidth increases, voice communication with higher bitrate quality has become possible. For example, the enhanced voice service (EVS) coding technology may reach 48k sampling frequency and 128kbps bitrate. However, this does not mean that all users can enjoy the experience of high-definition voice communication. For example, in such a scenario: the operator of the calling user supports 4G network, while the operator of the receiving user only supports 3G network, then both parties may only be able to choose adaptive multi-rate coding-narrowband (amr-nb) coding method for voice coding, rather than adaptive multi-rate coding-wideband (amr-wb) coding method with 16kHz sampling frequency. Due to the existence of these scenarios where low-quality bitrate voice has to be used due to hardware conditions, not everyone can enjoy the benefits of high-definition voice communication.
[0005] On the other hand, reducing the coding bit rate as much as possible while maintaining the sound quality is also a major research direction in voice communication. Therefore, under the constraints of limited storage and bandwidth resources, it is a valuable research direction to reconstruct low-quality bit rate speech through digital signal processing methods to make its sound quality close to that of high-quality speech. However, there is currently no corresponding feasible solution for reconstructing low-quality bit rate speech using software methods. For the reconstruction of low-quality bit rate speech, the method of filling or interpolating data is usually adopted, but this method is too rough and basically cannot restore the sound quality of high-quality speech. Summary of the invention
[0006] The present invention is proposed to solve at least one of the above problems. The present invention proposes a solution for speech quality enhancement based on deep learning, which enhances the low-quality speech quality based on a deep learning method, so that the low-quality speech quality can reach the sound quality of high-quality speech through deep neural network reconstruction, thereby achieving the sound quality improvement effect that cannot be achieved by traditional methods. The following briefly describes the solution for speech quality enhancement based on deep learning proposed by the present invention, and more details will be described in the specific implementation mode in conjunction with the accompanying drawings later.
[0007] According to one aspect of the present invention, a method for speech quality enhancement based on deep learning is provided, the method comprising: obtaining speech data to be processed, and performing feature extraction on the speech data to be processed to obtain features of the speech data to be processed; and based on the features of the speech data to be processed, reconstructing the speech data to be processed into output speech data using a trained speech reconstruction neural network, wherein the speech quality of the output speech data is higher than the speech quality of the speech data to be processed.
[0008] In one embodiment of the present invention, the training of the speech reconstruction neural network includes: obtaining a first speech sample and a second speech sample, wherein the speech quality of the second speech sample is lower than the speech quality of the first speech sample, and the second speech sample is obtained by transcoding the first speech sample; performing feature extraction on the first speech sample and the second speech sample respectively to obtain features of the first speech sample and features of the second speech sample respectively; and using the obtained features of the second speech sample as input of the input layer of the speech reconstruction neural network, and using the obtained features of the first speech sample as the target of the output layer of the speech reconstruction neural network, so as to train the speech reconstruction neural network.
[0009] In one embodiment of the present invention, the first speech sample has a first code rate, the second speech sample has a second code rate, and the first code rate is higher than or equal to the second code rate.
[0010] In one embodiment of the present invention, the first speech sample has a first sampling frequency, the second speech sample has a second sampling frequency, and the first sampling frequency is higher than or equal to the second sampling frequency.
[0011] In one embodiment of the present invention, the features obtained by feature extraction include frequency domain amplitude and / or energy information.
[0012] In one embodiment of the present invention, the features obtained by feature extraction also include spectrum phase information.
[0013] In one embodiment of the present invention, the feature extraction method includes short-time Fourier transform.
[0014] In one embodiment of the present invention, the training of the speech reconstruction neural network also includes: before performing feature extraction on the first speech sample and the second speech sample, the first speech sample and the second speech sample are respectively framed, and the feature extraction is performed frame by frame on the speech samples obtained after framing.
[0015] In one embodiment of the present invention, the training of the speech reconstruction neural network also includes: before framing the first speech sample and the second speech sample, respectively decoding the first speech sample and the second speech sample into time domain waveform data, and the framing is performed on the time domain waveform data obtained after decoding.
[0016] In one embodiment of the present invention, the reconstructing the speech data to be processed into output speech data using a trained speech reconstruction neural network includes: taking the features of the speech data to be processed as input of the trained speech reconstruction neural network, and outputting the reconstructed speech features from the trained speech reconstruction neural network; and generating a time domain speech waveform based on the reconstructed speech features as the output speech data.
[0017] According to another aspect of the present invention, a speech quality enhancement device based on deep learning is provided, the device comprising: a feature extraction module, used to obtain speech data to be processed, and perform feature extraction on the speech data to be processed to obtain features of the speech data to be processed; and a speech reconstruction module, used to reconstruct the speech data to be processed into output speech data based on the features of the speech data to be processed extracted by the feature extraction module, using a trained speech reconstruction neural network, wherein the speech quality of the output speech data is higher than the speech quality of the speech data to be processed.
[0018] In one embodiment of the present invention, the training of the speech reconstruction neural network includes: obtaining a first speech sample and a second speech sample, wherein the speech quality of the second speech sample is lower than the speech quality of the first speech sample, and the second speech sample is obtained by transcoding the first speech sample; performing feature extraction on the first speech sample and the second speech sample respectively to obtain features of the first speech sample and features of the second speech sample respectively; and using the obtained features of the second speech sample as input of the input layer of the speech reconstruction neural network, and using the obtained features of the first speech sample as the target of the output layer of the speech reconstruction neural network, so as to train the speech reconstruction neural network.
[0019] In one embodiment of the present invention, the first speech sample has a first code rate, the second speech sample has a second code rate, and the first code rate is higher than or equal to the second code rate.
[0020] In one embodiment of the present invention, the first speech sample has a first sampling frequency, the second speech sample has a second sampling frequency, and the first sampling frequency is higher than or equal to the second sampling frequency.
[0021] In one embodiment of the present invention, the features obtained by feature extraction include frequency domain amplitude and / or energy information.
[0022] In one embodiment of the present invention, the features obtained by feature extraction also include spectrum phase information.
[0023] In one embodiment of the present invention, the feature extraction method includes short-time Fourier transform.
[0024] In one embodiment of the present invention, the training of the speech reconstruction neural network also includes: before performing feature extraction on the first speech sample and the second speech sample, the first speech sample and the second speech sample are respectively framed, and the feature extraction is performed frame by frame on the speech samples obtained after framing.
[0025] In one embodiment of the present invention, the training of the speech reconstruction neural network also includes: before framing the first speech sample and the second speech sample, respectively decoding the first speech sample and the second speech sample into time domain waveform data, and the framing is performed on the time domain waveform data obtained after decoding.
[0026] In one embodiment of the present invention, the speech reconstruction module further includes: a reconstruction module, which is used to use the characteristics of the speech data to be processed as the input of the trained speech reconstruction neural network, and the trained speech reconstruction neural network outputs the reconstructed speech characteristics; and a generation module, which is used to generate a time domain speech waveform as the output speech data based on the reconstructed speech characteristics output by the reconstruction module.
[0027] According to another aspect of the present invention, a speech quality enhancement system based on deep learning is provided, the system comprising a storage device and a processor, the storage device storing a computer program run by the processor, and the computer program, when run by the processor, executes any of the above-mentioned speech quality enhancement methods based on deep learning.
[0028] According to yet another aspect of the present invention, a storage medium is provided, on which a computer program is stored, and when the computer program is run, the computer program executes any of the above-mentioned deep learning-based speech quality enhancement methods.
[0029] According to another aspect of the present invention, a computer program is provided. When the computer program is run by a computer or a processor, it is used to execute any of the above-mentioned deep learning-based speech quality enhancement methods. The computer program is also used to implement each module in any of the above-mentioned deep learning-based speech quality enhancement devices.
[0030] The method, device and system for enhancing speech quality based on deep learning according to the embodiments of the present invention enhance the sound quality of low-quality speech based on the deep learning method, so that the low-quality speech quality can reach the sound quality of high-quality speech through deep neural network reconstruction, thereby achieving the sound quality improvement effect that cannot be achieved by traditional methods. In addition, the method, device and system for enhancing speech quality based on deep learning according to the embodiments of the present invention can be conveniently deployed on the server side or the user side, and can efficiently achieve the enhancement of speech quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.
[0032] Figure 1 A schematic block diagram showing an example electronic device for implementing a method, device and system for speech quality enhancement based on deep learning according to an embodiment of the present invention;
[0033] Figure 2 A schematic flow chart of a method for enhancing speech quality based on deep learning according to an embodiment of the present invention is shown;
[0034] Figure 3 A schematic diagram showing the training of a speech reconstruction neural network according to an embodiment of the present invention is shown;
[0035] Figure 4A , Figure 4B 4C respectively show the spectrograms of high-quality speech, low-quality speech, and speech obtained by reconstructing the low-quality speech using the speech quality enhancement method based on deep learning according to an embodiment of the present invention;
[0036] Figure 5 A schematic block diagram showing a speech quality enhancement device based on deep learning according to an embodiment of the present invention; and
[0037] Figure 6 A schematic block diagram of a speech quality enhancement system based on deep learning according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0038] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.
[0039] First, refer to Figure 1 An example electronic device 100 is described for implementing the method, apparatus, and system for speech quality enhancement based on deep learning according to an embodiment of the present invention.
[0040] like Figure 1 As shown, the electronic device 100 includes one or more processors 102, one or more storage devices 104, an input device 106, and an output device 108, and these components are interconnected through a bus system 110 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structures of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device may also have other components and structures as required.
[0041] The processor 102 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.
[0042] The storage device 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the client functions (implemented by the processor) and / or other desired functions in the embodiments of the present invention described below. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.
[0043] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, etc. In addition, the input device 106 may also be any interface for receiving information.
[0044] The output device 108 can output various information (such as images or sounds) to the outside (such as a user), and can include one or more of a display, a speaker, etc. In addition, the output device 108 can also be any other device with an output function.
[0045] Exemplarily, an example electronic device for implementing the method, apparatus, and system for speech quality enhancement based on deep learning according to an embodiment of the present invention may be implemented as a terminal such as a smart phone, a tablet computer, or the like.
[0046] Next, we will refer to Figure 2 The method 200 for enhancing speech quality based on deep learning according to an embodiment of the present invention is described. Figure 2 As shown, the speech quality enhancement method 200 based on deep learning may include the following steps:
[0047] In step S210, speech data to be processed is acquired, and feature extraction is performed on the speech data to be processed to obtain features of the speech data to be processed.
[0048] In one embodiment, the voice data to be processed obtained in step S210 may be low-quality voice data that needs to be enhanced in sound quality and is received, stored or played in a voice communication terminal or a voice storage / playback device, such as voice data with a low bit rate and a low sampling frequency. Exemplarily, the voice data to be processed may include, but is not limited to: data streams of wireless voice calls, voices in a list being played by a user, or voice files stored in the cloud or on a client, etc. In other examples, the voice data to be processed obtained in step S210 may also be any data that needs to be enhanced in sound quality, such as voice data included in video data, etc. In addition, the voice data to be processed obtained in step S210 may come from files stored offline or from files played online.
[0049] In one embodiment, the method of extracting features from the acquired speech data to be processed may include but is not limited to short-time Fourier transform (STFT). Exemplarily, the features of the speech data to be processed obtained by extracting features from the acquired speech data to be processed may include frequency domain amplitude and / or energy information. Exemplarily, the features of the speech data to be processed obtained by extracting features from the acquired speech data to be processed may also include spectral phase information. Exemplarily, the features of the speech data to be processed obtained by extracting features from the acquired speech data to be processed may also be time domain features. In other examples, the features of the speech data to be processed obtained by extracting features from the acquired speech data to be processed may also include any other features that can characterize the speech data to be processed.
[0050] In one embodiment, before feature extraction is performed on the voice data to be processed, it can be framed, and the aforementioned feature extraction is performed frame by frame for the voice data obtained after framing. This situation can be applicable when the voice data to be processed obtained in step S210 is from a file stored offline or a complete file from any source. In another embodiment, if the voice data to be processed obtained in step S210 comes from a file played online, one or more frames of the voice data to be processed can be cached before feature extraction. Exemplarily, part of the data can be selected for feature extraction for each frame of voice data to be processed obtained after framing or caching, which can effectively reduce the amount of data and improve processing efficiency.
[0051] In another embodiment, before the voice data to be processed is subjected to the aforementioned frame processing, the voice data to be processed may be decoded first, and the aforementioned frame processing may be performed on the time domain waveform data obtained after decoding. This is because the acquired voice data to be processed is generally in an encoded form, and in order to obtain its complete voice time domain information, it may be decoded first.
[0052] In yet another embodiment, before the feature extraction is performed on the processed voice data, the processed voice data may be preprocessed, and the aforementioned feature extraction may be performed on the voice data obtained after the preprocessing. Exemplarily, the preprocessing of the processed voice data may include, but is not limited to, denoising, echo suppression, and automatic gain control, etc. Exemplarily, the preprocessing may be performed after the aforementioned decoding process. Therefore, in one example, the acquired processed voice data may be decoded, preprocessed, framed, and feature extracted in sequence to efficiently extract well-representative features. In other examples, the aforementioned preprocessing operation may also be performed after framing and before feature extraction.
[0053] Now continue with reference Figure 2, describing the subsequent steps of the speech quality enhancement method 200 based on deep learning according to an embodiment of the present invention.
[0054] In step S220, based on the characteristics of the speech data to be processed, the speech data to be processed is reconstructed into output speech data using a trained speech reconstruction neural network, wherein the speech quality of the output speech data is higher than the speech quality of the speech data to be processed.
[0055] In an embodiment of the present invention, the features of the speech data to be processed extracted in step S210 are input into a trained speech reconstruction neural network, and the speech reconstruction neural network reconstructs the input features to obtain reconstructed speech features, and the reconstructed speech features obtained by the reconstruction can be used to generate output speech data with higher speech quality than the acquired speech data to be processed. Therefore, the speech quality enhancement method of the present invention can accurately supplement the lost speech information in low-quality speech based on deep learning, which can not only efficiently achieve a great improvement in the sound quality of low-quality speech, but also does not affect the consideration of communication bandwidth (because the transmission is still low-quality speech data with a small amount of data, but the low-quality speech data can be reconstructed into high-quality speech data at the receiving end).
[0056] Combine the following Figure 3 The training process of the speech reconstruction neural network according to the embodiment of the present invention is described below. Figure 3 As shown, the training of the speech reconstruction neural network according to the embodiment of the present invention may include the following process:
[0057] In S310, a first voice sample and a second voice sample are obtained, wherein the voice quality of the second voice sample is lower than the voice quality of the first voice sample, and the second voice sample is obtained by transcoding the first voice sample.
[0058] In one example, the first voice sample may be a high-quality voice sample, and the second voice sample may be a low-quality voice sample. Exemplarily, the first voice sample may be a set of high-bitrate, high-sampling-frequency voice samples, including but not limited to voice data with sampling frequencies of 16kHz, 24kHz, and 32kHz. In one example, the first voice sample may be transcoded to obtain the second voice sample. For example, an AMR-WB voice sample with a sampling frequency of 16kHz and a bit rate of 23.85kbps may be used as the first voice sample, and the second voice sample may be obtained by transcoding it to an AMR-NB voice with a sampling frequency of 8kHz and a bit rate of 12.2kbps. For another example, the second voice sample may be obtained by converting the first voice sample in FLAC format into MP3 format without reducing the bit rate and sampling frequency. That is, the bit rate of the first voice sample may be higher than or equal to the bit rate of the second voice sample; the sampling frequency of the first voice sample may be higher than or equal to the sampling frequency of the second voice sample. Of course, this is only exemplary. The first voice sample (i.e., high-quality voice sample) may be transcoded to obtain the second voice sample (i.e., low-quality voice sample) in other cases, which may be adaptively adjusted based on the actual application scenario. Specifically, the first voice sample and the second voice sample to be selected may be determined based on the reconstruction requirements of the voice data to be processed obtained in step S210, that is, the first voice sample to be selected and the transcoding method to be adopted to transcode it into the second voice sample may be determined based on the above-mentioned reconstruction requirements.
[0059] Continue to refer Figure 3 In S320, feature extraction is performed on the first speech sample and the second speech sample to obtain features of the first speech sample and features of the second speech sample respectively.
[0060] Similar to what is described above in step S210, in one embodiment, the manner of performing feature extraction on each of the first speech sample and the second speech sample may include, but is not limited to, short-time Fourier transform. Exemplarily, the features obtained by performing feature extraction on each of the first speech sample and the second speech sample may include their respective frequency domain amplitude and / or energy information. Exemplarily, the features obtained by performing feature extraction on the first speech sample and the second speech sample may also include their respective spectral phase information. Exemplarily, the features obtained by performing feature extraction on the first speech sample and the second speech sample may also be their respective time domain features. In other examples, the features obtained by performing feature extraction on each of the first speech sample and the second speech sample may also include any other features that can characterize each of them.
[0061] In addition, similar to the above description in step S210, in one embodiment, before extracting features from the first voice sample and the second voice sample, the first voice sample and the second voice sample may be framed, and the feature extraction may be performed frame by frame on the first voice sample and the second voice sample, respectively, after the first voice sample and the second voice sample are framed. Exemplarily, a portion of data may be selected for feature extraction from each frame of voice sample, which may effectively reduce the amount of data and improve processing efficiency.
[0062] In another embodiment, before performing the aforementioned frame processing on the first voice sample and the second voice sample respectively, the first voice sample and the second voice sample can be decoded respectively, and the aforementioned frame processing can be performed on the respective time domain waveform data obtained after the first voice sample and the second voice sample are decoded respectively.
[0063] In another embodiment, before the first voice sample and the second voice sample are subjected to feature extraction, the first voice sample and the second voice sample may be preprocessed respectively, and the aforementioned feature extraction may be performed on the voice samples obtained after the preprocessing. Exemplarily, the preprocessing performed on the first voice sample and the second voice sample may include, but is not limited to, denoising, echo suppression, and automatic gain control. Exemplarily, the preprocessing may be performed after the aforementioned decoding process. Therefore, in one example, the first voice sample and the second voice sample may be decoded, preprocessed, framed, and feature extracted respectively in sequence to efficiently extract well-representative features. In other examples, the aforementioned preprocessing operation may also be performed after the first voice sample and the second voice sample are framed respectively and before feature extraction.
[0064] In S330, the obtained features of the second speech sample are used as inputs of the input layer of the speech reconstruction neural network, and the obtained features of the first speech sample are used as targets of the output layer of the speech reconstruction neural network to train the speech reconstruction neural network.
[0065] In one embodiment, the features of one or more frames of the second speech samples can be used as inputs of the input layer of the speech reconstruction neural network, and the features of one or more frames of the first speech samples can be used as targets of the output layer of the speech reconstruction neural network, thereby training a neural network regressor as the speech reconstruction neural network used in step S220.
[0066] Combination of the above Figure 3 The training process of the speech reconstruction neural network according to the embodiment of the present invention is exemplarily described. Figure 2As mentioned above, in step S220, based on the trained speech reconstruction neural network, the features of the speech data to be processed can be reconstructed into reconstructed speech features. Since the reconstructed speech features are frequency domain features, a time domain speech waveform output can be generated based on the reconstructed speech features. Exemplarily, the reconstructed speech features can be transformed by inverse Fourier transform to obtain a time domain speech waveform. The output speech waveform can be stored or cached for playback, thereby providing users with a better and improved speech quality experience. The following can be combined with Figure 4A-4C Let's experience the speech quality enhancement effect of the speech quality enhancement method based on deep learning according to the embodiment.
[0067] Figure 4A , Figure 4B 4C respectively show the spectrograms of high-quality speech, low-quality speech, and speech obtained by reconstructing low-quality speech using the speech quality enhancement method based on deep learning according to an embodiment of the present invention. Figure 4A A spectrogram 400 of high-quality speech is shown, taking PCM format, 16kHz sampling frequency, and 16bit quantization bit number as an example; Figure 4B A spectrogram 401 of low-quality speech in MP3 format, 8 kHz sampling frequency, and 8 kbps bit rate obtained by transcoding the high-quality speech is shown; Figure 4C The spectrogram 402 of the reconstructed speech with a sampling frequency of 16 kHz obtained by reconstructing the low-quality speech using the speech quality enhancement method based on deep learning according to an embodiment of the present invention is shown. Figure 4A-4C It is obvious that Figure 4A Compared with the spectrogram of high-quality speech, Figure 4B The spectrogram of the low-quality speech shown lacks many high-frequency components, but after reconstruction by the speech quality enhancement method based on deep learning according to an embodiment of the present invention, Figure 4C The spectrogram of the reconstructed speech shown restores these high-frequency components, achieves super-resolution of narrow-band speech, and significantly improves the sound quality of low-quality speech.
[0068] Based on the above description, the deep learning-based speech quality enhancement method according to an embodiment of the present invention enhances the low-quality speech quality based on the deep learning method, so that the low-quality speech quality can achieve the sound quality of high-quality speech through deep neural network reconstruction, thereby achieving the sound quality improvement effect that traditional methods cannot achieve.
[0069] The above exemplary describes the method for enhancing speech quality based on deep learning according to an embodiment of the present invention. Exemplarily, the method for enhancing speech quality based on deep learning according to an embodiment of the present invention can be implemented in a device, apparatus or system having a memory and a processor.
[0070] In addition, the speech quality enhancement method based on deep learning according to the embodiment of the present invention can be conveniently deployed on mobile devices such as smart phones, tablet computers, personal computers, headphones, speakers, etc. Alternatively, the speech quality enhancement method based on deep learning according to the embodiment of the present invention can also be deployed on the server side (or cloud). Alternatively, the speech quality enhancement method based on deep learning according to the embodiment of the present invention can also be distributedly deployed on the server side (or cloud) and at personal terminals.
[0071] Combine the following Figure 5 Another aspect of the present invention is to describe a speech quality enhancement device based on deep learning. Figure 5 A schematic block diagram of a speech quality enhancement device 500 based on deep learning according to an embodiment of the present invention is shown.
[0072] like Figure 5 As shown, the speech quality enhancement device 500 based on deep learning according to an embodiment of the present invention includes a feature extraction module 510 and a speech reconstruction module 520. The modules can respectively perform the above combined Figure 2 Each step / function of the method for speech quality enhancement based on deep learning is described below. The following only describes the main functions of each module of the device for speech quality enhancement based on deep learning 500, and omits the details described above.
[0073] The feature extraction module 510 is used to obtain the speech data to be processed and perform feature extraction on the speech data to obtain the features of the speech data to be processed. The speech reconstruction module 520 is used to reconstruct the speech data to be processed into output speech data based on the features of the speech data to be processed extracted by the feature extraction module using a trained speech reconstruction neural network, wherein the speech quality of the output speech data is higher than the speech quality of the speech data to be processed. Both the feature extraction module 510 and the speech reconstruction module 520 can be composed of Figure 1 The processor 102 in the electronic device shown executes program instructions stored in the storage device 104 to implement.
[0074] In one embodiment, the voice data to be processed acquired by the feature extraction module 510 may be low-quality voice data that needs to be enhanced in sound quality and is received, stored or played in a voice communication terminal or a voice storage / playback device, such as voice data with a low bit rate and a low sampling frequency. Exemplarily, the voice data to be processed may include, but is not limited to: data streams of wireless voice calls, voices in a list being played by a user, or voice files stored in the cloud or on a client, etc. In other examples, the voice data to be processed acquired by the feature extraction module 510 may also be any data that needs to be enhanced in sound quality, such as voice data included in video data, etc. In addition, the voice data to be processed acquired by the feature extraction module 510 may come from files stored offline or from files played online.
[0075] In one embodiment, the manner in which the feature extraction module 510 extracts features from the acquired voice data to be processed may include but is not limited to short-time Fourier transform (STFT). Exemplarily, the features of the voice data to be processed obtained by the feature extraction module 510 performing feature extraction on the acquired voice data to be processed may include frequency domain amplitude and / or energy information. Exemplarily, the features of the voice data to be processed obtained by the feature extraction module 510 performing feature extraction on the acquired voice data to be processed may also include spectral phase information. Exemplarily, the features of the voice data to be processed obtained by the feature extraction module 510 performing feature extraction on the acquired voice data to be processed may also be time domain features. In other examples, the features of the voice data to be processed obtained by the feature extraction module 510 performing feature extraction on the acquired voice data to be processed may also include any other features that can characterize the voice data to be processed.
[0076] In one embodiment, before the feature extraction module 510 performs feature extraction on the voice data to be processed, it can be framed first, and the aforementioned feature extraction is performed frame by frame on the voice data obtained after framing. This situation can be applicable when the voice data to be processed obtained by the feature extraction module 510 is from a file stored offline or a complete file from any source. In another embodiment, if the voice data to be processed obtained by the feature extraction module 510 is from a file played online, one or more frames of the voice data to be processed can be cached before feature extraction. Exemplarily, the feature extraction module 510 can select part of the data for feature extraction for each frame of voice data to be processed obtained after framing or after caching, which can effectively reduce the amount of data and improve processing efficiency.
[0077] In another embodiment, before the feature extraction module 510 performs the aforementioned frame processing on the speech data to be processed, the speech data to be processed may be first decoded, and the aforementioned frame processing may be performed on the time domain waveform data obtained after decoding. This is because the acquired speech data to be processed is generally in an encoded form, and in order to obtain its complete speech time domain information, it may be first decoded.
[0078] In yet another embodiment, before the feature extraction module 510 performs feature extraction on the processed voice data, the processed voice data may also be preprocessed, and the aforementioned feature extraction may be performed on the voice data obtained after the preprocessing. Exemplarily, the preprocessing of the processed voice data by the feature extraction module 510 may include, but is not limited to: denoising, echo suppression, and automatic gain control, etc. Exemplarily, the preprocessing may be performed after the aforementioned decoding process. Therefore, in one example, the feature extraction module 510 may sequentially decode, preprocess, frame, and feature extract the acquired processed voice data to efficiently extract well-representative features. In other examples, the aforementioned preprocessing operation may also be performed after framing and before feature extraction.
[0079] Based on the features of the speech data to be processed extracted by the feature extraction module 510 , the speech reconstruction module 520 may reconstruct the speech data to be processed into output speech data using a trained speech reconstruction neural network.
[0080] In the embodiment of the present invention, the speech reconstruction module 520 may further include a reconstruction module (not shown in Figure 5 ) and the generation module (not shown in Figure 5 ). The reconstruction module may include a trained speech reconstruction neural network, which takes the features of the speech data to be processed extracted by the feature extraction module 510 as input, and reconstructs the input features to obtain reconstructed speech features. The generation module generates output speech data with higher speech quality than the acquired speech data to be processed based on the reconstructed speech features output by the reconstruction module. Therefore, the speech quality enhancement device of the present invention can accurately supplement the lost speech information in low-quality speech based on deep learning, which can not only efficiently achieve a great improvement in the sound quality of low-quality speech, but also does not affect the consideration of communication bandwidth (because the transmitted data is still low-quality speech data with a small amount of data, but the low-quality speech data can be reconstructed into high-quality speech data at the receiving end).
[0081] In an embodiment of the present invention, the training of the speech reconstruction neural network used by the speech reconstruction module 520 may include: obtaining a first speech sample and a second speech sample, wherein the speech quality of the second speech sample is lower than the speech quality of the first speech sample, and the second speech sample is obtained by transcoding the first speech sample; performing feature extraction on the first speech sample and the second speech sample to obtain the features of the first speech sample and the features of the second speech sample respectively; and using the features of the second speech sample obtained as the input of the input layer of the speech reconstruction neural network, and using the features of the first speech sample obtained as the target of the output layer of the speech reconstruction neural network, so as to train the speech reconstruction neural network. Figure 3 See above about Figure 3 The description is provided to understand the training process of the speech reconstruction neural network used by the speech reconstruction module 520 of the speech quality enhancement device 500 based on deep learning according to an embodiment of the present invention. For the sake of brevity, too many details are not repeated here.
[0082] In one example, the first voice sample may be a high-quality voice sample, and the second voice sample may be a low-quality voice sample. Exemplarily, the first voice sample may be a set of high-bitrate, high-sampling-frequency voice samples, including but not limited to voice data with sampling frequencies of 16kHz, 24kHz, and 32kHz. In one example, the first voice sample may be transcoded to obtain the second voice sample. For example, an AMR-WB voice sample with a sampling frequency of 16kHz and a bit rate of 23.85kbps may be used as the first voice sample, and the second voice sample may be obtained by transcoding it to an AMR-NB voice with a sampling frequency of 8kHz and a bit rate of 12.2kbps. For another example, the second voice sample may be obtained by converting the first voice sample in FLAC format into MP3 format without reducing the bit rate and sampling frequency. That is, the bit rate of the first voice sample may be higher than or equal to the bit rate of the second voice sample; the sampling frequency of the first voice sample may be higher than or equal to the sampling frequency of the second voice sample. Of course, this is only exemplary. The first voice sample (i.e., high-quality voice sample) may be transcoded to obtain the second voice sample (i.e., low-quality voice sample) in other cases, which may be adaptively adjusted based on the actual application scenario. Specifically, the first voice sample and the second voice sample to be selected may be determined based on the reconstruction requirements of the voice data to be processed obtained by the feature extraction module 510, that is, the first voice sample to be selected and the transcoding method to be adopted to transcode it into the second voice sample may be determined based on the above-mentioned reconstruction requirements.
[0083] In one embodiment, the method of extracting features from each of the first voice sample and the second voice sample may include but is not limited to short-time Fourier transform. Exemplarily, the features obtained by extracting features from each of the first voice sample and the second voice sample may include their respective frequency domain amplitude and / or energy information. Exemplarily, the features obtained by extracting features from the first voice sample and the second voice sample may also include their respective spectral phase information. Exemplarily, the features obtained by extracting features from the first voice sample and the second voice sample may also be their respective time domain features. In other examples, the features obtained by extracting features from each of the first voice sample and the second voice sample may also include any other features that can characterize each of them.
[0084] In one embodiment, before extracting features from the first voice sample and the second voice sample, the first voice sample and the second voice sample may be framed, and the feature extraction may be performed frame by frame on the first voice sample and the second voice sample, respectively, after the first voice sample and the second voice sample are framed. Exemplarily, a portion of data may be selected for feature extraction from each frame of the voice sample, which may effectively reduce the amount of data and improve processing efficiency.
[0085] In another embodiment, before performing the aforementioned frame processing on the first voice sample and the second voice sample respectively, the first voice sample and the second voice sample can be decoded respectively, and the aforementioned frame processing can be performed on the respective time domain waveform data obtained after the first voice sample and the second voice sample are decoded respectively.
[0086] In another embodiment, before the first voice sample and the second voice sample are subjected to feature extraction, the first voice sample and the second voice sample may be preprocessed respectively, and the aforementioned feature extraction may be performed on the voice samples obtained after the preprocessing. Exemplarily, the preprocessing performed on the first voice sample and the second voice sample may include, but is not limited to, denoising, echo suppression, and automatic gain control. Exemplarily, the preprocessing may be performed after the aforementioned decoding process. Therefore, in one example, the first voice sample and the second voice sample may be decoded, preprocessed, framed, and feature extracted respectively in sequence to efficiently extract well-representative features. In other examples, the aforementioned preprocessing operation may also be performed after the first voice sample and the second voice sample are framed respectively and before feature extraction.
[0087] In one embodiment, the features of one or more frames of the second speech samples can be used as the input of the input layer of the speech reconstruction neural network, and the features of one or more frames of the first speech samples can be used as the target of the output layer of the speech reconstruction neural network, so as to train a neural network regressor as the speech reconstruction neural network used in the speech reconstruction module 520.
[0088] Based on the trained speech reconstruction neural network, the reconstruction module of the speech reconstruction module 520 can reconstruct the features of the speech data to be processed into reconstructed speech features. Since the reconstructed speech features are frequency domain features, the generation module of the speech reconstruction module 520 can generate a time domain speech waveform output based on the reconstructed speech features. Exemplarily, the generation module can transform the reconstructed speech features by inverse Fourier transform to obtain a time domain speech waveform. The output speech waveform can be stored or cached for playback, thereby providing users with a better and improved speech quality experience. It can be combined with Figure 4A-4C Refer to the above Figure 4A-4C The description of the embodiment can be used to appreciate the speech quality enhancement effect of the speech quality enhancement device based on deep learning according to the embodiment. For the sake of brevity, it will not be repeated here.
[0089] Based on the above description, the deep learning-based speech quality enhancement device according to the embodiment of the present invention enhances the low-quality speech quality based on the deep learning method, so that the low-quality speech quality can reach the sound quality of high-quality speech through deep neural network reconstruction, thereby achieving the sound quality improvement effect that traditional methods cannot achieve. In addition, the deep learning-based device according to the embodiment of the present invention can be conveniently deployed on the server or user side, and can efficiently achieve speech quality enhancement.
[0090] Figure 6 A schematic block diagram of a speech quality enhancement system 600 based on deep learning according to an embodiment of the present invention is shown. The speech quality enhancement system 600 based on deep learning includes a storage device 610 and a processor 620.
[0091] The storage device 610 stores a program for implementing the corresponding steps in the method for speech quality enhancement based on deep learning according to an embodiment of the present invention. The processor 620 is used to run the program stored in the storage device 610 to execute the corresponding steps of the method for speech quality enhancement based on deep learning according to an embodiment of the present invention, and to implement the corresponding modules in the device for speech quality enhancement based on deep learning according to an embodiment of the present invention.
[0092] In one embodiment, when the program is executed by the processor 620, the deep learning-based speech quality enhancement system 600 performs the following steps: obtaining speech data to be processed, and performing feature extraction on the speech data to be processed to obtain features of the speech data to be processed; and based on the features of the speech data to be processed, reconstructing the speech data to be processed into output speech data using a trained speech reconstruction neural network, wherein the speech quality of the output speech data is higher than the speech quality of the speech data to be processed.
[0093] In one embodiment, the training of the speech reconstruction neural network includes: obtaining a first speech sample and a second speech sample, wherein the speech quality of the second speech sample is lower than the speech quality of the first speech sample, and the second speech sample is obtained by transcoding the first speech sample; performing feature extraction on the first speech sample and the second speech sample respectively to obtain features of the first speech sample and features of the second speech sample respectively; and using the obtained features of the second speech sample as input of the input layer of the speech reconstruction neural network, and using the obtained features of the first speech sample as the target of the output layer of the speech reconstruction neural network, so as to train the speech reconstruction neural network.
[0094] In one embodiment, the first speech sample has a first code rate, the second speech sample has a second code rate, and the first code rate is higher than or equal to the second code rate.
[0095] In one embodiment, the first speech sample has a first sampling frequency, the second speech sample has a second sampling frequency, and the first sampling frequency is higher than or equal to the second sampling frequency.
[0096] In one embodiment, the features extracted include frequency domain amplitude and / or energy information.
[0097] In one embodiment, the features obtained by the feature extraction further include spectrum phase information.
[0098] In one embodiment, the feature extraction method includes short-time Fourier transform.
[0099] In one embodiment, the training of the speech reconstruction neural network further includes: before performing feature extraction on the first speech sample and the second speech sample, respectively framing the first speech sample and the second speech sample, and the feature extraction is performed frame by frame on the speech samples obtained after framing.
[0100] In one embodiment, the training of the speech reconstruction neural network also includes: before framing the first speech sample and the second speech sample, respectively decoding the first speech sample and the second speech sample into time domain waveform data, and the framing is performed on the time domain waveform data obtained after decoding.
[0101] In one embodiment, in one embodiment, when the program is run by the processor 620, the deep learning-based speech quality enhancement system 600 executes the process of reconstructing the speech data to be processed into output speech data using the trained speech reconstruction neural network, including: taking the features of the speech data to be processed as the input of the trained speech reconstruction neural network, and outputting the reconstructed speech features by the trained speech reconstruction neural network; and generating a time domain speech waveform based on the reconstructed speech features as the output speech data.
[0102] In addition, according to an embodiment of the present invention, a storage medium is also provided, on which program instructions are stored, and when the program instructions are run by a computer or a processor, they are used to execute the corresponding steps of the speech quality enhancement method based on deep learning according to an embodiment of the present invention, and are used to implement the corresponding modules in the speech quality enhancement device based on deep learning according to an embodiment of the present invention. The storage medium may, for example, include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.
[0103] In one embodiment, when the computer program instructions are executed by a computer, the various functional modules of the speech quality enhancement device based on deep learning according to an embodiment of the present invention can be implemented, and / or the speech quality enhancement method based on deep learning according to an embodiment of the present invention can be executed.
[0104] In one embodiment, when the computer program instructions are executed by a computer or a processor, the computer or processor executes the following steps: obtaining speech data to be processed, and performing feature extraction on the speech data to be processed to obtain features of the speech data to be processed; and based on the features of the speech data to be processed, reconstructing the speech data to be processed into output speech data using a trained speech reconstruction neural network, wherein the speech quality of the output speech data is higher than the speech quality of the speech data to be processed.
[0105] In one embodiment, the training of the speech reconstruction neural network includes: obtaining a first speech sample and a second speech sample, wherein the speech quality of the second speech sample is lower than the speech quality of the first speech sample, and the second speech sample is obtained by transcoding the first speech sample; performing feature extraction on the first speech sample and the second speech sample respectively to obtain features of the first speech sample and features of the second speech sample respectively; and using the obtained features of the second speech sample as input of the input layer of the speech reconstruction neural network, and using the obtained features of the first speech sample as the target of the output layer of the speech reconstruction neural network, so as to train the speech reconstruction neural network.
[0106] In one embodiment, the first speech sample has a first code rate, the second speech sample has a second code rate, and the first code rate is higher than or equal to the second code rate.
[0107] In one embodiment, the first speech sample has a first sampling frequency, the second speech sample has a second sampling frequency, and the first sampling frequency is higher than or equal to the second sampling frequency.
[0108] In one embodiment, the features extracted include frequency domain amplitude and / or energy information.
[0109] In one embodiment, the features obtained by the feature extraction further include spectrum phase information.
[0110] In one embodiment, the feature extraction method includes short-time Fourier transform.
[0111] In one embodiment, the training of the speech reconstruction neural network further includes: before performing feature extraction on the first speech sample and the second speech sample, respectively framing the first speech sample and the second speech sample, and the feature extraction is performed frame by frame on the speech samples obtained after framing.
[0112] In one embodiment, the training of the speech reconstruction neural network also includes: before framing the first speech sample and the second speech sample, respectively decoding the first speech sample and the second speech sample into time domain waveform data, and the framing is performed on the time domain waveform data obtained after decoding.
[0113] In one embodiment, the computer program instructions, when executed by a computer or processor, cause the computer or processor to execute the steps of reconstructing the speech data to be processed into output speech data using a trained speech reconstruction neural network, including: taking features of the speech data to be processed as input of the trained speech reconstruction neural network, and outputting reconstructed speech features from the trained speech reconstruction neural network; and generating a time domain speech waveform based on the reconstructed speech features as the output speech data.
[0114] Each module in the deep learning-based speech quality enhancement device according to an embodiment of the present invention can be implemented by running computer program instructions stored in a memory by a processor of an electronic device for deep learning-based speech quality enhancement according to an embodiment of the present invention, or can be implemented when computer instructions stored in a computer-readable storage medium of a computer program product according to an embodiment of the present invention are executed by a computer.
[0115] In addition, according to an embodiment of the present invention, a computer program is also provided, which can be stored in a cloud or local storage medium. When the computer program is run by a computer or a processor, it is used to perform the corresponding steps of the deep learning-based speech quality enhancement method of the embodiment of the present invention, and is used to implement the corresponding modules in the deep learning-based speech quality enhancement device according to the embodiment of the present invention.
[0116] The method, device, system, storage medium and computer program for speech quality enhancement based on deep learning according to the embodiments of the present invention enhance the sound quality of low-quality speech based on deep learning methods, so that the low-quality speech quality can reach the sound quality of high-quality speech through deep neural network reconstruction, thereby achieving the sound quality improvement effect that cannot be achieved by traditional methods. In addition, the method, device, system, storage medium and computer program for speech quality enhancement based on deep learning according to the embodiments of the present invention can be conveniently deployed on the server side or the user side, and can efficiently achieve speech quality enhancement.
[0117] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.
[0118] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0119] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.
[0120] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.
[0121] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.
[0122] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.
[0123] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.
[0124] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules according to embodiments of the present invention. The present invention can also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.
[0125] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.
[0126] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment. The protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A method for speech quality enhancement based on deep learning, characterized in that: The method comprises: Acquiring speech data to be processed, and decoding and feature extracting the speech data to be processed to obtain features of the speech data to be processed; and Based on the characteristics of the speech data to be processed, reconstructing the speech data to be processed into output speech data using a trained speech reconstruction neural network, wherein the speech quality of the output speech data is higher than the speech quality of the speech data to be processed, and the speech quality measurement index includes a bit rate; The step of reconstructing the to-be-processed speech data into output speech data by using the trained speech reconstruction neural network includes: Using the features of the speech data to be processed as input to the trained speech reconstruction neural network, and having the trained speech reconstruction neural network output reconstructed speech features; and A time domain speech waveform is generated based on the reconstructed speech feature as the output speech data.
2. The method according to claim 1, characterized in that The training of the speech reconstruction neural network includes: Acquire a first voice sample and a second voice sample, wherein the voice quality of the second voice sample is lower than the voice quality of the first voice sample, and the second voice sample is obtained by transcoding the first voice sample; Performing feature extraction on the first speech sample and the second speech sample to obtain features of the first speech sample and features of the second speech sample respectively; and The obtained features of the second speech sample are used as inputs of the input layer of the speech reconstruction neural network, and the obtained features of the first speech sample are used as targets of the output layer of the speech reconstruction neural network to train the speech reconstruction neural network.
3. The method according to claim 2, characterized in that The first speech sample has a first bit rate, the second speech sample has a second bit rate, and the first bit rate is higher than or equal to the second bit rate; The features extracted include frequency domain amplitude and / or energy information.
4. The method according to claim 3, characterized in that The first speech sample has a first sampling frequency, the second speech sample has a second sampling frequency, and the first sampling frequency is higher than or equal to the second sampling frequency.
5. The method according to claim 3, characterized in that: The features obtained by the feature extraction also include spectrum phase information.
6. The method according to claim 5, characterized in that The feature extraction method includes short-time Fourier transform.
7. The method according to claim 2, characterized in that: The training of the speech reconstruction neural network also includes: Before performing feature extraction on the first speech sample and the second speech sample, the first speech sample and the second speech sample are respectively divided into frames, and the feature extraction is performed frame by frame on the speech samples obtained after the division into frames.
8. The method according to claim 7, characterized in that The training of the speech reconstruction neural network also includes: Before the first speech sample and the second speech sample are framed, the first speech sample and the second speech sample are respectively decoded into time domain waveform data, and the frame division is performed on the time domain waveform data obtained after decoding.
9. A speech quality enhancement device based on deep learning, characterized in that: The device comprises: A feature extraction module, used to obtain speech data to be processed, and decode and extract features from the speech data to be processed to obtain features of the speech data to be processed; and A speech reconstruction module, configured to reconstruct the speech data to be processed into output speech data using a trained speech reconstruction neural network based on the features of the speech data to be processed extracted by the feature extraction module, wherein the speech quality of the output speech data is higher than the speech quality of the speech data to be processed, and the speech quality measurement index includes a bit rate; Wherein, the speech reconstruction module further comprises: A reconstruction module, used for taking the features of the speech data to be processed as input of the trained speech reconstruction neural network, and having the trained speech reconstruction neural network output reconstructed speech features; and A generating module is used to generate a time-domain speech waveform as the output speech data based on the reconstructed speech features output by the reconstructing module.
10. The device according to claim 9, characterized in that The training of the speech reconstruction neural network includes: Acquire a first voice sample and a second voice sample, wherein the voice quality of the second voice sample is lower than the voice quality of the first voice sample, and the second voice sample is obtained by transcoding the first voice sample; Performing feature extraction on the first speech sample and the second speech sample to obtain features of the first speech sample and features of the second speech sample respectively; and The obtained features of the second speech sample are used as inputs of the input layer of the speech reconstruction neural network, and the obtained features of the first speech sample are used as targets of the output layer of the speech reconstruction neural network to train the speech reconstruction neural network.
11. The device according to claim 10, characterized in that The first speech sample has a first bit rate, the second speech sample has a second bit rate, and the first bit rate is higher than or equal to the second bit rate; The features extracted include frequency domain amplitude and / or energy information.
12. The device according to claim 11, characterized in that The first speech sample has a first sampling frequency, the second speech sample has a second sampling frequency, and the first sampling frequency is higher than or equal to the second sampling frequency.
13. The device according to claim 11, characterized in that The features obtained by the feature extraction also include spectrum phase information.
14. The device according to claim 13, characterized in that The feature extraction method includes short-time Fourier transform.
15. The device according to claim 10, characterized in that The training of the speech reconstruction neural network also includes: Before performing feature extraction on the first speech sample and the second speech sample, the first speech sample and the second speech sample are respectively divided into frames, and the feature extraction is performed frame by frame on the speech samples obtained after the division into frames.
16. The device according to claim 15, characterized in that The training of the speech reconstruction neural network also includes: Before the first speech sample and the second speech sample are framed, the first speech sample and the second speech sample are respectively decoded into time domain waveform data, and the frame division is performed on the time domain waveform data obtained after decoding.
17. A speech quality enhancement system based on deep learning, characterized in that: The system includes a storage device and a processor, wherein the storage device stores a computer program executed by the processor, and when the computer program is executed by the processor, the deep learning-based speech quality enhancement method as described in any one of claims 1 to 8 is executed.
Citation Information
Patent Citations
Audio signal encoding and decoding method
CN101197576A
Vector code quantizer based on particle group
CN101414365A
Television program transmission system based on broadband mobile communication technology and working method thereof
CN102238373A
Voice enhancement method and device
CN103038825A
Method and system for service quality assurance facing mobile voice frequency
CN103151049A
