Frequency band extension method, device, equipment, medium and program product

Abstract

The present disclosure provides a frequency band extension method, device, equipment, medium and program product, which relates to the field of computer technology, and in particular to the field of speech technology. The specific implementation scheme is: denoising a noisy narrowband speech signal to obtain a denoised narrowband speech signal; determining the low-frequency transformation characteristics of the denoised narrowband speech signal in the transform domain, and performing frequency band extension on the low-frequency transformation characteristics to obtain a high-frequency transformation characteristic; determining a broadband speech signal according to the high-frequency transformation characteristic. The embodiments of the present disclosure can achieve a good spread spectrum effect in a noisy scene.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the field of voice technology, and specifically to a frequency band extension method, device, equipment, medium and program product. Background Art

[0002] In the current Public Switched Telephone Networks (PSTN) and some wireless communication systems, due to the limitations of voice acquisition equipment, channel bandwidth and encoding methods, voice signals can only retain low-frequency signals during transmission, for example, only 300Hz to 3400Hz. The loss of high-frequency parts will lead to a decrease in voice quality.

[0003] Voice band extension is to restore broadband voice signals from band-limited narrowband voice signals to improve voice quality. However, voice calls often contain noise, and in noisy scenarios, band extension faces greater challenges. How to achieve band extension in noisy scenarios is very important for improving voice quality. Summary of the invention

[0004] The present disclosure provides a frequency band extension method, apparatus, device, medium and program product.

[0005] According to one aspect of the present disclosure, there is provided a frequency band extension method, comprising:

[0006] Performing noise reduction processing on the noisy narrow-band speech signal to obtain a noise-reduced narrow-band speech signal;

[0007] Determining low-frequency transformation features of the denoised narrow-band speech signal in a transformation domain, and performing frequency band expansion on the low-frequency transformation features to obtain high-frequency transformation features;

[0008] A broadband speech signal is determined according to the high frequency conversion feature.

[0009] According to another aspect of the present disclosure, there is provided a frequency band extension device, comprising:

[0010] A speech noise reduction module is used to perform noise reduction processing on a noisy narrow-band speech signal to obtain a noise-reduced narrow-band speech signal;

[0011] A frequency band extension module, used for determining the low-frequency transformation features of the noise-reduced narrow-band speech signal in the transformation domain, and performing frequency band extension on the low-frequency transformation features to obtain high-frequency transformation features;

[0012] The broadband speech signal determination module is used to determine the broadband speech signal according to the high frequency conversion feature.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to at least one processor; wherein,

[0016] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the frequency band extension method of any embodiment of the present disclosure.

[0017] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the frequency band extension method of any embodiment of the present disclosure.

[0018] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the frequency band extension method of any embodiment of the present disclosure is implemented.

[0019] The embodiments of the present disclosure can achieve good spectrum spreading effect in noisy scenarios.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a schematic diagram of a frequency band extension method provided according to an embodiment of the present disclosure;

[0023] Figure 2a is a schematic diagram of a frequency band extension method provided according to an embodiment of the present disclosure;

[0024] Figure 2b is a flow chart of a frequency band extension method provided according to an embodiment of the present disclosure;

[0025] Figure 2c is a schematic diagram of a noisy narrowband speech signal provided according to an embodiment of the present disclosure;

[0026] Figure 2d is a schematic diagram of a broadband speech signal provided according to an embodiment of the present disclosure;

[0027] Figure 3a is a schematic diagram of a frequency band extension method provided according to an embodiment of the present disclosure;

[0028] Figure 3b is a schematic diagram of a broadband speech signal provided according to an embodiment of the present disclosure;

[0029] Figure 3c is a schematic diagram of a broadband speech signal after noise reduction provided according to an embodiment of the present disclosure;

[0030] Figure 4 is a schematic diagram of a frequency band extension device provided according to an embodiment of the present disclosure;

[0031] Figure 5 The block diagram is a block diagram of an electronic device for implementing the frequency band extension method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] Figure 1 This is a flow chart of a frequency band extension method disclosed in an embodiment of the present disclosure. This embodiment can be applied to the case where frequency band extension is performed on the basis of noise reduction processing. The method of this embodiment can be executed by a frequency band extension device, which can be implemented in software and / or hardware, and is specifically configured in an electronic device with certain data computing capabilities, which can be a client device or a server device, and the client device can be a mobile phone, a tablet computer, a vehicle terminal, a desktop computer, etc.

[0034] S110: Perform noise reduction processing on the noisy narrow-band speech signal to obtain a noise-reduced narrow-band speech signal.

[0035] Bandwidth expansion is required in many practical scenarios, such as the public telephone exchange network. Due to the limitations of voice collection equipment, channel bandwidth, and encoding methods, the frequency band of call voice is limited to 300Hz-3400Hz. The lack of high-frequency part will lead to a decline in voice quality, so band expansion is very important.

[0036] In the case of steady-state noise or non-steady-state noise in the call environment, if the narrowband voice signal containing noise is directly band-expanded, a large amount of noise will be contained in the broadband voice signal obtained in the end, affecting the user's communication experience. Among them, the narrowband voice signal containing noise is a noisy narrowband voice signal. Exemplarily, the noisy narrowband voice signal contains steady-state noise emitted by indoor air conditioners, vacuum cleaners, or exhaust hoods. Exemplarily, the noisy narrowband voice signal may also contain non-steady-state noise such as desktop knocking or door opening and closing.

[0037] In order to improve the voice quality, the solution of the embodiment of the present disclosure can first perform noise reduction processing after obtaining the narrowband voice signal to remove the noise contained therein and obtain the noise-reduced narrowband voice signal, so as to facilitate the frequency band expansion based on the noise-reduced narrowband voice signal, and avoid the wideband voice signal obtained by the frequency band expansion containing a large amount of noise affecting the communication experience. Specifically, traditional algorithms such as spectral subtraction or Wiener filtering can be used for voice noise reduction, and convolutional neural networks, recursive neural networks, or deep neural networks can also be used for voice noise reduction.

[0038] In a specific example, a noisy speech signal is subjected to a Fourier transform to obtain a speech spectrum of the noisy speech signal. Further, the speech spectrum is input into a convolutional neural network to obtain a denoised speech spectrum output by the convolutional neural network. Finally, an inverse Fourier transform is performed on the denoised speech spectrum to obtain a denoised narrowband speech signal.

[0039] In another specific example, the noisy speech signal is decomposed into subbands to obtain a speech spectrum of the noisy narrowband speech signal, wherein the speech spectrum includes an amplitude spectrum and a phase spectrum. Furthermore, in order to optimize the noise reduction effect, the amplitude value of each frequency point in the amplitude spectrum is converted to the logarithmic domain to obtain frequency domain feature data matching the noisy narrowband speech signal. The frequency domain feature data is then input into the noise reduction model to obtain the mask value output by the noise reduction model. Based on the mask value and the speech spectrum of the noisy narrowband speech signal, the noise-reduced speech spectrum is determined. Finally, the noise-reduced speech spectrum is subband fused to obtain a noise-reduced narrowband speech signal. Among them, the noise reduction model may include at least one layer of convolutional neural network, at least one layer of recursive neural network, a fully connected layer, and an activation layer.

[0040] S120, determining low-frequency transformation features of the noise-reduced narrow-band speech signal in the transformation domain, and performing frequency band expansion on the low-frequency transformation features to obtain high-frequency transformation features.

[0041] In the disclosed embodiment, in order to improve the quality of the speech signal, after the noise-reducing narrowband speech signal is subjected to noise reduction processing to obtain the noise-reducing narrowband speech signal, the noise-reducing narrowband speech signal will be band-expanded to predict and complete the high-frequency part of the noise-reducing narrowband speech signal. First, the noise-reducing narrowband speech signal is transformed to obtain the low-frequency transformation features of the noise-reducing narrowband speech signal in the transform domain. Furthermore, the low-frequency transformation features are band-expanded to obtain high-frequency transformation features. Specifically, the noise-reducing narrowband speech signal can be encoded and transformed into the transform domain to obtain the low-frequency transformation features. Furthermore, the low-frequency transformation features can be input into the spread spectrum model to obtain the high-frequency transformation features predicted by the spread spectrum model.

[0042] In a specific example, a denoised narrowband speech signal is encoded by an encoder composed of one-dimensional convolution to obtain low-frequency transformation features of the denoised narrowband speech signal in the transform domain. Further, the low-frequency transformation features are input into a spread spectrum model for band expansion to obtain high-frequency transformation features predicted by the spread spectrum model. The spread spectrum model may include at least one layer of residual blocks; the residual blocks include a dilated convolutional neural network.

[0043] S130: Determine a broadband speech signal according to high frequency conversion characteristics.

[0044] In the disclosed embodiment, after obtaining the high-frequency transformation features predicted by the spread spectrum model, the high-frequency transformation features are decoded to obtain the high-frequency part of the speech signal. Finally, the narrowband speech signal and the high-frequency part of the speech signal can be combined to obtain the final broadband speech signal.

[0045] Exemplarily, the high-frequency transformation feature can be decoded by a decoder composed of a single-bit convolution to obtain the high-frequency part of the speech signal. Finally, the narrowband speech signal and the high-frequency part of the speech signal are combined to obtain the final broadband speech signal.

[0046] The technical solution of the embodiment of the present disclosure first performs noise reduction processing on the noisy narrowband speech signal, then determines the low-frequency transformation features of the denoised narrowband speech signal after the noise reduction processing in the transform domain, and performs frequency band expansion on the low-frequency transformation features to obtain high-frequency transformation features, and finally determines the broadband speech signal based on the high-frequency transformation features, so as to achieve frequency band expansion of the speech signal in a noisy scene.

[0047] Figure 2a This is a schematic diagram of a frequency band extension method in an embodiment of the present disclosure, which further refines the above embodiment and provides specific steps of performing noise reduction processing on a noisy narrowband speech signal to obtain a noise-reduced narrowband speech signal, and performing frequency band extension on a low-frequency conversion feature to obtain a high-frequency conversion feature. Figure 2aA frequency band extension method provided in an embodiment of the present disclosure is described, including the following:

[0048] S210, determining a first frequency spectrum of the noisy narrow-band speech signal; the first frequency spectrum includes a first amplitude spectrum and a phase spectrum.

[0049] The first spectrum is a spectrum obtained by transforming the noisy narrowband speech signal into the frequency domain, wherein the first spectrum includes an amplitude spectrum and a phase spectrum. Exemplarily, the first spectrum is a spectrum obtained by Fourier transforming the noisy narrowband speech signal.

[0050] In the embodiment of the present disclosure, in order to prevent the noise contained in the noisy narrowband speech signal from affecting the signal quality, it is necessary to reduce the noise of the noisy narrowband speech signal. To achieve the noise reduction of the noisy narrowband speech signal, the noisy narrowband speech signal is first transformed into the frequency domain to obtain the first spectrum of the noisy narrowband speech signal. The first spectrum includes an amplitude spectrum and a phase spectrum. Exemplarily, the first spectrum of the narrowband speech signal can be obtained by performing Fourier transform on the noisy narrowband speech signal; the narrowband speech signal can also be converted into the frequency domain by sub-band decomposition to obtain the first spectrum.

[0051] Figure 2b This is a flow chart of a frequency band extension method provided in an embodiment of the present disclosure. In this process, a noisy narrowband speech signal can be transformed into the frequency domain by sub-band decomposition to obtain a first spectrum.

[0052] S220: Convert the amplitude value of each frequency point in the first amplitude spectrum into a logarithmic domain to obtain frequency domain feature data matching the noisy narrow-band speech signal.

[0053] In the disclosed embodiment, feature extraction is performed on the first amplitude spectrum. Specifically, the amplitude value of each frequency point in the amplitude spectrum is obtained, and the amplitude value is converted to the logarithmic domain to obtain frequency domain feature data extracted from the narrowband speech signal. By converting the amplitude value to the logarithmic domain, the subsequent speech noise reduction effect can be improved.

[0054] S230: Based on the frequency domain feature data, perform noise reduction processing on the noisy narrow-band speech signal to obtain a noise-reduced narrow-band speech signal.

[0055] In the disclosed embodiment, after the frequency domain feature data is obtained, the frequency domain feature data can be input into the noise reduction model for noise reduction processing to obtain a noise-reduced narrowband speech signal. Based on the extracted frequency domain feature data, noise reduction processing is performed, which can improve the noise reduction effect compared to noise reduction for the first spectrum.

[0056] For example, the frequency domain feature data can be input into the convolutional neural network model for noise reduction processing to obtain the noise-reduced speech spectrum, and the noise-reduced speech spectrum can be synthesized by sub-band to obtain the noise-reduced narrowband speech signal.

[0057] Optionally, based on the frequency domain feature data, performing noise reduction processing on the noisy narrow-band speech signal to obtain the noise-reduced narrow-band speech signal includes:

[0058] According to the frequency domain characteristic data, the mask value corresponding to each frequency point in the amplitude spectrum is determined;

[0059] Based on the mask value, the amplitude value of the corresponding frequency point in the first amplitude spectrum is adjusted to obtain a second amplitude spectrum;

[0060] Determine a second frequency spectrum after noise reduction based on the second amplitude spectrum and the phase spectrum;

[0061] A noise-reduced narrow-band speech signal is determined according to the second frequency spectrum.

[0062] In this optional embodiment, a specific method is provided for performing noise reduction processing on a noisy narrowband speech signal based on frequency domain feature data to obtain a noise-reduced narrowband speech signal: first, the frequency band feature data is input into the noise reduction model to determine the mask value corresponding to each frequency point in the amplitude spectrum. Further, the amplitude value of each frequency point in the first amplitude spectrum is multiplied by the mask value corresponding to the frequency point to obtain a second amplitude spectrum. Further, the second amplitude spectrum is combined with the phase spectrum contained in the first amplitude spectrum to form a second spectrum after noise reduction. Finally, the second spectrum is subjected to sub-band synthesis to obtain a noise-reduced narrowband speech signal. Predicting the mask value corresponding to each frequency point based on the frequency domain feature data can achieve a better noise reduction effect than directly predicting the mask value based on the first amplitude spectrum. Moreover, before performing band expansion, denoising the noisy narrowband speech signal can improve the quality of the speech signal obtained by subsequent band expansion.

[0063] Among them, the process of predicting the mask value through the denoising model is as follows Figure 2b As shown, after converting the narrowband speech signal to the frequency domain and performing feature extraction to obtain frequency domain feature data, the frequency domain feature data is input into the denoising model to obtain the mask value corresponding to each frequency point in the amplitude spectrum output by the denoising model. Among them, the denoising model includes at least one layer of convolutional neural network, at least one layer of recursive neural network, a fully connected layer and an activation layer. Specifically, firstly, feature extraction is performed on the frequency domain feature data through at least one layer of convolutional neural network to obtain the first frequency domain feature. Then, the first spectrum feature is recursively learned through at least one layer of recursive neural network to obtain the second spectrum feature. Further, the dimension of the second frequency domain feature is adjusted through the fully connected layer so that the dimension of the second frequency domain feature is consistent with the dimension of the frequency domain feature data. Through the activation layer, the output result of the fully connected layer is activated to obtain the mask value corresponding to each frequency point in the amplitude spectrum. Through the structure of the convolutional neural network and the recursive neural network, the denoising model has a small amount of calculation and is more friendly to instant messaging applications.

[0064] S240, determining the low-frequency transformation features of the denoised narrow-band speech signal in the transform domain, and inputting the low-frequency transformation features into a spread spectrum model to obtain high-frequency transformation features predicted by the spread spectrum model; the spread spectrum model includes at least one layer of residual blocks; the residual block includes a dilated convolutional neural network.

[0065] Among them, the spread spectrum model is a feedforward neural network with a fully convolutional structure, including at least one layer of residual blocks, and each residual block contains a hollow convolutional neural network structure.

[0066] In the disclosed embodiment, after the noisy narrowband speech signal is subjected to denoising to obtain the denoised narrowband speech signal, the denoised narrowband speech signal is encoded by an encoder with a one-dimensional convolution structure, and the denoised narrowband speech is transformed into a transform domain to obtain low-frequency transformation features. Further, the low-frequency transformation features are input into a spread spectrum model to obtain high-frequency transformation features predicted by the spread spectrum model. By performing frequency band expansion using a spread spectrum model with a full convolution structure, the model calculation amount is relatively small, so that the frequency band expansion method provided in this embodiment can be applied to instant messaging devices. In addition, the use of a residual block structure including a void convolutional neural network can accelerate the convergence speed while widening the receptive field, thereby improving the real-time performance of frequency band expansion.

[0067] Optionally, the normalization layer in the atrous convolutional neural network uses a weight normalization method to perform feature normalization operations.

[0068] In this optional embodiment, the atrous convolutional neural network included in the spread spectrum model includes a convolution layer, a pooling layer, a normalization layer, and an activation layer. Among them, the normalization layer uses a normalization method to perform feature normalization operations. Compared with the instance normalization method, the weight normalization method only normalizes the weights, which can avoid information loss caused by directly normalizing the feature data, thereby improving the spread spectrum effect of the speech signal.

[0069] S250: Determine a broadband speech signal according to the high frequency conversion feature.

[0070] In the disclosed embodiment, after obtaining the high-frequency transformation features predicted by the spread spectrum model, the high-frequency transformation features are decoded to obtain the high-frequency part of the speech signal. Finally, the narrowband speech signal and the high-frequency part of the speech signal can be combined to obtain the final broadband speech signal.

[0071] Figure 2c It is a signal schematic diagram of a noisy narrow-band speech signal in the time domain and the frequency domain respectively. The noisy narrow-band speech signal contains steady-state noise and non-steady-state noise. Figure 2d 1 is a schematic diagram of a broadband speech signal in the time domain and the frequency domain after the noisy narrowband speech signal is subjected to noise reduction and frequency band expansion by the solution provided by the embodiment of the present disclosure, Figure 2dBoth stationary noise and non-stationary noise are suppressed, and the frequency band of the noise-reduced speech signal is expanded.

[0072] The technical solution of the disclosed embodiment performs noise reduction processing on a noisy narrowband speech signal to obtain a noise-reduced narrowband speech signal, and then determines the low-frequency transformation features of the noise-reduced narrowband speech signal in the transform domain, and inputs the low-frequency transformation features into a spread spectrum model to obtain high-frequency transformation features predicted by the spread spectrum model. Noise reduction is performed before frequency band expansion, which can achieve frequency band expansion of the speech signal in a noisy scenario and improve the quality of the broadband speech signal obtained by spectrum expansion.

[0073] Figure 3a FIG. 1 is a schematic diagram of a frequency band extension method in an embodiment of the present disclosure, which further refines the above embodiment and provides specific steps after determining the broadband speech signal according to the high frequency conversion characteristics. Figure 3a A frequency band extension method provided in an embodiment of the present disclosure is described, including the following:

[0074] S310: Perform noise reduction processing on the noisy narrow-band speech signal to obtain a noise-reduced narrow-band speech signal.

[0075] S320: Determine low-frequency transformation features of the noise-reduced narrow-band speech signal in the transformation domain, and perform frequency band expansion on the low-frequency transformation features to obtain high-frequency transformation features.

[0076] S330: Determine a broadband speech signal according to the high frequency conversion feature.

[0077] S340: Perform noise estimation on the broadband speech signal to obtain estimated noise.

[0078] After the noisy narrowband speech signal is subjected to noise reduction processing to obtain the noise-reduced narrowband speech signal, the noise-reduced narrowband speech signal is further spread to obtain a broadband speech signal. However, since the broadband speech signal is predicted based on the noise-reduced narrowband speech signal, the prediction process may generate noise. Specifically, it can be reflected in the broadband speech signal containing continuous steady-state noise.

[0079] In the disclosed embodiment, in order to suppress the noise generated in the prediction process, the noise of the wideband speech signal can be further estimated by a noise suppression algorithm to obtain estimated noise, so as to suppress the signal generated in the wideband speech signal generation process.

[0080] S350: Perform noise suppression on the broadband speech signal based on the estimated noise.

[0081] In the embodiment of the present disclosure, after obtaining the estimated noise based on the broadband voice signal, the broadband voice signal can be subjected to noise suppression based on the estimated noise, thereby further improving the quality of the voice signal after spread spectrum, thereby improving the communication user experience.

[0082] S360: Perform equalization processing on the broadband speech signal through a dynamic gain control algorithm.

[0083] In the disclosed embodiment, when the noise reduction narrowband voice signal is spread, the high-frequency components of some segments may be too strong, which reduces the user's call experience. Therefore, the broadband voice signal can be further balanced through a dynamic gain control algorithm, so that the high-frequency components in the broadband voice signal are more balanced, further improving the quality of the voice signal.

[0084] Figure 3b This is a schematic diagram of the signal in the time domain and frequency domain of the broadband speech signal after noise reduction and bandwidth expansion of the noisy narrowband speech signal. It can be seen that Figure 3b It contains some noise generated during frequency band expansion, and some high-frequency components are too strong, which will affect the user's calling experience. Figure 3c It is a signal schematic diagram of the wideband speech signal in the time domain and the frequency domain after noise suppression and dynamic gain control are performed on the generated wideband speech signal. Figure 3c In this process, the noise generated when the frequency band is expanded is suppressed, and the parts with excessive high-frequency components are equalized, which further improves the quality of broadband speech signals.

[0085] The technical solution of the embodiment of the present disclosure performs noise reduction processing on the noisy narrowband speech signal to obtain the noise-reduced narrowband speech signal, and then determines the low-frequency transformation characteristics of the noise-reduced narrowband speech signal in the transform domain, and performs frequency band expansion on the low-frequency transformation characteristics to obtain high-frequency transformation characteristics, and determines the broadband speech signal according to the high-frequency transformation characteristics. Further, the broadband speech signal is subjected to noise estimation to obtain estimated noise, and based on the estimated noise, the broadband speech signal is subjected to noise suppression, and the broadband speech signal is subjected to equalization processing through a dynamic gain control algorithm. On the one hand, by performing noise reduction before frequency band expansion, the frequency band expansion of the speech signal can be achieved in a noisy scene. On the other hand, by further performing noise suppression and signal equalization processing on the broadband speech signal obtained by spectrum expansion, the noise generated when the high-frequency part is generated can be removed, so that the high-frequency component is more uniform, and the quality of the broadband speech signal is further improved.

[0086] According to an embodiment of the present disclosure, Figure 41 is a structural diagram of a frequency band extension device in an embodiment of the present disclosure, and the embodiment of the present disclosure is applicable to the case where frequency band extension is performed on the basis of noise reduction processing. The device is implemented by software and / or hardware, and is specifically configured in an electronic device with certain data computing capabilities.

[0087] like Figure 4 The frequency band extension device 400 shown in the figure comprises: a speech noise reduction module 410, a frequency band extension module 420 and a broadband speech signal determination module 430; wherein,

[0088] The speech noise reduction module 410 is used to perform noise reduction processing on the noisy narrow-band speech signal to obtain a noise-reduced narrow-band speech signal;

[0089] The frequency band extension module 420 is used to determine the low-frequency transformation features of the noise-reduced narrow-band speech signal in the transformation domain, and perform frequency band extension on the low-frequency transformation features to obtain high-frequency transformation features;

[0090] The wideband speech signal determination module 430 is used to determine the wideband speech signal according to the high frequency conversion feature.

[0091] The technical solution of the embodiment of the present disclosure first performs noise reduction processing on the noisy narrowband speech signal, then determines the low-frequency transformation features of the denoised narrowband speech signal after the noise reduction processing in the transform domain, and performs frequency band expansion on the low-frequency transformation features to obtain high-frequency transformation features, and finally determines the broadband speech signal based on the high-frequency transformation features, so as to achieve frequency band expansion of the speech signal in a noisy scene.

[0092] Furthermore, the frequency band extension module 420 is specifically configured to:

[0093] The low-frequency transformation feature is input into a spread spectrum model to obtain a high-frequency transformation feature predicted by the spread spectrum model; the spread spectrum model includes at least one layer of residual blocks; and the residual block includes a hole convolutional neural network.

[0094] Furthermore, the speech noise reduction module 410 includes:

[0095] A first spectrum determining unit, configured to determine a first spectrum of the noisy narrow-band speech signal; the first spectrum includes a first amplitude spectrum and a phase spectrum;

[0096] A frequency domain feature data determination unit, configured to convert the amplitude value of each frequency point in the first amplitude spectrum into a logarithmic domain to obtain frequency domain feature data matching the noisy narrow-band speech signal;

[0097] The speech noise reduction unit is used to perform noise reduction processing on the noisy narrow-band speech signal based on the frequency domain feature data to obtain a noise-reduced narrow-band speech signal.

[0098] Furthermore, the speech noise reduction unit is specifically used for:

[0099] Determining, according to the frequency domain characteristic data, a mask value corresponding to each frequency point in the amplitude spectrum;

[0100] Based on the mask value, adjusting the amplitude value of the corresponding frequency point in the first amplitude spectrum to obtain a second amplitude spectrum;

[0101] Determine a second frequency spectrum after noise reduction based on the second amplitude spectrum and the phase spectrum;

[0102] A noise-reduced narrow-band speech signal is determined according to the second frequency spectrum.

[0103] Furthermore, the normalization layer in the atrous convolutional neural network uses a weight normalization method to perform feature normalization operations.

[0104] Furthermore, the frequency band extension device 400 further includes:

[0105] A noise estimation module, configured to perform noise estimation on the broadband speech signal to obtain estimated noise after determining the broadband speech signal according to the high frequency transformation feature;

[0106] The noise suppression module is used to perform noise suppression on the wideband speech signal based on the estimated noise.

[0107] Furthermore, the frequency band extension device 400 further includes:

[0108] The signal equalization processing module is used to perform equalization processing on the broadband speech signal through a dynamic gain control algorithm.

[0109] The frequency band extension device provided in the embodiments of the present disclosure can execute the frequency band extension method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0110] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0111] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0112] Figure 5A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0113] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0114] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0115] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the frequency band extension method. For example, in some embodiments, the frequency band extension method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the frequency band extension method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the frequency band extension method in any other appropriate manner (e.g., by means of firmware).

[0116] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0118] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0119] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0120] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0121] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0122] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0123] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A frequency band extension method, comprising: Determine a first spectrum of a noisy narrowband speech signal; the first spectrum includes a first amplitude spectrum and a phase spectrum; Converting the amplitude value of each frequency point in the first amplitude spectrum to a logarithmic domain to obtain frequency domain feature data matched to the noisy narrowband speech signal; Inputting the frequency domain feature data into a denoising model to obtain a mask value corresponding to each frequency point in the amplitude spectrum; wherein the denoising model includes at least one layer of convolutional neural network, at least one layer of recurrent neural network, a fully connected layer, and an activation layer; Based on the mask value, adjusting the amplitude value of the corresponding frequency point in the first amplitude spectrum to obtain a second amplitude spectrum; Determine a second frequency spectrum after noise reduction based on the second amplitude spectrum and the phase spectrum; determining a noise-reduced narrow-band speech signal according to the second spectrum; Determine the low-frequency transformation features of the denoised narrow-band speech signal in the transform domain, and input the low-frequency transformation features into the spread spectrum model to obtain the high-frequency transformation features predicted by the spread spectrum model; wherein the low-frequency transformation features are obtained by encoding the denoised narrow-band speech signal through an encoder with a one-dimensional convolution structure, and transforming the denoised narrow-band speech signal to the transform domain; the spread spectrum model is a feedforward neural network with a full convolution structure, and the spread spectrum model includes at least one layer of residual blocks; the residual block contains a hollow convolution neural network; the hollow convolution neural network includes a convolution layer, a pooling layer, a normalization layer and an activation layer; the normalization layer in the hollow convolution neural network uses a weight normalization method to perform feature normalization operation; A broadband speech signal is determined according to the high frequency conversion feature.

2. The method according to claim 1, wherein according to the high frequency transformation feature, After determining the wideband speech signal, it also includes: Performing noise estimation on the broadband speech signal to obtain estimated noise; Based on the estimated noise, noise suppression is performed on the wideband speech signal.

3. The method according to any one of claims 1 to 2, further comprising: after determining the broadband speech signal according to the high frequency conversion feature: The broadband speech signal is equalized by a dynamic gain control algorithm.

4. A frequency band extension device, comprising: Speech noise reduction module, including: A first spectrum determination unit, used to determine a first spectrum of the noisy narrow-band speech signal; the first spectrum includes a first amplitude spectrum and a phase spectrum; A frequency domain feature data determination unit, configured to convert the amplitude value of each frequency point in the first amplitude spectrum into a logarithmic domain to obtain frequency domain feature data matching the noisy narrow-band speech signal; A speech denoising unit, configured to input the frequency domain feature data into a denoising model to obtain a mask value corresponding to each frequency point in the amplitude spectrum; wherein the denoising model comprises at least one layer of convolutional neural network, at least one layer of recurrent neural network, a fully connected layer and an activation layer; based on the mask value, adjusting the amplitude value of the corresponding frequency point in the first amplitude spectrum to obtain a second amplitude spectrum; based on the second amplitude spectrum and the phase spectrum, determining a second spectrum after denoising; and determining a denoised narrowband speech signal according to the second spectrum; A frequency band extension module, used for determining the low-frequency transformation features of the denoised narrow-band speech signal in the transform domain, and inputting the low-frequency transformation features into the spread spectrum model to obtain the high-frequency transformation features predicted by the spread spectrum model; wherein the low-frequency transformation features are obtained by encoding the denoised narrow-band speech signal through an encoder with a one-dimensional convolution structure, and transforming the denoised narrow-band speech signal into the transform domain; the spread spectrum model is a feedforward neural network with a full convolution structure, and the spread spectrum model includes at least one layer of residual blocks; the residual block includes a hollow convolution neural network; the hollow convolution neural network includes a convolution layer, a pooling layer, a normalization layer and an activation layer; the normalization layer in the hollow convolution neural network uses a weight normalization method to perform feature normalization operation; The broadband speech signal determination module is used to determine the broadband speech signal according to the high frequency conversion feature.

5. The apparatus according to claim 4, further comprising: A noise estimation module is used to estimate the noise according to the high frequency transformation characteristics. After determining the wideband speech signal, performing noise estimation on the wideband speech signal to obtain estimated noise; The noise suppression module is used to perform noise suppression on the wideband speech signal based on the estimated noise.

6. The device according to any one of claims 4-5, further comprising: The signal equalization processing module is used to perform equalization processing on the broadband speech signal through a dynamic gain control algorithm.

7. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the frequency band extension method according to any one of claims 1 to 3.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the frequency band extension method according to any one of claims 1-3.

9. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the frequency band extension method according to any one of claims 1 to 3.

Images