A method, device, equipment and storage medium for extending the frequency band of a voice signal

Abstract

The present application discloses a method for extending the frequency band of a speech signal, comprising the following steps: determining a source speech and obtaining a signal spectrum of the source speech; obtaining a harmonic signal and an unvoiced noise signal of an extended speech corresponding to the source speech based on the signal spectrum of the source speech; performing speech synthesis processing on the source speech, the signal spectrum of the source speech, and the harmonic signal and unvoiced noise signal of the extended speech corresponding to the source speech, to obtain the extended speech corresponding to the source speech. The application of the technical solution provided by the present application improves the speech quality of the speech signal obtained after the frequency band extension, thereby improving the user's listening experience. The present application also discloses a speech signal frequency band extension device, equipment, storage medium and computer program product, which have corresponding technical effects.

Description

Technical Field

[0001] The present application relates to the field of signal processing technology, and in particular to a method, device, equipment and storage medium for extending the frequency band of a speech signal. Background Art

[0002] In many scenarios such as voice calls between two or more users, entertainment interactions, etc., voice signal collection and transmission are involved. For example, voice signals are collected and encoded on the speaker's side, and then transmitted to the receiving end through the communication network. The receiving end can decode and output the received voice signals so that the other user can hear the specific voice signals.

[0003] However, due to the influence of factors such as channel bandwidth, coding method, and cost in the communication network, when collecting voice signals, most of them are collected at low sampling rates, or, after collecting high sampling rate voice signals, when transmitting voice signals, only the low-frequency part of the voice signals is retained, so that the receiving end obtains low sampling rate voice signals. However, the voice quality of low sampling rate voice signals is poor and cannot meet the real listening experience. Therefore, after receiving the voice signal at the receiving end, it is necessary to first expand the frequency band of the low sampling rate voice signal, restore the low sampling rate voice signal to a high sampling rate voice signal, and then play it to improve the user's listening experience.

[0004] Therefore, how to effectively expand the frequency band of a speech signal and improve the speech quality is a technical problem that those skilled in the art need to solve. Summary of the invention

[0005] The purpose of this application is to provide a method, device, equipment and storage medium for expanding the frequency band of a speech signal to improve the voice quality of the speech signal obtained after the frequency band expansion and enhance the listening experience.

[0006] In order to solve the above technical problems, this application provides the following technical solutions:

[0007] A method for extending the frequency band of a speech signal, comprising:

[0008] Determine a source speech, and obtain a signal spectrum of the source speech;

[0009] Based on the signal spectrum of the source speech, obtaining a harmonic signal and an unvoiced noise signal of the extended speech corresponding to the source speech;

[0010] Speech synthesis processing is performed on the source speech, the signal spectrum of the source speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech to obtain the extended speech corresponding to the source speech.

[0011] In a specific implementation of the present application, it also includes:

[0012] Based on the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech, predict the high-frequency signal missing from the extended speech corresponding to the source speech;

[0013] The performing speech synthesis processing on the source speech, the signal spectrum of the source speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech to obtain the extended speech corresponding to the source speech includes:

[0014] Speech synthesis processing is performed on the source speech, the signal spectrum of the source speech, and the high-frequency signal missing from the extended speech corresponding to the source speech to obtain the extended speech corresponding to the source speech.

[0015] In a specific implementation of the present application, obtaining the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech based on the signal spectrum of the source speech includes:

[0016] Performing encoding processing on the signal spectrum of the source speech to obtain an encoded signal of the source speech;

[0017] Generate a harmonic signal of an extended speech corresponding to the source speech by using the fundamental frequency of the source speech and the coded signal of the source speech;

[0018] The encoded signal of the source speech is used to generate an unvoiced noise signal of the extended speech corresponding to the source speech.

[0019] In a specific implementation of the present application, the obtaining of the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech based on the signal spectrum of the source speech, and the predicting of the high-frequency signal missing from the extended speech corresponding to the source speech based on the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech, include:

[0020] The signal spectrum of the source speech is input into the source filter model, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are obtained through prediction by the source filter model. The harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are superimposed to obtain the high-frequency signal missing from the extended speech corresponding to the source speech output by the source filter model.

[0021] In a specific implementation of the present application, the source filter model is obtained by pre-training through the following steps:

[0022] Obtain multiple training data, each training data corresponds to a data pair, each data pair includes a high sampling rate speech and a low sampling rate speech, and the low sampling rate speech in each data pair is generated according to the corresponding high sampling rate speech;

[0023] The pre-established source filter initial model is trained in turn using the high sampling rate speech and the low sampling rate speech in each training data until the loss error of the source filter initial model is less than a preset first error threshold, thereby obtaining the source filter model.

[0024] In a specific implementation of the present application, the method of sequentially using the high sampling rate speech and the low sampling rate speech in each training data to train the pre-established source filter initial model includes:

[0025] For each piece of training data, obtain the signal spectrum of the low sampling rate speech in the current training data;

[0026] Inputting the signal spectrum of the current low sampling rate speech into a pre-established source filter initial model, obtaining the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech through prediction by the source filter initial model, and superimposing the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech to obtain the high-frequency signal missing from the extended speech corresponding to the current low sampling rate speech output by the source filter initial model;

[0027] Based on the comparison result of the high-frequency signal missing from the extended speech corresponding to the current low-sampling rate speech and the high-sampling rate speech in the current training data, the parameters of the source filter initial model are adjusted to train the source filter initial model after the parameter adjustment using the high-sampling rate speech and the low-sampling rate speech in the subsequent training data.

[0028] In a specific implementation of the present application, performing speech synthesis processing on the source speech, the signal spectrum of the source speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech to obtain the extended speech corresponding to the source speech includes:

[0029] The source speech, the signal spectrum of the source speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are input into a sequence generation model, and speech synthesis processing is performed by the sequence generation model to obtain the extended speech corresponding to the source speech output by the sequence generation model.

[0030] In a specific embodiment of the present application, the sequence generation model is obtained by pre-training through the following steps:

[0031] Obtain multiple training data, each training data corresponds to a data pair, each data pair includes a high sampling rate speech and a low sampling rate speech, and the low sampling rate speech in each data pair is generated according to the corresponding high sampling rate speech;

[0032] The pre-established sequence generation initial model is trained in turn using the high sampling rate speech and the low sampling rate speech in each training data until the loss error of the sequence generation initial model is less than a preset second error threshold, thereby obtaining the sequence generation model.

[0033] In a specific implementation of the present application, the method of sequentially using the high sampling rate speech and the low sampling rate speech in each piece of training data to train the pre-established sequence generation initial model includes:

[0034] For each piece of training data, respectively obtain the signal spectrum of the low sampling rate speech in the current training data and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech;

[0035] Inputting the current low sampling rate speech, the signal spectrum of the current low sampling rate speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech into a pre-established sequence generation initial model, performing speech synthesis processing through the sequence generation initial model, and obtaining the extended speech corresponding to the current low sampling rate speech output by the sequence generation initial model;

[0036] Based on the comparison result of the extended speech corresponding to the current low sampling rate speech and the high sampling rate speech in the current training data, the parameters of the initial model for sequence generation are adjusted to train the initial model for sequence generation after the parameter adjustment using the high sampling rate speech and the low sampling rate speech in the latter training data.

[0037] In a specific implementation of the present application, the obtaining of multiple pieces of training data includes:

[0038] Collect multiple high sampling rate voices;

[0039] Extract low sampling rate speech from each high sampling rate speech respectively;

[0040] The data pairs consisting of each high sampling rate speech and the corresponding low sampling rate speech are determined as training data one by one.

[0041] In a specific implementation of the present application, after the multiple high sampling rate voices are collected and before the low sampling rate voices are extracted from each high sampling rate voice, the method further includes:

[0042] Each high sampling rate speech collected is cleaned.

[0043] A voice signal frequency band extension device, comprising:

[0044] A source speech determination module, used to determine the source speech;

[0045] A spectrum acquisition module, used to acquire the signal spectrum of the source speech;

[0046] A signal acquisition module, used for obtaining a harmonic signal and an unvoiced noise signal of an extended speech corresponding to the source speech based on a signal spectrum of the source speech;

[0047] The extended speech acquisition module is used to perform speech synthesis processing on the source speech, the signal spectrum of the source speech, and the harmonic signal and unvoiced noise signal of the extended speech corresponding to the source speech to obtain the extended speech corresponding to the source speech.

[0048] A voice signal frequency band extension device, comprising:

[0049] Memory for storing computer programs;

[0050] A processor is used to implement the steps of any of the above-mentioned methods for extending the frequency band of a speech signal when executing the computer program.

[0051] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned methods for extending the frequency band of a speech signal are implemented.

[0052] A computer program product, comprising computer instructions, which are stored in a computer-readable storage medium and are suitable for being read and executed by a processor, so that a computer device having the processor executes any of the above-mentioned methods for extending the frequency band of a speech signal.

[0053] By applying the technical solution provided in the embodiments of the present application, the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are first obtained based on the signal spectrum of the source speech, which conforms to the actual signal composition, and then the source speech, the signal spectrum of the source speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are subjected to speech synthesis processing. The obtained extended speech corresponding to the source speech is more comprehensive, and therefore, is conducive to improving the speech quality of the speech signal obtained after frequency band expansion, thereby enhancing the user's listening experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 This is a flow chart of an implementation method of a voice signal frequency band extension method in an embodiment of the present application;

[0056] Figure 2 This is a structural schematic diagram of a voice signal frequency band extension system in an embodiment of the present application;

[0057] Figure 3 This is a schematic diagram of the structure of a voice signal frequency band extension device in an embodiment of the present application;

[0058] Figure 4 This is a structural diagram of a voice signal frequency band extension device in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0060] The core of this application is to provide a method for extending the frequency band of a voice signal. This method can be applied to various scenarios such as voice calls, entertainment interactions, voice repair, voice synthesis, voice recognition, and voice playback. Taking the voice call scenario as an example, due to the influence of many factors such as channel bandwidth, coding method, and cost in the communication network, when the sender's device collects the sender's voice signal, most of the collected voice is low-sampling rate voice. The sender's device sends the collected low-sampling rate voice to the receiving device through the communication network. After the receiving device receives the low-sampling rate voice, it can perform frequency band expansion on it through the technical solution provided in the embodiment of this application. The obtained extended voice is a high-sampling rate voice, and then the high-sampling rate voice is played and output to the receiver, so that the receiver will enjoy a real listening experience.

[0061] The speech signal frequency band expansion method provided in the present application, after determining the source speech, further obtains the signal spectrum of the source speech, and then obtains the harmonic signal and unvoiced noise signal of the extended speech corresponding to the source speech based on the signal spectrum of the source speech, which conforms to the actual signal composition, and finally performs speech synthesis processing on the source speech, the signal spectrum of the source speech, and the harmonic signal and unvoiced noise signal of the extended speech corresponding to the source speech, so that the obtained extended speech corresponding to the source speech is more comprehensive, and therefore is conducive to improving the speech quality of the speech signal obtained after the frequency band expansion, thereby enhancing the user's listening experience.

[0062] See also Figure 1 As shown, it is a flow chart of an implementation of a method for extending the frequency band of a speech signal provided in an embodiment of the present application. The method may include the following steps:

[0063] S110: Determine the source speech and obtain the signal spectrum of the source speech.

[0064] In the embodiment of the present application, the source voice can be determined according to actual needs. For example, in a voice call scenario, the low sampling rate voice received by the receiving device is determined as the source voice. In another example, in a speaker playback scenario, the low sampling rate voice currently to be played is determined as the source voice. The source voice can be a regular voice signal or a singing voice.

[0065] After the source speech is determined, the signal spectrum of the source speech can be further obtained. The speech signal is not a stationary signal, but it can be regarded as stationary in a short period of time. Therefore, after the source speech is determined, the source speech can be processed by short-time Fourier transform (STFT), so that the spectrum of multiple signal segments arranged in time order can be obtained, that is, the signal spectrum of the source speech, to facilitate the processing of subsequent steps.

[0066] S120: Based on the signal spectrum of the source speech, obtain a harmonic signal and an unvoiced noise signal of the extended speech corresponding to the source speech.

[0067] The speech signal is composed of a harmonic signal and an unvoiced noise signal. The unvoiced noise signal refers to the signal corresponding to the sound when the vocal cords do not vibrate during pronunciation. After obtaining the signal spectrum of the source speech, the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech can be obtained based on the signal spectrum of the source speech. The harmonic signal and the unvoiced noise signal obtained in this way are more consistent with the actual signal composition.

[0068] S130: Perform speech synthesis processing on the source speech, the signal spectrum of the source speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech to obtain the extended speech corresponding to the source speech.

[0069] After the source speech is determined, the signal spectrum of the source speech is obtained, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are obtained, these signals can be processed by speech synthesis to obtain the extended speech corresponding to the source speech. In this way, the obtained extended speech is more comprehensive.

[0070] By applying the method provided in the embodiment of the present application, the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are first obtained based on the signal spectrum of the source speech, which conforms to the actual signal composition, and then the source speech, the signal spectrum of the source speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are subjected to speech synthesis processing. The obtained extended speech corresponding to the source speech is more comprehensive, and therefore, is conducive to improving the speech quality of the speech signal obtained after frequency band expansion, thereby enhancing the user's listening experience.

[0071] In one embodiment of the present application, the method may further include the following steps:

[0072] Based on the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech, predict the high-frequency signal missing from the extended speech corresponding to the source speech;

[0073] Accordingly, step S130 may include the following steps:

[0074] Speech synthesis processing is performed on the source speech, the signal spectrum of the source speech, and the high-frequency signal missing from the extended speech corresponding to the source speech to obtain the extended speech corresponding to the source speech.

[0075] In an embodiment of the present application, after obtaining the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech based on the signal spectrum of the source speech, the high-frequency signal missing from the extended speech corresponding to the source speech can be predicted based on the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech. Specifically, the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech can be superimposed, and then the high-frequency part can be extracted from the superimposed speech signal to obtain the high-frequency signal missing from the extended speech corresponding to the source speech.

[0076] After predicting the high-frequency signal missing from the extended voice corresponding to the source voice, the source voice, the signal spectrum of the source voice, and the high-frequency signal missing from the extended voice corresponding to the source voice can be processed by voice synthesis to obtain the extended voice corresponding to the source voice. The extended voice obtained in this way has a better high-frequency acquisition effect, higher voice quality, and a more realistic listening experience for users.

[0077] In one embodiment of the present application, step S120 may include the following steps:

[0078] Step 1: Encode the signal spectrum of the source speech to obtain the encoded signal of the source speech;

[0079] Step 2: Generate a harmonic signal of the extended speech corresponding to the source speech by using the fundamental frequency of the source speech and the coded signal of the source speech;

[0080] Step 3: Using the coded signal of the source speech, generate the unvoiced noise signal of the extended speech corresponding to the source speech.

[0081] For ease of description, the above three steps are combined for explanation.

[0082] After the signal spectrum of the source speech is obtained, the signal spectrum of the source speech may be encoded, such as by using an encoder to encode the spectrum to obtain an encoded signal of the source speech.

[0083] Because any complex waveform can be decomposed into a fundamental wave and higher harmonics that are multiples of the fundamental wave, after obtaining the coded signal of the source speech, the fundamental frequency of the source speech and the coded signal of the source speech can be further used to generate the harmonic signal of the extended speech corresponding to the source speech. Specifically, the fundamental frequency of the source speech and the coded signal of the source speech can be input into a harmonic generator, and the harmonic signal of the extended speech corresponding to the source speech can be generated by the harmonic generator. Among them, the fundamental frequency of the source speech, that is, the vibration frequency of the vocal cords of a person when pronouncing, can be obtained by analyzing the source speech.

[0084] At the same time, after obtaining the coded signal of the source speech, the coded signal of the source speech can be further used to generate the unvoiced noise signal of the extended speech corresponding to the source speech. Specifically, the coded signal of the source speech can be input into the noise generator, and the unvoiced noise signal of the extended speech corresponding to the source speech is generated by the noise generator.

[0085] The harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are obtained, and the generation process of the speech signal is reasonably analyzed, which can improve the accuracy of frequency band expansion of the source speech.

[0086] In one embodiment of the present application, after obtaining the signal spectrum of the source speech, the signal spectrum of the source speech can be input into a source filter model. After prediction by the source filter model, the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are obtained, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are superimposed to obtain the high-frequency signal missing from the extended speech corresponding to the source speech output by the source filter model.

[0087] In the embodiment of the present application, the source filter model can be obtained by pre-training, such as using the pre-obtained training data including high sampling rate speech and low sampling rate speech to train the source filter model. The source filter model can describe the entire pronunciation process with input excitation and filter according to the pronunciation mechanism of the human being. Figure 2As shown, the source filter model may include an encoder, a harmonic generator, a noise generator, an adder, etc.

[0088] After obtaining the signal spectrum of the source speech, the signal spectrum of the source speech can be input into the source filter model, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech can be obtained through the prediction of the source filter model. Specifically, the source filter model can be first encoded by the encoder to obtain the encoded signal of the source speech, and then the harmonic generation process can be performed by the harmonic generator of the source filter model to generate the harmonic signal of the extended speech corresponding to the source speech. At the same time, the noise generator of the source filter model is used to perform noise generation processing to generate the unvoiced noise signal of the extended speech corresponding to the source speech. The source filter model reasonably analyzes the generation process of the speech signal. After the superimposer of the source filter model can superimpose the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech, the source filter model can output the high-frequency signal missing from the extended speech corresponding to the source speech.

[0089] The harmonic generator of the source filter model is used to generate the harmonic signal, and the noise generator of the source filter model is used to generate the unvoiced noise signal. The generation process of the speech signal is reasonably analyzed, which can improve the accuracy of frequency band expansion of the source speech.

[0090] In one embodiment of the present application, the source filter model can be pre-trained by the following steps:

[0091] The first step is to obtain multiple training data, each training data corresponds to a data pair, each data pair includes a high sampling rate speech and a low sampling rate speech, and the low sampling rate speech in each data pair is generated based on the high sampling rate speech;

[0092] The second step: use the high sampling rate speech and low sampling rate speech in each training data to train the pre-established source filter initial model in turn until the loss error of the source filter initial model is less than the preset first error threshold, and obtain the source filter model.

[0093] For ease of description, the above two steps are combined for illustration.

[0094] In an embodiment of the present application, a source filter initial model can be pre-established based on a neural network, and the source filter initial model can include an encoder, a harmonic generator, a noise generator, a superimposer, etc., and each parameter is initialized. The source filter model can be obtained by pre-training the source filter initial model using training data, and the source filter model has the same structure as the source filter initial model.

[0095] Multiple training data can be obtained first, each training data corresponds to a data pair, each data pair includes a high sampling rate speech and a low sampling rate speech, the low sampling rate speech in each data pair is generated based on the corresponding high sampling rate speech, that is, the high sampling rate speech and the low sampling rate speech in each training data correspond to each other.

[0096] Specifically, a plurality of high sampling rate speech can be collected first, and then the low sampling rate speech can be extracted from each high sampling rate speech, and then the data pairs consisting of each high sampling rate speech and the corresponding low sampling rate speech can be determined as training data one by one. The low sampling rate speech in each training data obtained in this way is extracted from the high sampling rate speech in the training data, ensuring the corresponding relationship between the high sampling rate speech and the low sampling rate speech in each training data.

[0097] After collecting multiple high sampling rate speech and before extracting low sampling rate speech from each high sampling rate speech, each high sampling rate speech collected can also be cleaned to obtain a clean speech signal, thereby improving the subsequent training efficiency of the model.

[0098] After obtaining multiple training data, the pre-established source filter initial model can be trained in turn using the high sampling rate speech and low sampling rate speech in each training data, and the loss error of the source filter initial model can be determined. When the loss error of the source filter initial model is less than the preset first error threshold, it can be considered that the source filter initial model obtained by the current training has a high accuracy, and it can be determined as the source filter model and applied to the actual speech signal band extension scenario. When the loss error of the source filter initial model is greater than or equal to the first error threshold, it can be considered that the accuracy of the source filter initial model obtained by the current training is not high, and it can continue to be trained.

[0099] The first error threshold can be set and adjusted according to actual conditions.

[0100] In one embodiment of the present application, training a pre-established source filter initial model using the high sampling rate speech and the low sampling rate speech in each training data in turn may include the following steps:

[0101] Step 1: For each piece of training data, obtain the signal spectrum of the low sampling rate speech in the current training data;

[0102] Step 2: Input the signal spectrum of the current low sampling rate speech into a pre-established source filter initial model, obtain the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech through the prediction of the source filter initial model, and superimpose the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech to obtain the high-frequency signal missing from the extended speech corresponding to the current low sampling rate speech output by the source filter initial model;

[0103] Step 3: Based on the comparison results of the high-frequency signal missing from the extended speech corresponding to the current low-sampling rate speech and the high-sampling rate speech in the current training data, the parameters of the source filter initial model are adjusted to train the parameter-adjusted source filter initial model using the high-sampling rate speech and low-sampling rate speech in the subsequent training data.

[0104] For ease of description, the above three steps are combined for explanation.

[0105] In the embodiment of the present application, the low sampling rate speech in each piece of training data is generated according to the corresponding high sampling rate speech, and each piece of training data can be used in turn to train the pre-established source filter initial model.

[0106] For each piece of training data, the low sampling rate speech in the current training data can be short-time Fourier transform processed to obtain the signal spectrum of the current low sampling rate speech. The current training data is the training data targeted by the current operation, and the current low sampling rate speech is the low sampling rate speech in the current training data.

[0107] After obtaining the signal spectrum of the current low sampling rate speech, it can be input into the pre-established source filter initial model. The source filter initial model can perform corresponding processing on the signal spectrum of the current low sampling rate speech through the various parts it includes. Specifically, the encoder can be used to perform encoding processing first to obtain the coded signal of the current low sampling rate speech, and then the fundamental frequency of the current low sampling rate speech and the coded signal of the current low sampling rate speech are used to generate the harmonic signal of the extended speech corresponding to the current low sampling rate speech through a harmonic generator. At the same time, the coded signal of the current low sampling rate speech is used to generate the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech through a noise generator, and then the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech are superimposed by a superimposer, and finally the high-frequency signal of the extended speech missing corresponding to the current low sampling rate speech output by the source filter initial model is obtained.

[0108] The high-frequency signal missing from the extended voice corresponding to the current low-sampling rate voice outputted by the obtained source filter initial model has been band-extended compared to the current low-sampling rate voice. To determine the prediction accuracy of the source filter initial model, the high-frequency signal missing from the extended voice corresponding to the current low-sampling rate voice can be compared with the high-sampling rate voice in the current training data. According to the comparison result, it can be determined whether to continue training the source filter initial model or to end the training process. If it is determined that the source filter initial model is to continue training, the source filter initial model can be parameter-adjusted based on the comparison result. For example, the parameters of the encoder, harmonic generator, noise generator and other parts in the source filter initial model are adjusted. The specific adjustment rules can be pre-set with the goal of reducing the loss error of the source filter initial model.

[0109] The source filter initial model is trained using the current training data, and after the parameters are adjusted, the source filter initial model after the parameter adjustment can be continued to be trained using the high sampling rate speech and low sampling rate speech in the next training data. In this way, the loss error of the source filter initial model can be gradually reduced by continuously training the source filter initial model and continuously adjusting the parameters, so that the prediction accuracy of the source filter initial model is continuously improved. If the loss error of the source filter initial model is less than the preset first error threshold, the source filter initial model obtained by the current training can be determined as the source filter model and applied to the actual speech signal band extension scenario.

[0110] In one embodiment of the present application, for each piece of training data, in the process of training the source filter initial model using the high sampling rate speech and the low sampling rate speech in the current training data, the loss error of the source filter initial model can be determined by the following steps:

[0111] The first step: determining the high-frequency signal missing from the extended speech corresponding to the low-sampling rate speech in the current training data and the high-sampling rate speech in the current training data as inputs of the first loss function;

[0112] The second step: based on the output of the first loss function, determine the loss error of the source filter initial model.

[0113] For ease of description, the above two steps are combined for illustration.

[0114] For each piece of training data, in the process of training the source filter initial model using the current training data, the high-frequency signal missing from the extended speech corresponding to the current low sampling rate speech output by the source filter initial model can be obtained. The current training data is the training data targeted by the current operation, and the current low sampling rate speech is the low sampling rate speech in the current training data.

[0115] The first loss function of the source filter initial model can be pre-set according to actual conditions. The high-frequency signal missing from the extended speech corresponding to the current low sampling rate speech and the high sampling rate speech in the current training data are determined as the input of the first loss function. Through the calculation of the first loss function, the output of the first loss function can be obtained. Based on the output of the first loss function, the loss error of the source filter initial model can be determined.

[0116] The loss error of the source filter initial model is determined by using the first loss function, so as to improve the prediction accuracy of the source filter initial model.

[0117] In one embodiment of the present application, after obtaining the harmonic signal and unvoiced noise signal of the extended speech corresponding to the source speech, the source speech, the signal spectrum of the source speech, and the harmonic signal and unvoiced noise signal of the extended speech corresponding to the source speech can be input into a sequence generation model, and speech synthesis processing is performed by the sequence generation model to obtain the extended speech corresponding to the source speech output by the sequence generation model.

[0118] In the embodiment of the present application, the sequence generation model can be obtained by pre-training. Specifically, the sequence generation model can be obtained by pre-training using training data including high sampling rate speech and low sampling rate speech.

[0119] After the source speech is determined and the signal spectrum of the source speech and the harmonic signal and unvoiced noise signal of the extended speech corresponding to the source speech are obtained, the source speech, the signal spectrum of the source speech and the harmonic signal and unvoiced noise signal of the extended speech corresponding to the source speech can be further input into the sequence generation model, and the sequence generation model can output the extended speech corresponding to the source speech after performing corresponding speech synthesis processing. Of course, after obtaining the harmonic signal and unvoiced noise signal of the extended speech corresponding to the source speech, the high-frequency signal missing from the extended speech corresponding to the source speech can be predicted based on the harmonic signal and unvoiced noise signal of the extended speech corresponding to the source speech, and then the source speech, the signal spectrum of the source speech and the high-frequency signal missing from the extended speech corresponding to the source speech are input into the sequence generation model, and after the corresponding speech synthesis processing performed by the sequence generation model, the extended speech corresponding to the source speech output by the sequence generation model is obtained.

[0120] The harmonic signal, unvoiced noise signal and missing high-frequency signal of the extended speech corresponding to the source speech can be obtained through the source filter model. The source filter model is based on the combination of neural network and signal, and can obtain the high-frequency missing part of the signal after the source speech is band-expanded. The source filter model is combined with the sequence generation model to effectively expand the frequency band of the speech signal and improve the speech quality of the extended speech.

[0121] In one embodiment of the present application, the sequence generation model can be pre-trained by the following steps:

[0122] Step 1: obtaining a plurality of training data, each training data corresponds to a data pair, each data pair includes a high sampling rate speech and a low sampling rate speech, and the low sampling rate speech in each data pair is generated according to the corresponding high sampling rate speech;

[0123] Step 2: Use the high sampling rate speech and low sampling rate speech in each training data to train the pre-established sequence generation initial model in turn until the loss error of the sequence generation initial model is less than the preset second error threshold, and obtain the sequence generation model.

[0124] In the embodiment of the present application, an initial sequence generation model can be pre-established based on a neural network, and corresponding parameters can be initialized. The initial sequence generation model and the sequence generation model have the same structure.

[0125] The process of obtaining multiple training data may refer to the process of obtaining training data when training the source filter model. The same training data may be used for the training of the source filter model and the training of the sequence generation model.

[0126] After obtaining multiple pieces of training data, each piece of training data may be used in turn to train the initial model for sequence generation.

[0127] It should be noted that, in the embodiment of the present application, the training of the source filter initial model and the training of the sequence generation initial model can be performed separately or simultaneously. For example, the source filter initial model can be trained with each training data in sequence, and after the source filter model is obtained, the sequence generation initial model can be trained with each training data in sequence. It is also possible to train the sequence generation initial model with the current training data after the source filter initial model is trained with the current training data for each training data.

[0128] The initial model for sequence generation is trained in turn using the high sampling rate speech and the low sampling rate speech in each training data. After each training is completed, the loss error of the initial model for sequence generation can be determined. When the loss error of the initial model for sequence generation is less than the preset second error threshold, it can be considered that the initial model for sequence generation obtained by the current training has a high accuracy, and the initial model for sequence generation obtained by the current training can be determined as the sequence generation model and applied to the actual speech signal band expansion scenario. When the loss error of the initial model for sequence generation is greater than or equal to the second error threshold, it can be considered that the initial model for sequence generation obtained by the current training is not accurate and needs to be further trained.

[0129] The second error threshold can be set and adjusted according to actual conditions.

[0130] In one embodiment of the present application, training a pre-established sequence generation initial model using high sampling rate speech and low sampling rate speech in each piece of training data in turn may include the following steps:

[0131] Step 1: for each piece of training data, respectively obtain the signal spectrum of the low sampling rate speech in the current training data and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech;

[0132] Step 2: inputting the current low sampling rate speech, the signal spectrum of the current low sampling rate speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech into a pre-established sequence generation initial model, performing speech synthesis processing through the sequence generation initial model, and obtaining the extended speech corresponding to the current low sampling rate speech output by the sequence generation initial model;

[0133] Step 3: Based on the comparison results of the extended speech corresponding to the current low sampling rate speech and the high sampling rate speech in the current training data, the parameters of the initial model for sequence generation are adjusted to train the initial model for sequence generation after the parameters are adjusted using the high sampling rate speech and low sampling rate speech in the subsequent training data.

[0134] For the convenience of description, the above three steps are combined for explanation.

[0135] After obtaining multiple training data, for each training data, the low sampling rate speech in the current training data can be short-time Fourier transform processed to obtain the signal spectrum of the current low sampling rate speech, and then the harmonic signal and unvoiced noise signal of the current low sampling rate speech can be obtained based on the signal spectrum of the current low sampling rate speech. Specifically, the signal spectrum of the current low sampling rate speech can be input into the source filter initial model, and the harmonic signal and unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech can be obtained through the source filter initial model. The source filter initial model can be a model during the training process, or it can be a model after the training is completed. The current training data is the training data targeted by the current operation, and the current low sampling rate speech is the low sampling rate speech in the current training data.

[0136] After the above processing, the current low sampling rate speech, the signal spectrum of the current low sampling rate speech, the harmonic signal of the extended speech corresponding to the current low sampling rate speech and the unvoiced noise signal can be input into the sequence generation initial model. The sequence generation initial model can perform corresponding speech synthesis processing, and then the extended speech corresponding to the current low sampling rate speech output by the sequence generation initial model can be obtained.

[0137] The extended speech corresponding to the current low sampling rate speech output by the sequence generation initial model is the speech signal finally obtained after the frequency band expansion of the current low sampling rate speech. In order to judge the prediction accuracy of the source filter initial model and the sequence generation initial model, the extended speech corresponding to the current low sampling rate speech can be compared with the high sampling rate speech in the current training data. Figure 2 The discriminator shown in FIG. 1 is compared with the discriminator shown in FIG. According to the comparison result, it can be determined whether to continue training the initial model for sequence generation or to end the training process. If it is determined to continue training the initial model for sequence generation, the parameters of the initial model for sequence generation can be adjusted based on the comparison result. The specific adjustment rules can be pre-set with the goal of reducing the loss error of the initial model for sequence generation.

[0138] After the initial sequence generation model is trained using the current training data and the parameters are adjusted, the initial sequence generation model after the parameters are adjusted can be continuously trained using the high sampling rate speech and low sampling rate speech in the next training data. In this way, the loss error of the initial sequence generation model can be gradually reduced by continuously training the initial sequence generation model and continuously adjusting the parameters, so that the overall prediction accuracy of the source filter initial model and the initial sequence generation model can be continuously improved. If the loss error of the initial sequence generation model is less than the preset second error threshold, the initial sequence generation model obtained by the current training can be determined as the sequence generation model and applied to the actual speech signal band extension scenario.

[0139] In one embodiment of the present application, for each piece of training data, in the process of training the initial model for sequence generation using the high sampling rate speech and the low sampling rate speech in the current training data, the loss error of the initial model for sequence generation can be determined by the following steps:

[0140] The first step: determining the extended speech corresponding to the low sampling rate speech in the current training data and the high sampling rate speech in the current training data as inputs of the second loss function;

[0141] The second step is to determine the loss error of the initial model for sequence generation based on the output of the second loss function.

[0142] For ease of description, the above two steps are combined for illustration.

[0143] For each piece of training data, in the process of training the initial model for sequence generation using the current training data, the extended speech corresponding to the current low sampling rate speech output by the initial model for sequence generation can be obtained. The current training data is the training data for the current operation, and the current low sampling rate speech is the low sampling rate speech in the current training data.

[0144] The second loss function of the sequence generation initial model can be pre-set according to actual conditions. The extended speech corresponding to the current low sampling rate speech and the high sampling rate speech in the current training data are determined as inputs of the second loss function. The output of the second loss function can be obtained by calculating the second loss function. Based on the output of the second loss function, the loss error of the sequence generation initial model can be determined.

[0145] By using the second loss function to determine the loss error of the sequence generation initial model, the overall prediction accuracy of the source filter initial model and the sequence generation initial model can be improved.

[0146] In general, the technical solution provided by the embodiment of the present application may include two stages, one is the model training stage, and the other is the actual application stage. The model training stage includes two processes, one is the training process of the source filter model, and the other is the training process of the sequence generation model. After the model training is completed, it is applied to the actual speech signal band extension scenario.

[0147] For ease of understanding, Figure 2 Taking the overall structure of the speech signal band extension system shown in the figure as an example, the technical solution provided by the embodiment of the present application is described again. The speech signal band extension system includes a source filter model and a sequence generation model. The source filter model is obtained by training the source filter initial model using training data, and the sequence generation model is obtained by training the sequence generation initial model using training data. The source filter model has the same structure as the source filter initial model, and the sequence generation model has the same structure as the sequence generation initial model, but the names are different at different stages.

[0148] In the process of training the source filter model, for each training data, the low sampling rate speech in the current training data can be subjected to short-time Fourier transform (STFT) processing to obtain the signal spectrum of the current low sampling rate speech, and then the signal spectrum of the current low sampling rate speech is input into the source filter initial model, and the encoder, harmonic generator, noise generator, superimposer and other parts of the source filter initial model are subjected to corresponding processing to obtain the high-frequency signal missing from the extended speech corresponding to the current low sampling rate speech output by the source filter initial model, and based on the comparison result of the high-frequency signal missing from the extended speech corresponding to the current low sampling rate speech and the high sampling rate speech in the current training data, the source filter initial model is parameter adjusted to continue training the source filter initial model after the parameter adjustment using the next training data, until the loss error of the source filter initial model is determined to be less than the first error threshold value using the first loss function, and the source filter initial model obtained by the current training is determined as the source filter model.

[0149] In the process of training the sequence generation model, for each training data, the signal spectrum of the low sampling rate speech in the current training data and the high-frequency signal missing from the extended speech corresponding to the current low sampling rate speech output by the source filter initial model can be obtained, and the current low sampling rate speech, the signal spectrum of the current low sampling rate speech, and the high-frequency signal missing from the extended speech corresponding to the current low sampling rate speech are input into the sequence generation initial model (WaveNet) for speech synthesis processing, so as to obtain the extended speech corresponding to the current low sampling rate speech output by the sequence generation initial model. The extended speech corresponding to the current low sampling rate speech and the high sampling rate speech in the current training data are compared by a discriminator, and whether to continue training can be determined based on the comparison result. If training is continued, the parameters of the sequence generation initial model can be adjusted, so as to continue to train the sequence generation initial model after the parameter adjustment using the next training data, until the loss error of the sequence generation initial model is determined to be less than the second error threshold by using the second loss function, and the sequence generation initial model obtained by the current training is determined as the sequence generation model.

[0150] In the actual application stage, the source speech is processed by short-time Fourier transform (STFT) to obtain the signal spectrum of the source speech, and then the signal spectrum of the source speech is input into the source filter model. After the source filter model performs harmonic generation, noise generation, superposition and other processing, the high-frequency signal missing from the extended speech corresponding to the source speech output by the source filter model is obtained. The source speech, the signal spectrum of the source speech and the high-frequency signal missing from the extended speech corresponding to the source speech are input into the sequence generation model (WaveNet). After the speech synthesis processing of the sequence generation model, the extended speech corresponding to the source speech output by the sequence generation model is obtained. The frequency band expansion of the source speech is achieved.

[0151] The embodiment of the present application combines the source filter model with the sequence generation model, which has strong stability and versatility for the frequency band expansion of speech signals. Moreover, the source filter technology based on speech signals decomposes the signals, reducing the difficulty of the overall model in predicting the expanded speech.

[0152] In addition, in the embodiment of the present application, the low sampling rate speech in each training data is generated based on the corresponding high sampling rate speech. During the training process, the high-frequency signal, the extended speech and the corresponding real high sampling rate speech are compared. Better results can be achieved using less training data, thereby improving training efficiency.

[0153] Corresponding to the above method embodiment, the embodiment of the present application further provides a speech signal frequency band extension device. The speech signal frequency band extension device described below and the speech signal frequency band extension method described above can refer to each other.

[0154] See also Figure 3As shown, the device may include the following modules:

[0155] A source speech determination module 310, used to determine the source speech;

[0156] The spectrum acquisition module 320 is used to acquire the signal spectrum of the source speech;

[0157] A signal acquisition module 330, configured to obtain a harmonic signal and an unvoiced noise signal of an extended speech corresponding to the source speech based on a signal spectrum of the source speech;

[0158] The extended speech acquisition module 340 is used to perform speech synthesis processing on the source speech, the signal spectrum of the source speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech to obtain the extended speech corresponding to the source speech.

[0159] By using the device provided in the embodiment of the present application, the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are first obtained based on the signal spectrum of the source speech, which conforms to the actual signal composition, and then the source speech, the signal spectrum of the source speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are subjected to speech synthesis processing. The obtained extended speech corresponding to the source speech is more comprehensive, and therefore, is conducive to improving the speech quality of the speech signal obtained after frequency band expansion, thereby enhancing the user's listening experience.

[0160] In a specific implementation of the present application, a high-frequency signal prediction module is also included, which is used to:

[0161] Based on the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech, predict the high-frequency signal missing from the extended speech corresponding to the source speech;

[0162] Accordingly, the extended speech acquisition module 340 is used to:

[0163] Speech synthesis processing is performed on the source speech, the signal spectrum of the source speech, and the high-frequency signal missing from the extended speech corresponding to the source speech to obtain the extended speech corresponding to the source speech.

[0164] In a specific implementation of the present application, the signal acquisition module 330 is used to:

[0165] Encoding the signal spectrum of the source speech to obtain a coded signal of the source speech;

[0166] Generate a harmonic signal of an extended speech corresponding to the source speech by using the fundamental frequency of the source speech and the coded signal of the source speech;

[0167] The coded signal of the source speech is used to generate an unvoiced noise signal of the extended speech corresponding to the source speech.

[0168] In a specific implementation of the present application, the signal acquisition module 330 and the high-frequency signal prediction module are used to:

[0169] The signal spectrum of the source speech is input into the source filter model. After prediction by the source filter model, the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are obtained. The harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are superimposed to obtain the high-frequency signal missing from the extended speech corresponding to the source speech output by the source filter model.

[0170] In a specific implementation of the present application, a source filter model training module is also included, which is used to pre-train and obtain a source filter model through the following steps:

[0171] Obtain multiple training data, each training data corresponds to a data pair, each data pair includes a high sampling rate speech and a low sampling rate speech, and the low sampling rate speech in each data pair is generated according to the corresponding high sampling rate speech;

[0172] The pre-established source filter initial model is trained in turn using the high sampling rate speech and the low sampling rate speech in each training data until the loss error of the source filter initial model is less than a preset first error threshold, thereby obtaining a source filter model.

[0173] In a specific implementation of the present application, the source filter model training module is used to:

[0174] For each piece of training data, obtain the signal spectrum of the low sampling rate speech in the current training data;

[0175] Inputting the signal spectrum of the current low sampling rate speech into a pre-established source filter initial model, obtaining the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech through prediction by the source filter initial model, and superimposing the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech to obtain the high-frequency signal missing from the extended speech corresponding to the current low sampling rate speech output by the source filter initial model;

[0176] Based on the comparison results of the high-frequency signal missing from the extended speech corresponding to the current low-sampling rate speech and the high-sampling rate speech in the current training data, the parameters of the source filter initial model are adjusted to train the parameter-adjusted source filter initial model using the high-sampling rate speech and low-sampling rate speech in the subsequent training data.

[0177] In a specific implementation of the present application, the extended speech acquisition module 340 is used to:

[0178] The source speech, the signal spectrum of the source speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are input into the sequence generation model, and the sequence generation model is used to perform speech synthesis processing to obtain the extended speech corresponding to the source speech output by the sequence generation model.

[0179] In a specific embodiment of the present application, a sequence generation model training module is also included, which is used to pre-train and obtain a sequence generation model through the following steps:

[0180] Obtain multiple training data, each training data corresponds to a data pair, each data pair includes a high sampling rate speech and a low sampling rate speech, and the low sampling rate speech in each data pair is generated according to the corresponding high sampling rate speech;

[0181] The pre-established initial sequence generation model is trained in turn using the high sampling rate speech and the low sampling rate speech in each training data until the loss error of the initial sequence generation model is less than the preset second error threshold, thereby obtaining the sequence generation model.

[0182] In a specific embodiment of the present application, the sequence generation model training module is used to:

[0183] For each piece of training data, respectively obtain the signal spectrum of the low sampling rate speech in the current training data and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech;

[0184] Inputting the current low sampling rate speech, the signal spectrum of the current low sampling rate speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech into a pre-established sequence generation initial model, performing speech synthesis processing through the sequence generation initial model, and obtaining the extended speech corresponding to the current low sampling rate speech output by the sequence generation initial model;

[0185] Based on the comparison results of the extended speech corresponding to the current low sampling rate speech and the high sampling rate speech in the current training data, the parameters of the initial model for sequence generation are adjusted, so as to train the initial model for sequence generation after the parameters are adjusted using the high sampling rate speech and the low sampling rate speech in the subsequent training data.

[0186] In a specific implementation of the present application, the source filter model training module or the sequence generation model training module is used to:

[0187] Collect multiple high sampling rate voices;

[0188] Extract low sampling rate speech from each high sampling rate speech respectively;

[0189] The data pairs consisting of each high sampling rate speech and the corresponding low sampling rate speech are determined as training data one by one.

[0190] In a specific implementation of the present application, the source filter model training module or the sequence generation model training module is further used to:

[0191] After a plurality of high sampling rate speech is collected and before a low sampling rate speech is extracted from each high sampling rate speech, a cleaning process is performed on each high sampling rate speech collected.

[0192] Corresponding to the above method embodiment, the embodiment of the present application further provides a voice signal frequency band extension device, including:

[0193] Memory for storing computer programs;

[0194] A processor is used to implement the steps of the above-mentioned method for extending the frequency band of a speech signal when executing a computer program.

[0195] like Figure 4 , which is a schematic diagram of the structure of a voice signal frequency band extension device, the voice signal frequency band extension device may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11, and the communication interface 12 communicate with each other through the communication bus 13.

[0196] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0197] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in the embodiment of the method for extending the frequency band of a speech signal.

[0198] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0199] Determine the source speech and obtain the signal spectrum of the source speech;

[0200] Based on the signal spectrum of the source speech, a harmonic signal and an unvoiced noise signal of the extended speech corresponding to the source speech are obtained;

[0201] The source speech, the signal spectrum of the source speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are subjected to speech synthesis processing to obtain the extended speech corresponding to the source speech.

[0202] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function (such as a spectrum acquisition function, a harmonic generation function), etc.; the data storage area may store data created during use, such as harmonic signal data, clear noise signal data, etc.

[0203] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0204] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0205] Of course, it should be noted that Figure 4 The structure shown does not constitute a limitation on the voice signal frequency band extension device in the embodiment of the present application. In actual applications, the voice signal frequency band extension device may include Figure 4 More or fewer components than shown, or combinations of certain components.

[0206] Corresponding to the above method embodiment, the embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned voice signal frequency band expansion method are implemented.

[0207] In addition, it should be noted that: the embodiment of the present application also provides a computer program product or a computer program, which may include a computer instruction, and the computer instruction may be stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor may execute the computer instruction so that the computer device executes the description of the voice signal frequency band extension method in the corresponding embodiment of the foregoing text, and therefore, it will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiment involved in the present application, please refer to the description of the method embodiment of the present application.

[0208] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0209] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0210] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0211] Specific examples are used herein to illustrate the principles and implementation methods of the present application, and the description of the above embodiments is only used to help understand the technical solution and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for extending the frequency band of a speech signal, characterized in that: include: Determine a source speech, and obtain a signal spectrum of the source speech; Based on the signal spectrum of the source speech, obtaining a harmonic signal and an unvoiced noise signal of the extended speech corresponding to the source speech; Speech synthesis processing is performed on the source speech, the signal spectrum of the source speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech to obtain the extended speech corresponding to the source speech.

2. The method for extending the frequency band of a speech signal according to claim 1, characterized in that: Also includes: Based on the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech, predict the high-frequency signal missing from the extended speech corresponding to the source speech; The performing speech synthesis processing on the source speech, the signal spectrum of the source speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech to obtain the extended speech corresponding to the source speech includes: Speech synthesis processing is performed on the source speech, the signal spectrum of the source speech, and the high-frequency signal missing from the extended speech corresponding to the source speech to obtain the extended speech corresponding to the source speech.

3. The method for extending the frequency band of a speech signal according to claim 1, characterized in that: The step of obtaining a harmonic signal and an unvoiced noise signal of an extended speech corresponding to the source speech based on the signal spectrum of the source speech comprises: Performing encoding processing on the signal spectrum of the source speech to obtain an encoded signal of the source speech; Generate a harmonic signal of an extended speech corresponding to the source speech by using the fundamental frequency of the source speech and the coded signal of the source speech; The encoded signal of the source speech is used to generate an unvoiced noise signal of the extended speech corresponding to the source speech.

4. The method for extending the frequency band of a speech signal according to claim 2, characterized in that: The method of obtaining a harmonic signal and an unvoiced noise signal of an extended speech corresponding to the source speech based on the signal spectrum of the source speech, and predicting a high-frequency signal missing from the extended speech corresponding to the source speech based on the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech, comprises: The signal spectrum of the source speech is input into the source filter model, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are obtained through prediction by the source filter model. The harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are superimposed to obtain the high-frequency signal missing from the extended speech corresponding to the source speech output by the source filter model.

5. The method for extending the frequency band of a speech signal according to claim 4, characterized in that: The source filter model is obtained by pre-training through the following steps: Obtain multiple training data, each training data corresponds to a data pair, each data pair includes a high sampling rate speech and a low sampling rate speech, and the low sampling rate speech in each data pair is generated according to the corresponding high sampling rate speech; The pre-established source filter initial model is trained in turn using the high sampling rate speech and the low sampling rate speech in each training data until the loss error of the source filter initial model is less than a preset first error threshold, thereby obtaining the source filter model.

6. The method for extending the frequency band of a speech signal according to claim 5, characterized in that: The method of sequentially training the pre-established source filter initial model by using the high sampling rate speech and the low sampling rate speech in each training data comprises: For each piece of training data, obtain the signal spectrum of the low sampling rate speech in the current training data; Inputting the signal spectrum of the current low sampling rate speech into a pre-established source filter initial model, obtaining the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech through prediction by the source filter initial model, and superimposing the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech to obtain the high-frequency signal missing from the extended speech corresponding to the current low sampling rate speech output by the source filter initial model; Based on the comparison result of the high-frequency signal missing from the extended speech corresponding to the current low-sampling rate speech and the high-sampling rate speech in the current training data, the parameters of the source filter initial model are adjusted to train the source filter initial model after the parameter adjustment using the high-sampling rate speech and the low-sampling rate speech in the subsequent training data.

7. The method for extending the frequency band of a speech signal according to claim 1, characterized in that: The performing speech synthesis processing on the source speech, the signal spectrum of the source speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech to obtain the extended speech corresponding to the source speech includes: The source speech, the signal spectrum of the source speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the source speech are input into a sequence generation model, and speech synthesis processing is performed by the sequence generation model to obtain the extended speech corresponding to the source speech output by the sequence generation model.

8. The method for extending the frequency band of a speech signal according to claim 7, characterized in that: The sequence generation model is obtained by pre-training through the following steps: Obtain multiple training data, each training data corresponds to a data pair, each data pair includes a high sampling rate speech and a low sampling rate speech, and the low sampling rate speech in each data pair is generated according to the corresponding high sampling rate speech; The pre-established sequence generation initial model is trained in turn using the high sampling rate speech and the low sampling rate speech in each training data until the loss error of the sequence generation initial model is less than a preset second error threshold, thereby obtaining the sequence generation model.

9. The method for extending the frequency band of a speech signal according to claim 8, characterized in that: The method of sequentially using the high sampling rate speech and the low sampling rate speech in each piece of training data to train the pre-established sequence generation initial model includes: For each piece of training data, respectively obtain the signal spectrum of the low sampling rate speech in the current training data and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech; Inputting the current low sampling rate speech, the signal spectrum of the current low sampling rate speech, and the harmonic signal and the unvoiced noise signal of the extended speech corresponding to the current low sampling rate speech into a pre-established sequence generation initial model, performing speech synthesis processing through the sequence generation initial model, and obtaining the extended speech corresponding to the current low sampling rate speech output by the sequence generation initial model; Based on the comparison result of the extended speech corresponding to the current low sampling rate speech and the high sampling rate speech in the current training data, the parameters of the initial model for sequence generation are adjusted to train the initial model for sequence generation after the parameter adjustment using the high sampling rate speech and the low sampling rate speech in the latter training data.

10. The method for extending the frequency band of a speech signal according to claim 5 or 8, characterized in that: The obtaining of multiple pieces of training data includes: Collect multiple high sampling rate voices; Extract low sampling rate speech from each high sampling rate speech respectively; The data pairs consisting of each high sampling rate speech and the corresponding low sampling rate speech are determined as training data one by one.

11. The method for extending the frequency band of a speech signal according to claim 10, characterized in that: After the multiple high sampling rate voices are collected and before the low sampling rate voices are extracted from each high sampling rate voice, the method further includes: Each high sampling rate speech collected is cleaned.

12. A speech signal frequency band extension device, characterized in that: include: A source speech determination module, used to determine the source speech; A spectrum acquisition module, used to acquire the signal spectrum of the source speech; A signal acquisition module, used for obtaining a harmonic signal and an unvoiced noise signal of an extended speech corresponding to the source speech based on a signal spectrum of the source speech; The extended speech acquisition module is used to perform speech synthesis processing on the source speech, the signal spectrum of the source speech, and the harmonic signal and unvoiced noise signal of the extended speech corresponding to the source speech to obtain the extended speech corresponding to the source speech.

13. A voice signal frequency band extension device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method for extending the frequency band of a speech signal as claimed in any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for extending the frequency band of a speech signal according to any one of claims 1 to 11 are implemented.

15. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium and are suitable for being read and executed by a processor, so that a computer device having the processor executes the speech signal frequency band extension method according to any one of claims 1 to 11.

Images