A long frame shift speech phase spectrum prediction method and device

Abstract

The application provides a long frame shift speech phase spectrum prediction method and device, and relates to the technical field of speech signal processing.The method comprises the following steps: converting a long frame shift speech logarithmic amplitude spectrum into a short frame shift speech logarithmic amplitude spectrum based on a lossless interpolation technology; using the short frame shift speech logarithmic amplitude spectrum to perform phase spectrum prediction, so as to obtain a short frame shift speech phase spectrum; and generating a long frame shift speech phase spectrum according to the short frame shift speech phase spectrum.The application can improve the prediction result accuracy of the long frame shift speech phase spectrum.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech signal processing, and particularly relates to a long-frame-shift speech phase spectrum prediction method and device. BACKGROUND

[0002] Speech phase prediction, also known as speech phase reconstruction, aims to recover the corresponding phase spectrum according to the amplitude spectrum or related features of the speech. In the early stage, the Griffin-Lim algorithm was used for speech phase prediction. The Griffin-Lim algorithm is a famous iterative speech phase spectrum estimation algorithm that iteratively estimates the phase spectrum from the amplitude spectrum through short-time Fourier transform (STFT) and inverse short-time Fourier transform (ISTFT). With the development of deep learning and neural networks, some phase prediction methods combined with neural networks have been gradually proposed. For example, a neural network speech phase spectrum prediction method based on a parallel estimation architecture and an anti-winding loss, which realizes direct prediction of the speech winding phase spectrum from the speech log amplitude spectrum through a neural network model.

[0003] However, whether it is a traditional iterative speech phase spectrum estimation algorithm such as the Griffin-Lim algorithm or a neural network speech phase spectrum prediction method based on a parallel estimation architecture and an anti-winding loss, the performance thereof is very sensitive to the length of the short-time Fourier transform frame shift of the speech phase spectrum. Due to the winding characteristics of the phase, the value range of the winding phase spectrum curve changes over time is limited within the principal value interval, and there will be a time-domain discontinuity phenomenon at the boundary of the principal value interval. For the same speech, when a longer frame shift of the short-time Fourier transform is performed, the number of generated phase spectrum frames is smaller, and therefore this time-domain discontinuity phenomenon is more obvious on the long-frame-shift phase spectrum. The inter-frame phase correlation of the long-frame-shift phase spectrum is smaller, the frame independence is stronger, and the time-domain continuity is poorer, which makes it more difficult to recover and predict. When the frame shift is shorter, the existing speech phase spectrum prediction method can accurately recover the phase spectrum, and even some methods can achieve an effect close to natural speech; when the frame shift is longer, the waveform reconstructed by the traditional iterative speech phase spectrum estimation algorithm almost completely loses the spectral details, the speech waveform reconstructed by the neural network speech phase spectrum prediction method based on a parallel estimation architecture and an anti-winding loss also has a significant decrease in quality, obvious noise appears, and it is difficult to accurately model the time-domain discontinuity of the phase spectrum.

[0004] Therefore, there is a technical problem in the prior art that it is difficult to improve the prediction result accuracy of the long-frame-shift speech phase spectrum. SUMMARY

[0005] In order to solve the technical problem that it is difficult to improve the prediction result accuracy of long frame shift speech phase spectrum in the prior art, the present application provides a long frame shift speech phase spectrum prediction method and device.

[0006] The technical scheme of the present application is as follows:

[0007] The present application provides a long frame shift speech phase spectrum prediction method, comprising:

[0008] Converting the long frame shift speech log amplitude spectrum into a short frame shift speech log amplitude spectrum based on lossless interpolation technology;

[0009] Using the short frame shift speech log amplitude spectrum to perform phase spectrum prediction to obtain a short frame shift speech phase spectrum;

[0010] Generating a long frame shift speech phase spectrum according to the short frame shift speech phase spectrum.

[0011] Optionally, converting the long frame shift speech log amplitude spectrum into a short frame shift speech log amplitude spectrum based on lossless interpolation technology specifically comprises:

[0012] Dividing the long frame shift speech log amplitude spectrum into a plurality of first frame sequences;

[0013] For each first frame sequence, performing lossless interpolation on the first frame sequence to obtain a second frame sequence;

[0014] Splicing each second frame sequence along the frequency axis to obtain the short frame shift speech log amplitude spectrum.

[0015] Optionally, performing lossless interpolation on the first frame sequence to obtain a second frame sequence specifically comprises:

[0016] Inserting a preset number of sample points after each sample point in the first frame sequence to obtain a corresponding second frame sequence.

[0017] Optionally, performing lossless interpolation on the first frame sequence to obtain a second frame sequence specifically comprises:

[0018] Using a lossless interpolation module based on signal processing or a lossless interpolation module based on neural network to perform lossless interpolation on the first frame sequence to obtain a corresponding second frame sequence.

[0019] Optionally, the lossless interpolation module based on signal processing comprises a zero-padder and a low-pass filter;

[0020] The lossless interpolation module based on signal processing is specifically used for:

[0021] The zero inserter is used to insert a preset number of zero points after each sample point in the first frame sequence to obtain a zero-inserter sequence;

[0022] The low-pass filter is used to replace each zero point in the zero-inserter sequence with an interpolation point to obtain a corresponding second frame sequence.

[0023] Optionally, the neural network-based lossless interpolation module comprises a deconvolution layer with a step size D and a replacement module.

[0024] The neural network-based lossless interpolation module is specifically used for:

[0025] The deconvolution layer is used to convert the first frame sequence into an intermediate sequence with a length of D*F, wherein F is the length of the first frame sequence.

[0026] The replacement module is used to replace each sample point in the intermediate sequence with a sample point corresponding to the sample point in the first frame sequence to obtain a corresponding second frame sequence.

[0027] Optionally, the long-frame-shift speech phase spectrum is generated according to the short-frame-shift speech phase spectrum, and specifically comprises:

[0028] Each non-interpolation frame in the short-frame-shift speech phase spectrum is extracted.

[0029] The extracted non-interpolation frames are spliced in the frequency axis to obtain the long-frame-shift speech phase spectrum.

[0030] Optionally, after the long-frame-shift speech phase spectrum is generated according to the short-frame-shift speech phase spectrum, the method of the present application further comprises:

[0031] The long-frame-shift speech phase spectrum and the long-frame-shift speech log amplitude spectrum are combined to obtain a short-time complex spectrum.

[0032] The short-time complex spectrum is subjected to inverse short-time Fourier transform to obtain a speech waveform.

[0033] Optionally, the short-frame-shift speech phase spectrum is generated by using the short-frame-shift speech log amplitude spectrum for phase spectrum prediction, and specifically comprises:

[0034] The short-frame-shift speech log amplitude spectrum is input into a preset phase prediction neural network to obtain a short-frame-shift speech phase spectrum output by the preset phase prediction neural network.

[0035] The preset phase prediction neural network comprises a residual convolutional network, parallel first and second linear convolutional layers, and a phase calculation unit, and a loss function of the preset phase prediction neural network comprises a linear combination of an instantaneous phase loss, a group delay loss, and an instantaneous angular frequency loss.

[0036] The application further provides a long-frame-shift speech phase spectrum prediction device, comprising:

[0037] An interpolation module is configured to convert a long-frame-shift speech log amplitude spectrum into a short-frame-shift speech log amplitude spectrum based on a lossless interpolation technology.

[0038] A prediction module is configured to perform phase spectrum prediction using the short-frame-shift speech log amplitude spectrum to obtain a short-frame-shift speech phase spectrum.

[0039] An extraction module is configured to generate a long-frame-shift speech phase spectrum according to the short-frame-shift speech phase spectrum.

[0040] The application has the following beneficial effects:

[0041] A long-frame-shift speech phase spectrum prediction method comprises the following steps: converting a long-frame-shift speech log amplitude spectrum into a short-frame-shift speech log amplitude spectrum based on a lossless interpolation technology; performing phase spectrum prediction using the short-frame-shift speech log amplitude spectrum to obtain a short-frame-shift speech phase spectrum; and generating a long-frame-shift speech phase spectrum according to the short-frame-shift speech phase spectrum. Based on this, since the long-frame-shift speech log amplitude spectrum is converted into the short-frame-shift speech log amplitude spectrum based on the lossless interpolation technology, the phase spectrum prediction is performed using the short-frame-shift speech log amplitude spectrum, and the phase spectrum prediction is not directly performed on the long-frame-shift speech log amplitude spectrum, so that the prediction result accuracy of the long-frame-shift speech phase spectrum can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0043] Figure 1 is a schematic diagram of the overall architecture of a long-frame-shift speech phase spectrum prediction method provided by the embodiments of the present application;

[0044] Figure 2 is a flowchart of a long-frame-shift speech phase spectrum prediction method provided by the embodiments of the present application;

[0045] Figure 3is a structure schematic diagram of a lossless interpolation module based on signal processing provided by an embodiment of the present application.

[0046] Figure 4 is a structure schematic diagram of a lossless interpolation module based on neural network provided by an embodiment of the present application.

[0047] Figure 5 is a structure schematic diagram of a long frame shift speech phase spectrum prediction device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0049] In order to facilitate understanding of the technical solutions of the present application, first, some terms possibly involved in the present application are described.

[0050] Phase wrapping: assuming that the principal value interval of the phase is (-pi, pi], the phase will jump at the boundaries -pi and pi, showing a discontinuous phenomenon, which is called phase wrapping of the speech signal.

[0051] Amplitude spectrum and phase spectrum: after the speech signal is subjected to short-time Fourier transform, a short-time complex spectrum can be obtained, then the short-time complex spectrum is calculated according to the amplitude calculation formula, and the amplitude spectrum of the speech signal can be obtained, the amplitude spectrum reflects the amplitudes of different frequency sinusoidal signals constituting the speech signal, and the phase spectrum of the speech signal predicted according to the amplitude spectrum reflects the phases of different frequency sinusoidal signals in the speech signal.

[0052] Logarithmic amplitude spectrum: after the amplitude spectrum is taken as a natural logarithm, the logarithmic amplitude spectrum of the signal can be obtained.

[0053] The technical solutions of the present application will be described in detail below with reference to the drawings.

[0054] Figure 1 is a whole architecture schematic diagram of a long frame shift speech phase spectrum prediction method provided by an embodiment of the present application. As shown in Figure 1 The long frame shift speech phase spectrum prediction method is divided into three stages of interpolation, prediction and extraction.

[0055] In the interpolation stage, first, the input long-frame-shift speech logarithmic amplitude spectrum is frequency-division processed to obtain a plurality of first frame sequences. Then, for each first frame sequence, lossless interpolation is performed on the first frame sequence to obtain a second frame sequence; the lossless interpolation on the first frame sequence can expand the number of frames of the first frame sequence. Finally, the second frame sequences are spliced to obtain a short-frame-shift speech logarithmic amplitude spectrum. The lossless interpolated short-frame-shift speech logarithmic amplitude spectrum completely preserves the original information of the long-frame-shift speech logarithmic amplitude spectrum, and no information loss occurs.

[0056] In the prediction stage, the short-frame-shift speech logarithmic amplitude spectrum is used for phase spectrum prediction to obtain a short-frame-shift speech phase spectrum, so that the application can avoid directly performing phase spectrum prediction on the long-frame-shift speech logarithmic amplitude spectrum, and thus the application can improve the prediction result accuracy of the long-frame-shift speech phase spectrum.

[0057] In the extraction stage, the non-interpolation frames in the short-frame-shift speech phase spectrum are extracted by the frame skipping extraction method and spliced along the frequency axis to obtain a long-frame-shift speech phase spectrum.

[0058] Figure 2 is a flowchart of a long-frame-shift speech phase spectrum prediction method provided by an embodiment of the application. The long-frame-shift speech phase spectrum prediction method of the embodiment of the application will be specifically described below with reference to Figure 1 and Figure 2 .

[0059] With reference to Figure 1 and Figure 2 , the flowchart includes:

[0060] Step 201: converting a long-frame-shift speech logarithmic amplitude spectrum into a short-frame-shift speech logarithmic amplitude spectrum based on a lossless interpolation technology.

[0061] Specifically, step 201: converting a long-frame-shift speech logarithmic amplitude spectrum into a short-frame-shift speech logarithmic amplitude spectrum based on a lossless interpolation technology, can specifically include:

[0062] 1. Frequency-division processing the long-frame-shift speech logarithmic amplitude spectrum to obtain a plurality of first frame sequences.

[0063] 2. For each first frame sequence, performing lossless interpolation on the first frame sequence to obtain a second frame sequence.

[0064] 3. Splicing the second frame sequences along the frequency axis to obtain a short-frame-shift speech logarithmic amplitude spectrum.

[0065] Optionally, the lossless interpolation on the first frame sequence to obtain a second frame sequence can specifically include:

[0066] Inserting a preset number of sample points after each sample point in the first frame sequence to obtain a corresponding second frame sequence.

[0067] In one specific example, the long frame shift speech log amplitude spectrum with frame shift T / F is obtained After that, the long frame shift speech log amplitude spectrum logA L is divided into frequency to obtain N first frame sequences That is:

[0068]

[0069] Wherein, represents the real number set, T represents the number of speech time domain waveform points, F and N represent the frame number and frequency point number of the spectrum respectively.

[0070] Next, each first frame sequence passes through the same lossless interpolation module respectively, so as to insert D-1 sample points after each sample point in each first frame sequence to obtain each second frame sequence , the length of which is D times of the first frame sequence .

[0071] The second frame sequences are spliced according to the frequency axis to obtain the short frame shift speech log amplitude spectrum logA S with frame shift T / (DF). That is:

[0072]

[0073] Step 202: using the short frame shift speech log amplitude spectrum to perform phase spectrum prediction to obtain the short frame shift speech phase spectrum.

[0074] Specifically, step 202: using the short frame shift speech log amplitude spectrum to perform phase spectrum prediction to obtain the short frame shift speech phase spectrum, which can specifically include:

[0075] inputting the short frame shift speech log amplitude spectrum into a preset phase prediction neural network to obtain the short frame shift speech phase spectrum output by the preset phase prediction neural network.

[0076] Wherein, the preset phase prediction neural network includes a residual convolutional network, a parallel first linear convolutional layer and a second linear convolutional layer, and a phase calculation unit.

[0077] Taking the above example, the short frame shift speech log amplitude spectrum logA S passes through the residual convolutional network, and then passes through the first linear convolutional layer and the second linear convolutional layer respectively to obtain the pseudo real part output by the first linear convolutional layer and the pseudo imaginary part output by the second linear convolutional layer. Then, the phase calculation unit (binary activation function) uses the pseudo real part and pseudo imaginary part calculating short frame shift speech phase spectrum

[0078]

[0079] wherein Φ() represents a function for performing phase calculation, and an expression of the function is as follows:

[0080]

[0081] wherein Φ(0,0)=0. When x≥0, Sgn * (x)=1; when x<0, Sgn * (x)=-1.

[0082] the pseudo real part the pseudo imaginary part and the calculated short frame shift speech phase spectrum are all matrices of DF rows and N columns, therefore, in formula (4), the two input matrices of pseudo real part and pseudo imaginary part are calculated element by element, to obtain an element at a corresponding position in the short frame shift speech phase spectrum, for example, an element at the first row and the first column of the pseudo real part and an element at the first row and the first column of the pseudo imaginary part are substituted into formula (4), and a result of the calculation is taken as an element at the first row and the first column in the short frame shift speech phase spectrum.

[0083] In the embodiments of the present specification, the first linear convolution layer, the second linear convolution layer and the phase calculation unit constitute a parallel estimation architecture, which is one of the keys to realize direct and accurate prediction of speech phase spectrum. The architecture simulates the calculation process of the real part and the imaginary part of the short-time complex spectrum of the speech signal to the phase spectrum, strictly limits the phase value predicted by the model in the principal value interval (-π, π], and thus realizes the prediction of the wrapped phase spectrum.

[0084] Step 203: generating a long frame shift speech phase spectrum according to the short frame shift speech phase spectrum.

[0085] Specifically, step 203: generating a long frame shift speech phase spectrum according to the short frame shift speech phase spectrum, can specifically include:

[0086] 1. extracting non-interpolation frames in the short frame shift speech phase spectrum.

[0087] 2. splicing the extracted non-interpolation frames according to the frequency axis to obtain the long frame shift speech phase spectrum.

[0088] In a specific example, the predicted short frame shift speech phase spectrum The long frame shift speech phase spectrum is obtained by extracting and splicing the frames at the corresponding positions of the long frame shift in a frame extraction manner (extracting the 1st frame, the 1+D frame,..., the 1+(F-1)D frame)

[0089]

[0090] wherein, that is, the long frame shift logarithmic amplitude spectrum logA L corresponding to the phase spectrum.

[0091] The technical scheme is adopted in the embodiment of the application, a long frame shift speech phase spectrum prediction method comprises: converting a long frame shift speech logarithmic amplitude spectrum into a short frame shift speech logarithmic amplitude spectrum based on a lossless interpolation technology; performing phase spectrum prediction on the short frame shift speech logarithmic amplitude spectrum to obtain a short frame shift speech phase spectrum; and generating a long frame shift speech phase spectrum according to the short frame shift speech phase spectrum. Based on this, after the long frame shift speech logarithmic amplitude spectrum is converted into the short frame shift speech logarithmic amplitude spectrum based on the lossless interpolation technology, the phase spectrum prediction is performed on the short frame shift speech logarithmic amplitude spectrum, and the phase spectrum prediction is avoided on the long frame shift speech logarithmic amplitude spectrum directly, so that the embodiment of the application can improve the prediction result accuracy of the long frame shift speech phase spectrum.

[0092] In the embodiment of the application, the lossless interpolation module can be a lossless interpolation module based on signal processing or a lossless interpolation module based on a neural network.

[0093] Figure 3 is a structure diagram of a lossless interpolation module based on signal processing provided by the embodiment of the application. As Figure 3 shown, the lossless interpolation module based on signal processing comprises a zero-padder 31 and a low-pass filter 32.

[0094] The lossless interpolation module based on signal processing is specifically used for:

[0095] For any first frame sequence, the zero-padder 31 is used to insert a preset number of zero points after each sample point in the first frame sequence to obtain a zero-padded sequence; then, the low-pass filter 32 is used to replace each zero point in the zero-padded sequence with an interpolation point to obtain a corresponding second frame sequence.

[0096] In a specific example, for any first frame sequence it first inserts D-1 zero points after each sample point thereof by the zero-padder 31 to obtain a zero-padded sequence with a length of DF that is,

[0097]

[0098] Where f represents the sample point index, set There are DF points, for example, f = 1, then that is The value of the first point. Formula (6) means: in a sequence of length F After each sample point, add D-1 zeros to construct a sequence of length DF.

[0099] Next, the zero-padding sequence Through frequency response H D (e jω Low-pass filter 32:

[0100]

[0101] The zero-value points are replaced with interpolation points, but the original interpolation points remain unchanged, generating the final interpolated second frame sequence. This process can be implemented in the frequency domain, that is:

[0102]

[0103] Where DTFT and IDTFT represent the discrete-time Fourier transform and inverse discrete-time Fourier transform, respectively. ⊙ represents the multiplication of continuous functions.

[0104] Secondly, this process can also be implemented in the time domain.

[0105] In this embodiment, the lossless interpolation module based on signal processing has the advantages of being easy to implement and simple to operate.

[0106] Figure 4 This is a schematic diagram of the structure of a lossless interpolation module based on a neural network provided in an embodiment of the present invention. Figure 4 As shown, the lossless interpolation module based on the neural network includes a deconvolution layer 41 with a stride of D and a replacement module 42.

[0107] The lossless interpolation module based on neural networks is specifically used for:

[0108] For any first frame sequence, a deconvolution layer 41 is used to convert the first frame sequence into an intermediate sequence of length D*F, where F is the length of the first frame sequence; then, for each sample point in the intermediate sequence, a replacement module 42 is used to replace the sample point with the corresponding sample point in the first frame sequence to obtain the corresponding second frame sequence.

[0109] In a specific example, the first frame sequence An intermediate sequence of length DF is first generated by a trainable deconvolutional layer with a stride of D

[0110]

[0111] Next, for each sample point in the intermediate sequence , the sample point is replaced by the sample point corresponding to the sample point in the first frame sequence using the replacement module 42, and the sample points at other positions remain unchanged, generating the final interpolated second frame sequence That is:

[0112]

[0113] In this embodiment, the deconvolutional layer 41 in the neural network-based lossless interpolation module is trainable, so that the module can be jointly trained with the preset phase prediction neural network, the interpolation process is learnable, and has good generalization ability.

[0114] In the method of the present embodiment, after generating the long frame shift speech phase spectrum based on the short frame shift speech phase spectrum, the method of the present embodiment can further include:

[0115] combining the long frame shift speech phase spectrum with the long frame shift speech log amplitude spectrum to obtain a short-time complex spectrum; and then performing inverse short-time Fourier transform on the short-time complex spectrum to obtain a speech waveform That is:

[0116]

[0117] In the present embodiment, the present application can improve the prediction result accuracy of the long frame shift speech phase spectrum, and further, the quality of the reconstructed speech waveform is high.

[0118] In the present embodiment, the loss function of the preset phase prediction neural network includes a linear combination of instantaneous phase loss, group delay loss and instantaneous angular frequency loss.

[0119] Referring to Figure 1 , the loss function of the preset phase prediction neural network is defined between the predicted short frame shift speech phase spectrum and the natural short frame shift speech phase spectrum . Among them, the extraction process of the natural short frame shift phase spectrum P S is:

[0120] First, the natural speech time domain waveform In the middle, the short-time complex spectrum is extracted by a short-time Fourier transform set as T / (DF) by frame shifting where is a complex set. Then the real part and the imaginary part of the short-time complex spectrum S are calculated Finally, P is calculated by a binary activation function Φ S , i.e. S P S = Φ(R S , I AW ).

[0121] The loss function is a phase unwrapping loss used in the preset phase prediction neural network, including an instantaneous phase loss a group delay loss and an instantaneous angular frequency loss

[0122]

[0123]

[0124]

[0125] wherein, is a linear unwrapping function, used to avoid the problem of training error expansion caused by phase wrapping, and round represents rounding; f AW (X) represents an element-wise unwrapping function calculation on the matrix X; represents the average value calculated on all elements in the matrix Y; Δ DF and

[0126] Δ DT respectively represent the difference along the frequency axis and the difference along the time axis; represents the calculation of the expected value.

[0127] The training criterion of the preset phase prediction neural network is to minimize the linear combination of the instantaneous phase loss, the group delay loss and the instantaneous angular frequency loss, i.e.

[0128]

[0129] It should be noted that in the embodiments of the present application, the training process of the preset phase prediction neural network only includes the interpolation stage and the prediction stage described above.

[0130] Based on one overall inventive concept, the present application also provides a long-frame-shift speech phase spectrum prediction device. Figure 5 is a structural schematic diagram of a long-frame-shift speech phase spectrum prediction device provided by the embodiments of the present application. As Figure 5 shown, the device comprises:

[0131] The interpolation module 51 is configured to convert the long-frame-shift speech log-amplitude spectrum into a short-frame-shift speech log-amplitude spectrum based on a lossless interpolation technology.

[0132] The prediction module 52 is configured to perform phase spectrum prediction on the short-frame-shift speech log-amplitude spectrum to obtain a short-frame-shift speech phase spectrum.

[0133] The extraction module 53 is configured to generate a long-frame-shift speech phase spectrum based on the short-frame-shift speech phase spectrum.

[0134] Optionally, the interpolation module 51 can include:

[0135] The frequency division unit is configured to divide the long-frame-shift speech log-amplitude spectrum into a plurality of first frame sequences.

[0136] The interpolation unit is configured to perform lossless interpolation on each first frame sequence to obtain a second frame sequence.

[0137] The splicing unit is configured to splice the second frame sequences in the frequency axis to obtain the short-frame-shift speech log-amplitude spectrum.

[0138] Optionally, the interpolation unit can be specifically configured to:

[0139] Insert a preset number of sample points after each sample point in the first frame sequence to obtain the corresponding second frame sequence.

[0140] Optionally, the interpolation unit can be specifically configured to:

[0141] Perform lossless interpolation on the first frame sequence using a lossless interpolation module based on signal processing or a lossless interpolation module based on a neural network to obtain the corresponding second frame sequence.

[0142] The lossless interpolation module based on signal processing includes a zero-padder and a low-pass filter.

[0143] The lossless interpolation module based on signal processing is specifically configured to:

[0144] Use the zero-padder to insert a preset number of zero points after each sample point in the first frame sequence to obtain a zero-padded sequence.

[0145] Use the low-pass filter to replace each zero point in the zero-padded sequence with an interpolation point to obtain the corresponding second frame sequence.

[0146] The lossless interpolation module based on the neural network includes a deconvolution layer with a step size of D and a replacement module.

[0147] The lossless interpolation module based on the neural network is specifically configured to:

[0148] Using a deconvolutional layer, the first frame sequence is converted into an intermediate sequence of length D*F, where F is the length of the first frame sequence;

[0149] For each sample point in the intermediate sequence, the replacement module is used to replace the sample point with the corresponding sample point in the first frame sequence, thus obtaining the corresponding second frame sequence.

[0150] Optionally, the extraction module 53 can be specifically used for:

[0151] Extract the non-interpolated frames from the phase spectrum of short-frame shifted speech;

[0152] By splicing the extracted non-interpolated frames along the frequency axis, a long-frame shift speech phase spectrum is obtained.

[0153] Optionally, the apparatus of this embodiment may further include a speech waveform generation module, used for:

[0154] The phase spectrum of long-frame shifted speech and the logarithmic amplitude spectrum of long-frame shifted speech are combined to obtain the short-time complex spectrum;

[0155] The speech waveform is obtained by performing an inverse short-time Fourier transform on the short-time complex spectrum.

[0156] Optionally, the prediction module 52 can be specifically used for:

[0157] The short frame shift speech logarithmic amplitude spectrum is input into a preset phase prediction neural network to obtain the short frame shift speech phase spectrum output by the preset phase prediction neural network.

[0158] The preset phase prediction neural network includes a residual convolutional network, parallel first and second linear convolutional layers, and a phase calculation unit. The loss function of the preset phase prediction neural network includes a linear combination of instantaneous phase loss, group delay loss, and instantaneous angular frequency loss.

[0159] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0160] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0161] The steps in the methods of the various embodiments of the present invention can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in the various embodiments can be replaced or combined.

[0162] The modules and sub-modules in the various embodiments of the present invention can be merged, divided, and deleted according to actual needs.

[0163] In the embodiments provided by this invention, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0164] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0165] Furthermore, the functional modules or sub-modules in the various embodiments of the present invention can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0166] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0167] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0168] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0169] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for predicting the phase spectrum of long-frame shifted speech, characterized in that, include: Based on lossless interpolation technology, the long-frame shift speech logarithmic amplitude spectrum is converted into a short-frame shift speech logarithmic amplitude spectrum, including: frequency division of the long-frame shift speech logarithmic amplitude spectrum to obtain multiple first frame sequences; lossless interpolation of each first frame sequence to obtain a second frame sequence; and concatenation of each second frame sequence along the frequency axis to obtain the short-frame shift speech logarithmic amplitude spectrum. Specifically, lossless interpolation of the first frame sequence to obtain the second frame sequence includes: using a zero-padding unit in a signal processing-based lossless interpolation module to insert a preset number of zero points after each sample point in the first frame sequence to obtain a zero-padding sequence; and using a low-pass filter in the signal processing-based lossless interpolation module to replace each zero point in the zero-padding sequence with an interpolation point to obtain the corresponding second frame sequence. Phase spectrum prediction is performed using the logarithmic amplitude spectrum of the short frame shifted speech to obtain the phase spectrum of the short frame shifted speech. Generating a long frame shifted speech phase spectrum based on the short frame shifted speech phase spectrum includes: extracting non-interpolated frames from the short frame shifted speech phase spectrum; and splicing the extracted non-interpolated frames along the frequency axis to obtain the long frame shifted speech phase spectrum.

2. The method according to claim 1, characterized in that, Lossless interpolation is performed on the first frame sequence to obtain the second frame sequence, which also includes: The lossless interpolation module based on the neural network performs lossless interpolation on the first frame sequence to obtain the corresponding second frame sequence.

3. The method according to claim 2, characterized in that, The lossless interpolation module based on neural networks includes a deconvolution layer with a stride of D and a replacement module. The lossless interpolation module based on neural networks is specifically used for: The deconvolutional layer is used to convert the first frame sequence into an intermediate sequence of length D*F, where F is the length of the first frame sequence. For each sample point in the intermediate sequence, the replacement module is used to replace the sample point with the corresponding sample point in the first frame sequence to obtain the corresponding second frame sequence.

4. The method according to claim 1, characterized in that, After generating the long frame shift speech phase spectrum based on the short frame shift speech phase spectrum, the process further includes: The long-frame shift speech phase spectrum and the long-frame shift speech logarithmic amplitude spectrum are combined to obtain a short-time complex spectrum; The speech waveform is obtained by performing an inverse short-time Fourier transform on the short-time complex spectrum.

5. The method according to claim 1, characterized in that, Phase spectrum prediction is performed using the logarithmic amplitude spectrum of the short-frame shifted speech, resulting in the phase spectrum of the short-frame shifted speech, specifically including: The short frame shift speech logarithmic amplitude spectrum is input into a preset phase prediction neural network to obtain the short frame shift speech phase spectrum output by the preset phase prediction neural network. The preset phase prediction neural network includes a residual convolutional network, parallel first and second linear convolutional layers, and a phase calculation unit. The loss function of the preset phase prediction neural network includes a linear combination of instantaneous phase loss, group delay loss, and instantaneous angular frequency loss.

6. A long-frame shift speech phase spectrum prediction device, characterized in that, include: The interpolation module is used to convert the logarithmic amplitude spectrum of long-frame-shifted speech into the logarithmic amplitude spectrum of short-frame-shifted speech based on lossless interpolation techniques. The prediction module is used to perform phase spectrum prediction using the logarithmic amplitude spectrum of the short frame shift speech to obtain the phase spectrum of the short frame shift speech. An extraction module is used to generate a long frame shift speech phase spectrum based on the short frame shift speech phase spectrum; The interpolation module is specifically used for: frequency division of the long-frame shift speech logarithmic amplitude spectrum to obtain multiple first frame sequences; lossless interpolation of each first frame sequence to obtain a second frame sequence; and splicing the second frame sequences along the frequency axis to obtain the short-frame shift speech logarithmic amplitude. Specifically, the lossless interpolation of the first frame sequence to obtain the second frame sequence by the interpolation module includes: using a zero-padding unit in the signal processing-based lossless interpolation module to insert a preset number of zero points after each sample point in the first frame sequence to obtain a zero-padding sequence; and using a low-pass filter in the signal processing-based lossless interpolation module to replace each zero point in the zero-padding sequence with an interpolation point to obtain the corresponding second frame sequence. The extraction module is specifically used to: extract the non-interpolated frames from the short frame shift speech phase spectrum; and splice the extracted non-interpolated frames according to the frequency axis to obtain the long frame shift speech phase spectrum.

Images