Voice signal reconstruction method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a voice signal reconstruction method and device, equipment and a storage medium. The method comprises the following steps: acquiring an initial voice signal, and performing Fourier transform calculation on the initial voice signal to obtain frequency spectrum information; determining a corresponding initial Mel spectrum based on the frequency spectrum information, and performing feature extraction operation on the initial Mel spectrum through a feature extraction network in a target vocoder to obtain an initial feature vector; analyzing the initial feature vector through a frequency spectrum generation network in the target vocoder to obtain a target amplitude spectrum and a target phase spectrum; and performing inverse Fourier transform on the target amplitude spectrum and the target phase spectrum to obtain a reconstructed target voice signal. The embodiment of the invention aims to reconstruct the voice signal more accurately, so that the naturalness and definition of the generated voice signal are higher.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for reconstructing a speech signal, a device for reconstructing a speech signal, a computer device, and a computer-readable storage medium. Background Art

[0002] The reconstruction of speech signals usually refers to analyzing the speech signals at the sending end, extracting the characteristic parameters of the speech signals, encoding and encrypting them to match the channel, and transmitting them to the receiving end through the information channel. The receiving end restores (reconstructs) the original speech waveform based on the received characteristic parameters.

[0003] In the related technology, speech is mainly generated by using the intermediate features predicted by the acoustic model through the vocoder. However, this method does not model the phase information well, which easily leads to the lack of naturalness of the speech, poor reconstruction of high-frequency details, which makes the speech sound blurry, and insufficient time domain continuity, which leads to incoherent or jittery output speech. At the same time, the related technology is highly dependent on the quality of the input data, which easily leads to poor performance of the generated speech under noise or distortion conditions. Therefore, it is necessary to propose a method for reconstructing speech signals, which can reconstruct speech signals more accurately and make the generated speech signals more natural and clear. Summary of the invention

[0004] The present application provides a speech signal reconstruction method, a speech signal reconstruction device, a computer device and a computer-readable storage medium, which are intended to reconstruct the speech signal more accurately and make the generated speech signal more natural and clear.

[0005] To achieve the above object, the present application also provides a method for reconstructing a speech signal, comprising:

[0006] Acquire an initial speech signal, and perform Fourier transform calculation on the initial speech signal to obtain spectrum information;

[0007] Determine a corresponding initial Mel spectrum based on the frequency spectrum information, and perform a feature extraction operation on the initial Mel spectrum through a feature extraction network in a target vocoder to obtain an initial feature vector;

[0008] Analyzing the initial feature vector through a spectrum generation network in the target vocoder to obtain a target amplitude spectrum and a target phase spectrum;

[0009] Performing inverse Fourier transform on the target amplitude spectrum and the target phase spectrum to obtain a reconstructed target speech signal.

[0010] To achieve the above object, the present application also provides a speech signal reconstruction device, comprising:

[0011] An acquisition module is used to acquire an initial speech signal and perform Fourier transform calculation on the initial speech signal to obtain spectrum information;

[0012] A determination module, configured to determine a corresponding initial Mel spectrum based on the frequency spectrum information, and perform a feature extraction operation on the initial Mel spectrum through a feature extraction network in a target vocoder to obtain an initial feature vector;

[0013] An analysis module, configured to analyze the initial feature vector through a spectrum generation network in the target vocoder to obtain a target amplitude spectrum and a target phase spectrum;

[0014] The speech signal reconstruction module is used to perform inverse Fourier transform on the target amplitude spectrum and the target phase spectrum to obtain a reconstructed target speech signal.

[0015] In addition, to achieve the above-mentioned purpose, the present application also provides a computer device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the steps of the speech signal reconstruction method provided in any one of the embodiments of the present application when executing the computer program.

[0016] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the steps of the speech signal reconstruction method described in any one of the embodiments of the present application.

[0017] The embodiment of the present application discloses a method for reconstructing a speech signal, a device for reconstructing a speech signal, a computer device, and a computer-readable storage medium. The method includes obtaining an initial speech signal, and performing Fourier transform calculation on the initial speech signal to obtain spectrum information. Furthermore, the corresponding initial Mel spectrum can be determined based on the spectrum information, and the initial Mel spectrum can be subjected to feature extraction operation through the feature extraction network in the target vocoder to obtain an initial feature vector. Thus, the initial feature vector can be analyzed through the spectrum generation network in the target vocoder to obtain a target amplitude spectrum and a target phase spectrum. Finally, the target amplitude spectrum and the target phase spectrum can be inversely Fourier transformed to obtain a reconstructed target speech signal. The present application extracts the spectrum information of the initial speech signal through Fourier transform, combines the feature extraction network and the spectrum generation network of the target vocoder, accurately generates the target amplitude spectrum and the target phase spectrum, and reconstructs the target speech signal through inverse Fourier transform. The present application can make full use of the modeling capabilities of the spectrum features and the deep learning model, significantly improve the accuracy and naturalness of speech reconstruction, effectively reduce speech distortion and noise interference, and thus achieve higher quality speech restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a scene schematic diagram of a method for reconstructing a speech signal provided in an embodiment of the present application;

[0020] Figure 2 It is a flowchart of a method for reconstructing a speech signal provided in an embodiment of the present application;

[0021] Figure 3 is a schematic block diagram of a speech signal reconstruction device provided in an embodiment of the present application;

[0022] Figure 4 It is a schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions. In addition, although the functional modules are divided in the device schematic, in some cases, the module division can be different from that in the device schematic.

[0025] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0026] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0027] like Figure 1 As shown, the speech signal reconstruction method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. The application environment includes a terminal device 110 and a server 120, wherein the terminal device 110 can communicate with the server 120 through a network. Specifically, the server 120 can obtain an initial speech signal and perform Fourier transform calculation on the initial speech signal to obtain spectrum information. Further, the corresponding initial Mel spectrum can be determined based on the spectrum information, and the initial Mel spectrum can be subjected to feature extraction operation through the feature extraction network in the target vocoder to obtain an initial feature vector. Thus, the initial feature vector can be analyzed by the spectrum generation network in the target vocoder to obtain a target amplitude spectrum and a target phase spectrum. Finally, the target amplitude spectrum and the target phase spectrum can be inversely Fourier transformed to obtain a reconstructed target speech signal, thereby sending the reconstructed target speech signal to the terminal device 110. Among them, the server 120 can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal device 110 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0028] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of a method for reconstructing a speech signal provided in an embodiment of the present application. Figure 2 As shown, the method for reconstructing the speech signal can be implemented through steps S11 to S14.

[0029] Step S11: Acquire an initial speech signal, and perform Fourier transform calculation on the initial speech signal to obtain spectrum information.

[0030] The initial speech signal is a continuous time domain signal, usually represented as audio waveform data. The initial speech signal may be composed of a series of amplitude values ​​that vary with time to reflect the temporal changes of the sound wave. Furthermore, the initial speech signal may be recorded by a microphone or other audio acquisition device and stored in digital form, which is not limited in the present application.

[0031] It should be noted that Fourier transform is a mathematical method for converting time domain signals into frequency domain. Spectrum information is a representation of speech signals in the frequency domain. The spectrum information may include amplitude spectrum information and phase spectrum information, which is not limited in this application.

[0032] For example, the initial speech signal can be divided into overlapping frames, and then each frame signal is multiplied by a window function to obtain each frame signal after windowing. Finally, each frame signal after windowing can be subjected to spectrum analysis to obtain a spectrum signal.

[0033] The above embodiment converts the time domain signal of the initial speech signal into a frequency domain representation through Fourier transform, and extracts the spectrum information to provide basic data support for subsequent speech feature extraction and reconstruction.

[0034] Step S12: determining the corresponding initial Mel spectrum based on the frequency spectrum information, and performing feature extraction operation on the initial Mel spectrum through a feature extraction network in the target vocoder to obtain an initial feature vector.

[0035] It should be noted that Mel spectrum is a speech spectrum representation method based on the Mel frequency scale. Mel spectrum emphasizes the perception of low frequencies and weakens the perception of high frequencies by mapping frequency information to a frequency scale that conforms to the characteristics of human auditory perception. The feature extraction network is a deep neural network (such as a convolutional neural network or a variant network) that is used to extract high-level features from the input Mel spectrum.

[0036] For example, the spectral information can be mapped through a Mel filter to obtain an initial Mel spectrum, and then the initial Mel spectrum is input into a feature extraction network in the target vocoder. The input format of the initial Mel spectrum is usually a two-dimensional matrix, with the horizontal axis being the time frame and the vertical axis being the Mel frequency channel. Thus, an initial feature vector of a high-dimensional vector can be output through the feature extraction network. The initial feature vector may include, but is not limited to, information such as the semantics, pitch, and rhythm of the initial speech signal, which is not limited in this application.

[0037] The above implementation method converts the spectrum information into an initial Mel spectrum that conforms to the auditory characteristics of the human ear, extracts speech features through the feature extraction network in the target vocoder, and generates an initial feature vector. This process plays a key role in speech reconstruction, not only improving the effectiveness of feature representation, but also optimizing the modeling basis of speech signals.

[0038] Step S13: Analyze the initial feature vector through the spectrum generation network in the target vocoder to obtain a target amplitude spectrum and a target phase spectrum.

[0039] It should be noted that the spectrum generation network is a deep learning model that aims to convert the input speech feature vector (such as Mel-spectrogram features or other high-dimensional feature representations) into complete spectrum information, including target amplitude spectrum and target phase spectrum. The target amplitude spectrum is used to describe the amplitude strength of the speech signal at each frequency. The target phase spectrum is used to describe the phase information of the speech signal at each frequency, which determines the temporal characteristics and naturalness of the speech.

[0040] For example, the initial feature vector can be mapped to a higher-dimensional representation space as the initial stage of spectrum generation, and then the deconvolution layer is used to gradually restore the time and frequency details in the feature vector to generate a spectrum representation, that is, to obtain the target amplitude spectrum and the target phase spectrum.

[0041] The above implementation method uses the spectrum generation network in the target vocoder to perform deep learning analysis on the initial feature vector and output the target amplitude spectrum and target phase spectrum. This process converts the input high-dimensional speech feature vector into a spectrum representation, directly providing the basic information required for speech reconstruction, ensuring the clarity, naturalness and speech perception quality of the generated speech.

[0042] Step S14: performing inverse Fourier transform on the target amplitude spectrum and the target phase spectrum to obtain a reconstructed target speech signal.

[0043] Among them, the target amplitude spectrum is used to describe the energy distribution of the speech signal in the frequency domain, reflecting the characteristics of the speech such as sound intensity and timbre; the target phase spectrum is used to describe the phase information of each frequency component in the speech signal, determining the temporal characteristics and naturalness of the speech; the inverse Fourier transform can be used to restore the amplitude spectrum and phase spectrum in the frequency domain to the original signal in the time domain.

[0044] For example, a complex spectrum in the frequency domain can be generated from the target amplitude spectrum and the target phase spectrum, and then an inverse Fourier transform is performed on the complex spectrum to obtain a time domain signal, that is, a reconstructed target speech signal. In addition, the reconstructed target speech signal can also be subjected to an artifact or noise removal operation or a volume normalization operation to ensure the consistency of the output volume, thereby improving the sound quality of the reconstructed target speech signal.

[0045] The above implementation method combines the target amplitude spectrum and the target phase spectrum into complete frequency domain information, and restores it to a time domain speech signal through inverse Fourier transform. Through the above efficient algorithm and precise phase estimation, the target speech signal finally generated can be close to the real speech in auditory perception.

[0046] The method for reconstructing a speech signal disclosed in an embodiment of the present application includes obtaining an initial speech signal, and performing Fourier transform calculation on the initial speech signal to obtain spectrum information. Furthermore, the corresponding initial Mel spectrum can be determined based on the spectrum information, and the initial Mel spectrum can be subjected to feature extraction operation through the feature extraction network in the target vocoder to obtain an initial feature vector. Thus, the initial feature vector can be analyzed through the spectrum generation network in the target vocoder to obtain a target amplitude spectrum and a target phase spectrum. Finally, the target amplitude spectrum and the target phase spectrum can be inverse Fourier transformed to obtain a reconstructed target speech signal. The present application extracts the spectrum information of the initial speech signal through Fourier transform, combines the feature extraction network and the spectrum generation network of the target vocoder, accurately generates the target amplitude spectrum and the target phase spectrum, and reconstructs the target speech signal through inverse Fourier transform. The present application can make full use of the modeling capabilities of the spectrum features and the deep learning model, significantly improve the accuracy and naturalness of speech reconstruction, effectively reduce speech distortion and noise interference, and thus achieve higher quality speech restoration.

[0047] Optionally, the spectrum information includes at least an initial amplitude spectrum, and determining a corresponding initial Mel spectrum based on the spectrum information includes: performing a frequency reduction operation on the initial amplitude spectrum through a Mel filter to obtain an initial Mel spectrum; wherein the frequency reduction operation at least includes an operation of multiplying the initial amplitude spectrum by a Mel filter group coefficient matrix.

[0048] Among them, the Mel filter bank is a group of bandpass filters used to convert the initial amplitude spectrum into a Mel frequency domain representation. The Mel filter is usually a triangular filter, the bandwidth of each filter increases with the increase of frequency, and the center frequency of the Mel filter is distributed on the Mel frequency scale.

[0049] For example, the initial amplitude spectrum may be down-converted by a Mel filter to obtain an initial Mel spectrum, wherein the down-converting operation refers to reducing the dimension of the initial amplitude spectrum by a Mel filter bank to obtain a more compact initial Mel spectrum.

[0050] For example, a Mel filter bank coefficient matrix may be constructed, wherein each row of the Mel filter bank coefficient matrix corresponds to a Mel filter and each column corresponds to a spectrum point. Further, the initial amplitude spectrum is multiplied by the Mel filter bank coefficient matrix to achieve weighted summation of each Mel filter within the corresponding frequency range, thereby calculating the initial Mel spectrum.

[0051] In the above implementation, the initial amplitude spectrum is down-converted by the Mel filter, which can extract the most important frequency information for human auditory perception in the speech signal. The core of the down-conversion operation is to multiply the initial amplitude spectrum with the Mel filter bank coefficient matrix to obtain the initial Mel spectrum. This process not only retains key information, but also reduces the data dimension, laying the foundation for subsequent feature extraction and speech reconstruction.

[0052] Optionally, the feature extraction network includes a residual block, and a feature extraction operation is performed on the initial Mel spectrum through the feature extraction network in the target vocoder to obtain an initial feature vector, including: stacking the residual blocks to obtain a stacked residual block; and performing a feature extraction operation on the initial Mel spectrum through the stacked residual block to obtain an initial feature vector.

[0053] Based on the above embodiment, the number of layers of the residual block includes 16 layers, the initial feature vector includes a hidden state feature vector, and the residual block structure includes a one-dimensional convolution, an activation function, and a matrix point layer.

[0054] Among them, the purpose of the feature extraction network is to extract higher-level feature representations, namely initial feature vectors, from the input initial Mel spectrum. These features can better express the structure and content of the speech signal and provide support for subsequent processing.

[0055] It should be understood that since the residual block has the ability to efficiently extract deep features and can alleviate the gradient vanishing and overfitting problems in the deep network, the feature extraction network can be implemented in the form of residual block stacking.

[0056] For example, the residual block includes a one-dimensional convolution, an activation function, and a matrix point layer. The one-dimensional convolution is used to capture the local patterns and features of the Mel spectrum in the time dimension; the activation function is used to introduce nonlinear capabilities so that the model can learn complex features; the matrix point layer combines information from different channels through linear transformation to adjust the dimension of the feature channel.

[0057] For example, 16 layers of residual blocks can be stacked in layers to form a feature extraction network. The stacking operation gradually constructs a more complex and higher-level speech feature representation through feature extraction of each residual block, thereby obtaining an initial feature vector.

[0058] For example, the initial feature vector includes a hidden state feature vector, which is a feature of a specific dimension after dimension reduction or selection, thereby being able to further participate in spectrum generation or other subsequent processing tasks in the target vocoder.

[0059] In the above embodiment, the initial Mel spectrum is subjected to a feature extraction operation of a 16-layer residual block through the feature extraction network in the target vocoder, and the initial feature vector can be efficiently generated. The structure of the residual block (including one-dimensional convolution, activation function, matrix point layer and residual connection) ensures the stability and extraction capability of the deep network, and provides a high-quality input feature representation for the subsequent spectrum generation network.

[0060] Optionally, the spectrum generation network includes a first convolution layer and a second convolution layer, and the initial feature vector is analyzed by the spectrum generation network in the target vocoder to obtain a target amplitude spectrum and a target phase spectrum, including: analyzing the initial feature vector by the first convolution layer to obtain a target amplitude spectrum, wherein the activation function of the first convolution layer is an exponential function and is non-negative; and analyzing the initial feature vector by the second convolution layer to obtain a target phase spectrum.

[0061] For example, the first convolution layer is used to generate the target amplitude spectrum, and the second convolution layer is used to generate the target phase spectrum. Specifically, the first convolution layer can perform a linear transformation on the input initial feature vector, capture the local pattern in the feature vector, and map it to the amplitude spectrum space, thereby obtaining the target amplitude spectrum. In addition, the first convolution layer can implement non-negativity constraints through exponential functions, while enhancing the sensitivity of amplitude changes and highlighting the significant features of the target amplitude spectrum; the second convolution layer can perform a linear transformation on the input feature vector, extract phase-related features, and obtain the target phase spectrum.

[0062] It should be noted that, unlike the generation of amplitude spectrum, the target phase spectrum usually does not require non-negativity constraints, so the output of the second convolutional layer is directly used as the phase spectrum result.

[0063] The above implementation method uses the spectrum generation network to generate the target amplitude spectrum using the first convolution layer and the target phase spectrum using the second convolution layer, completing the conversion from the feature vector to the complete target spectrum. The amplitude spectrum is constrained to be non-negative by the exponential activation function, reflecting the frequency amplitude characteristics of the speech signal; the phase spectrum is directly output to represent the phase characteristics of the frequency. This method is simple and efficient in structure, laying a solid foundation for the subsequent steps of speech reconstruction.

[0064] Optionally, the target amplitude spectrum and the target phase spectrum are inversely transformed to obtain a reconstructed target speech signal, including: determining the complex form of the target phase spectrum to obtain a target phase spectrum in complex form; constructing a complex frequency spectrum based on the target amplitude spectrum and the target phase spectrum in complex form; and performing an inverse Fourier transform on the complex frequency spectrum to obtain a reconstructed target speech signal.

[0065] It should be noted that the complex phase spectrum is used to reconstruct the complete spectrum in combination with the amplitude information, and the complex spectrum can be constructed using the target amplitude spectrum and the complex target phase spectrum. Thus, the complex spectrum can be inverse Fourier transformed to obtain the reconstructed target speech signal.

[0066] In the above implementation, the complex spectrum is reconstructed through the complex form of the target amplitude spectrum and the target phase spectrum, and the inverse Fourier transform is performed on it, so that the time domain characteristics of the speech signal can be completely restored. This method effectively combines the amplitude and phase information, can retain the details and semantic integrity of the speech signal, and improve the quality of the reconstructed speech. Compared with direct spectrum processing, this process has higher accuracy and restoration, and provides a reliable implementation path for speech reconstruction.

[0067] Optionally, the initial mel spectrum is subjected to feature extraction operation through a feature extraction network in a target vocoder to obtain an initial feature vector, and the method further includes: introducing a loss function in the target vocoder, wherein the loss function includes an amplitude spectrum loss, a phase spectrum loss and an adversarial loss; iteratively training the loss function for the purpose of reducing the value of the loss function until an expected threshold requirement is met; obtaining an iterative target vocoder based on the iteratively trained loss function; and performing a feature extraction operation on the initial mel spectrum through a feature extraction network in the iterative target vocoder to obtain an initial feature vector.

[0068] Among them, the amplitude spectrum loss is used to measure the difference between the target amplitude spectrum and the reconstructed amplitude spectrum to ensure that the energy distribution of the reconstructed speech is as close to the original speech as possible; the phase spectrum loss is used to measure the difference between the target phase spectrum and the reconstructed phase spectrum to ensure that the phase information of the reconstructed speech is accurate, thereby improving the sound quality and naturalness of the speech; the adversarial loss is used to improve the authenticity and naturalness of the reconstructed speech by introducing the discriminator part of the generative adversarial network. After introducing the above loss functions, the target vocoder can optimize the speech reconstruction performance from multiple aspects such as amplitude information, phase information and subjective quality.

[0069] It can be understood that in order to train a target vocoder with higher accuracy, the target vocoder can be repeatedly trained to continuously reduce the loss function until the loss function meets the expected threshold requirement, and then the initial mel spectrum can be subjected to feature extraction operation based on the feature extraction network in the iterated target vocoder to obtain a relatively accurate initial feature vector.

[0070] It should be noted that the present application does not limit the above-mentioned preset method and expected threshold. For example, the preset method may be a gradient descent algorithm, a batch gradient descent algorithm, a stochastic gradient descent algorithm, etc. The present application takes the gradient descent algorithm as an example for explanation.

[0071] The purpose of the gradient descent algorithm is to find the minimum value of the loss function by iteration, or to converge to the minimum value. In a geometric sense, the gradient descent algorithm means that the gradient decreases fastest in the direction opposite to the vector where the function changes the fastest, so it is easier to find the minimum value of the function. Based on this, in an embodiment of the present application, the gradient descent algorithm can be used to perform repeated iterative training so that the loss function is continuously reduced, thereby reducing the error of the calculation result.

[0072] In the embodiment of the present application, the denoising model is repeatedly trained by using a gradient descent algorithm so that the loss function is continuously reduced to obtain an iterative target vocoder, and then a relatively accurate initial feature vector can be obtained based on the feature extraction network in the iterative target vocoder.

[0073] See also Figure 3 , Figure 3 1 is a schematic block diagram of a speech signal reconstruction device provided in an embodiment of the present application. The speech signal reconstruction device can be configured in a server to execute the above-mentioned speech signal reconstruction method.

[0074] like Figure 4 As shown, the speech signal reconstruction device 200 includes: an acquisition module 201, a determination module 202, an analysis module 203 and a speech signal reconstruction module 204.

[0075] The acquisition module 201 is used to acquire an initial speech signal and perform Fourier transform calculation on the initial speech signal to obtain spectrum information;

[0076] A determination module 202 is used to determine a corresponding initial Mel spectrum based on the frequency spectrum information, and perform a feature extraction operation on the initial Mel spectrum through a feature extraction network in a target vocoder to obtain an initial feature vector;

[0077] An analysis module 203 is used to analyze the initial feature vector through a spectrum generation network in the target vocoder to obtain a target amplitude spectrum and a target phase spectrum;

[0078] The speech signal reconstruction module 204 is used to perform inverse Fourier transform on the target amplitude spectrum and the target phase spectrum to obtain a reconstructed target speech signal.

[0079] The determination module 202 is further configured to perform a frequency reduction operation on the initial amplitude spectrum through a Mel filter to obtain the initial Mel spectrum; wherein the frequency reduction operation at least includes an operation of multiplying the initial amplitude spectrum by the Mel filter bank coefficient matrix.

[0080] The determination module 202 is further configured to perform a stacking operation on the residual blocks to obtain a stacked residual block; and perform a feature extraction operation on the initial Mel spectrum using the stacked residual block to obtain the initial feature vector.

[0081] The analysis module 203 is also used to analyze the initial feature vector through the first convolution layer to obtain the target amplitude spectrum, wherein the activation function of the first convolution layer is an exponential function and is non-negative; and to analyze the initial feature vector through the second convolution layer to obtain the target phase spectrum.

[0082] The speech signal reconstruction module 204 is also used to determine the complex form of the target phase spectrum to obtain the target phase spectrum in complex form; construct a complex frequency spectrum based on the target amplitude spectrum and the target phase spectrum in complex form; and perform an inverse Fourier transform on the complex frequency spectrum to obtain the reconstructed target speech signal.

[0083] The determination module 202 is further used to introduce a loss function into the target vocoder, wherein the loss function includes an amplitude spectrum loss, a phase spectrum loss and an adversarial loss; iteratively train the loss function for the purpose of reducing the value of the loss function until an expected threshold requirement is met; based on the iteratively trained loss function, obtain the iterative target vocoder; perform a feature extraction operation on the initial Mel spectrum through a feature extraction network in the iterative target vocoder to obtain the initial feature vector.

[0084] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0085] The method and apparatus of the present application can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer terminal devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices.

[0086] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in the form of a computer program. Figure 4 Runs on the computer device shown.

[0087] See also Figure 4 , Figure 4 1 is a schematic diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0088] like Figure 4 As shown, the computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a volatile storage medium, a non-volatile storage medium and an internal memory.

[0089] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any method for reconstructing a speech signal.

[0090] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0091] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any method for reconstructing a speech signal.

[0092] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that the structure of the computer device is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0093] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0094] Among them, in some embodiments, the processor is used to obtain an initial speech signal, and perform Fourier transform calculation on the initial speech signal to obtain spectrum information; determine the corresponding initial Mel spectrum based on the spectrum information, and perform feature extraction operation on the initial Mel spectrum through a feature extraction network in a target vocoder to obtain an initial feature vector; analyze the initial feature vector through a spectrum generation network in the target vocoder to obtain a target amplitude spectrum and a target phase spectrum; perform inverse Fourier transform on the target amplitude spectrum and the target phase spectrum to obtain a reconstructed target speech signal.

[0095] In some implementations, the processor is further configured to perform a frequency reduction operation on the initial amplitude spectrum through a Mel filter to obtain the initial Mel spectrum; wherein the frequency reduction operation at least includes an operation of multiplying the initial amplitude spectrum by the Mel filter group coefficient matrix.

[0096] In some implementations, the processor is further configured to perform a stacking operation on the residual blocks to obtain a stacked residual block; and perform a feature extraction operation on the initial Mel spectrum using the stacked residual block to obtain the initial feature vector.

[0097] In some embodiments, the processor is further used to analyze the initial feature vector through the first convolution layer to obtain the target amplitude spectrum, wherein the activation function of the first convolution layer is an exponential function and is non-negative; and analyze the initial feature vector through the second convolution layer to obtain the target phase spectrum.

[0098] In some embodiments, the processor is further used to determine the complex form of the target phase spectrum to obtain a target phase spectrum in complex form; construct a complex frequency spectrum based on the target amplitude spectrum and the target phase spectrum in complex form; and perform an inverse Fourier transform on the complex frequency spectrum to obtain the reconstructed target speech signal.

[0099] In some embodiments, the processor is further used to introduce a loss function in the target vocoder, wherein the loss function includes an amplitude spectrum loss, a phase spectrum loss, and an adversarial loss; the loss function is iteratively trained for the purpose of reducing the value of the loss function until an expected threshold requirement is met; based on the iteratively trained loss function, the iterative target vocoder is obtained; and a feature extraction operation is performed on the initial Mel spectrum through a feature extraction network in the iterative target vocoder to obtain the initial feature vector.

[0100] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed, any speech signal reconstruction method provided in the embodiment of the present application is implemented.

[0101] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0102] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like.

[0103] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for reconstructing a speech signal, characterized in that: The method comprises: Acquire an initial speech signal, and perform Fourier transform calculation on the initial speech signal to obtain spectrum information; Determine a corresponding initial Mel spectrum based on the frequency spectrum information, and perform a feature extraction operation on the initial Mel spectrum through a feature extraction network in a target vocoder to obtain an initial feature vector; Analyzing the initial feature vector through a spectrum generation network in the target vocoder to obtain a target amplitude spectrum and a target phase spectrum; Performing inverse Fourier transform on the target amplitude spectrum and the target phase spectrum to obtain a reconstructed target speech signal.

2. The method according to claim 1, characterized in that The frequency spectrum information at least includes an initial amplitude spectrum, and determining the corresponding initial Mel spectrum based on the frequency spectrum information includes: Performing a frequency reduction operation on the initial amplitude spectrum through a Mel filter to obtain the initial Mel spectrum; The frequency reduction operation at least includes an operation of multiplying the initial amplitude spectrum by the Mel filter bank coefficient matrix.

3. The method according to claim 1, characterized in that The feature extraction network includes a residual block, and the feature extraction operation is performed on the initial Mel spectrum through the feature extraction network in the target vocoder to obtain an initial feature vector, including: Performing a stacking operation on the residual blocks to obtain a stacked residual block; The stacked residual block is used to perform a feature extraction operation on the initial Mel spectrum to obtain the initial feature vector.

4. The method according to claim 3, characterized in that The number of layers of the residual block includes 16 layers, the initial feature vector includes a hidden state feature vector, and the residual block structure includes a one-dimensional convolution, an activation function, and a matrix point layer.

5. The method according to claim 1, characterized in that The spectrum generation network includes a first convolution layer and a second convolution layer, and the initial feature vector is analyzed by the spectrum generation network in the target vocoder to obtain a target amplitude spectrum and a target phase spectrum, including: Analyzing the initial feature vector through the first convolution layer to obtain the target amplitude spectrum, wherein the activation function of the first convolution layer is an exponential function and is non-negative; and The initial feature vector is analyzed through the second convolutional layer to obtain the target phase spectrum.

6. The method according to claim 1, characterized in that The step of performing inverse Fourier transform on the target amplitude spectrum and the target phase spectrum to obtain a reconstructed target speech signal includes: Determine the complex form of the target phase spectrum to obtain the complex form of the target phase spectrum; Constructing a complex frequency spectrum according to the target amplitude spectrum and the target phase spectrum in complex form; Performing an inverse Fourier transform on the complex spectrum to obtain the reconstructed target speech signal.

7. The method according to claim 1, characterized in that The step of performing a feature extraction operation on the initial Mel spectrum through a feature extraction network in the target vocoder to obtain an initial feature vector also includes: Introducing a loss function in the target vocoder, wherein the loss function includes an amplitude spectrum loss, a phase spectrum loss, and an adversarial loss; Iteratively training the loss function for the purpose of reducing the value of the loss function until the expected threshold requirement is met; Based on the loss function after iterative training, obtaining the target vocoder after iteration; The initial feature vector is obtained by performing a feature extraction operation on the initial Mel spectrum through a feature extraction network in the iterated target vocoder.

8. A speech signal reconstruction device, characterized in that: The reconstruction device comprises: An acquisition module is used to acquire an initial speech signal and perform Fourier transform calculation on the initial speech signal to obtain spectrum information; A determination module, configured to determine a corresponding initial Mel spectrum based on the frequency spectrum information, and perform a feature extraction operation on the initial Mel spectrum through a feature extraction network in a target vocoder to obtain an initial feature vector; An analysis module, configured to analyze the initial feature vector through a spectrum generation network in the target vocoder to obtain a target amplitude spectrum and a target phase spectrum; The speech signal reconstruction module is used to perform inverse Fourier transform on the target amplitude spectrum and the target phase spectrum to obtain a reconstructed target speech signal.

9. A computer device, characterized in that: include: A memory and a processor; wherein the memory is connected to the processor and is used to store a program; the processor is used to implement the steps of the speech signal reconstruction method as described in any one of claims 1 to 7 by running the program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the steps of the method for reconstructing a speech signal according to any one of claims 1 to 7.

Images