Voice generation method and device based on network model, electronic equipment and medium

Abstract

The embodiment of the invention provides a voice generation method and device based on a network model, electronic equipment and a medium, and belongs to the technical field of data processing. The network model comprises a plurality of convolution blocks; the method comprises the steps of obtaining a to-be-processed voice signal; performing framing operation on the to-be-processed voice signal to obtain a plurality of voice frames; for each voice frame, performing feature extraction on the voice frame through a network model to obtain a spectrum feature, and determining a feature dimension corresponding to the spectrum feature; inputting the spectral features into a convolution block to enable the convolution block to carry out feature transformation on the spectral features based on the feature dimensions, and outputting a short-time Fourier transform coefficient; and performing inverse Fourier transform on the short-time Fourier transform coefficient to obtain a target voice waveform. According to the embodiment of the invention, signal aliasing can be avoided, and the sound quality of generated voice is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a speech generation method, device, electronic device and medium based on a network model. Background Art

[0002] The development of speech generation models has a long history. Current speech generation models capture long-distance dependencies by autoregressively modeling speech waveforms in the time domain and using dilated convolutions to expand the receptive field. However, with the advancement of technology, existing models based on upsampling with transposed convolutions have also exposed some shortcomings.

[0003] The current model based on transposed convolution for upsampling will cause signal aliasing, which is a common problem in the process of signal sampling and reconstruction, causing errors and distortion. The main reason for aliasing is that the signal sampling frequency is not enough to meet the Nyquist-Shannon sampling theorem, that is, the sampling frequency should be at least twice the highest frequency of the signal. If the sampling frequency is lower than this critical value, frequency aliasing will occur. Aliasing will cause high-frequency components to be incorrectly mapped to low-frequency areas, causing signal distortion. This phenomenon will lead to a decrease in sound quality and unclear pronunciation in speech generation, seriously affecting the naturalness and intelligibility of speech. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a speech generation method, device, electronic device and medium based on a network model, which can avoid signal aliasing and improve the sound quality of the generated speech.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a speech generation method based on a network model, wherein the network model includes a plurality of convolution blocks; the method includes: Obtaining a voice signal to be processed; Performing a frame operation on the speech signal to be processed to obtain a plurality of speech frames; For each of the speech frames, extract features of the speech frame through a network model to obtain spectral features, and determine feature dimensions corresponding to the spectral features; Inputting the spectral feature into the convolution block, so that the convolution block performs feature transformation on the spectral feature based on the feature dimension and outputs a short-time Fourier transform coefficient; The short-time Fourier transform coefficients are subjected to inverse Fourier transform to obtain a target speech waveform.

[0006] In some embodiments, the inputting the spectral feature into the convolution block so that the convolution block performs feature transformation on the spectral feature based on the feature dimension and outputs a short-time Fourier transform coefficient comprises: Inputting the spectral feature into the convolution block, so that the convolution block performs feature transformation on the spectral feature based on the feature dimension and outputs a convolution feature; Performing feature segmentation on the convolution feature through a preset convolution layer, and outputting an amplitude spectrum and a phase spectrum; Short-time Fourier transform coefficients are generated based on the magnitude spectrum and the phase spectrum.

[0007] In some embodiments, the convolution block includes a separation convolution layer, a convolution layer, and an aggregation convolution layer; the inputting the spectral feature into the convolution block so that the convolution block performs a feature transformation on the spectral feature based on the feature dimension and outputs the convolution feature, including: Inputting the spectral feature into the convolution block to perform a separation convolution operation on the convolution feature through the separation convolution layer to obtain a first feature map; Performing a feature capture operation on the first feature map through the convolution layer to obtain a second feature map; The first feature map and the second feature map are subjected to feature aggregation through the aggregation convolution layer, and convolution features having the same feature dimension as the feature are output.

[0008] In some embodiments, the framing operation is performed on the speech signal to be processed to obtain a plurality of speech frames, including: Performing a frame division operation on the speech signal to be processed according to a preset frame length and a preset frame shift to obtain a plurality of segmented speech frames; For each of the segmented speech frames, a windowing operation is performed on the frame boundary of the segmented speech frame by using a preset window function to obtain multiple speech frames.

[0009] In some embodiments, extracting features from the speech frame using a network model to obtain spectral features includes: For each of the speech frames, Fourier transform is performed on the speech frame through a network model to obtain a spectrum representation; Combining all the frequency spectrum representations to obtain a frequency spectrum diagram; Extracting coefficients from the spectrum to obtain Mel-frequency cepstrum coefficients; Performing energy extraction on the spectrum graph to obtain spectrum energy; Performing a weighted average operation on the spectrum representation according to the spectrum graph to obtain a spectrum centroid; A spectrum feature is obtained according to the Mel-frequency cepstral coefficients, the spectrum energy and the spectrum centroid.

[0010] In some embodiments, performing an inverse Fourier transform on the short-time Fourier transform coefficients to obtain a target speech waveform includes: For each short-time Fourier transform coefficient of the speech frame, waveform recovery is performed on the short-time Fourier transform coefficient by inverse short-time Fourier transform to obtain a signal segment; All the signal segments are superimposed to obtain a target speech waveform.

[0011] In some embodiments, before performing a frame operation on the speech signal to be processed to obtain a plurality of speech frames, the method further includes: Performing a denoising operation on the speech signal to be processed, and performing speech enhancement on the speech signal to be processed after the denoising operation; The speech signal to be processed after speech enhancement is normalized.

[0012] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a speech generation device based on a network model, wherein the network model includes a plurality of convolution blocks; the device includes: A signal acquisition module, used to acquire a speech signal to be processed; A signal framing module, used for performing a framing operation on the speech signal to be processed to obtain a plurality of speech frames; A feature extraction module, used for extracting features of each speech frame through a network model to obtain spectral features, and determining feature dimensions corresponding to the spectral features; A feature transformation module, used for inputting the spectral feature into the convolution block, so that the convolution block performs feature transformation on the spectral feature based on the feature dimension and outputs a short-time Fourier transform coefficient; The inverse Fourier transform module is used to perform an inverse Fourier transform on the short-time Fourier transform coefficients to obtain a target speech waveform.

[0013] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, an electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the network model-based speech generation method as described in the first aspect when executing the computer program.

[0014] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the network model-based speech generation method as described in the first aspect.

[0015] The network model-based speech generation method, device, electronic device and storage medium proposed in the present application first obtain the speech signal to be processed. Since the speech signal to be processed is non-stationary, the embodiment of the present application will perform a frame operation on the speech signal to be processed to obtain multiple speech frames, which can divide the continuous signal processing into multiple independent frames, thereby improving the accuracy of subsequent spectrum analysis. For each speech frame, the network model is used to extract features of the speech frame, so that the energy distribution of the speech frame at different time points and frequencies can be determined, and the spectral features can be obtained. It can capture the dynamic changes and rhythm information of the speech and determine the feature dimensions corresponding to the spectral features. , then, the spectral features are input into the convolution block so that the convolution block performs feature transformation on the spectral features based on the feature dimension, thereby ensuring that the feature dimension of the spectral features of each input convolution block is consistent with the output dimension, further ensuring the time resolution of the input spectral features, and outputting the short-time Fourier transform coefficients, so that the intensity and energy distribution of the speech signal to be processed at different frequencies can be obtained, and at the same time, the phase information of each frequency component in the speech signal to be processed can be obtained. Finally, the short-time Fourier transform coefficients are inversely Fourier transformed to obtain the target speech waveform, thereby avoiding unclear speech due to signal aliasing and improving the sound quality of the generated speech. The embodiment of the present application does not perform transformation in the time dimension, but restores the waveform by calculating the short-time Fourier transform coefficients of the input features, thereby reducing a large number of upsampling calculations in the current waveform processing process and reducing the parameters of the entire model. And it avoids the aliasing phenomenon that may occur during the upsampling process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a network model-based speech generation method provided in an embodiment of the present application; Figure 2 yes Figure 1 Flow chart of step S104 in FIG. Figure 3 yes Figure 2 Flow chart of step S201 in FIG. Figure 4 yes Figure 1 Flow chart of step S102 in FIG. Figure 5 It is a specific flow chart of extracting features of speech frames through a network model provided in an embodiment of the present application; Figure 6 yes Figure 1 Flow chart of step S105 in FIG. Figure 7 is a specific flow chart of a speech generation method based on a network model provided in another embodiment of the present application; Figure 8 It is a structural schematic diagram of a speech generation device based on a network model provided in an embodiment of the present application; Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0020] First, some nouns involved in this application are analyzed: Natural Language Processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0021] Hanning Window (Hann): Hanning Window is a commonly used window function used to reduce spectrum leakage. It is implemented by multiplying a cosine square window function at both ends of the signal.

[0022] Hamming Window (Ham): The Hamming Window is another commonly used window function, which is similar to the Hanning Window, but the coefficients multiplied by the Hamming Window at both ends are slightly different, usually 0.54, which makes the Hamming Window slightly better than the Hanning Window in sidelobe attenuation in the frequency domain.

[0023] Discrete Cosine Transform (DCT): DCT is a real number transform commonly used for signal and image compression. It converts the signal into the frequency domain and concentrates the energy in the low-frequency part, which helps data compression.

[0024] Spectral Subtraction (SS): Spectral subtraction is a speech enhancement technique that improves speech quality by estimating the spectrum of noise and subtracting the noise spectrum from the spectrum of noisy speech.

[0025] Wiener Filtering (WF): Wiener filtering is a linear filtering technique used to recover the original signal from a noisy signal. It finds the optimal filter coefficient by minimizing the mean square value of the error and is widely used in signal processing and image processing.

[0026] Kalman Filtering (KF): Kalman filtering is a recursive filter used to estimate the state of a linear dynamic system. It uses the dynamic model of the system and observed data to estimate the system state and minimize the estimation error in the presence of noise.

[0027] Based on this, the embodiments of the present application provide a speech generation method and device, an electronic device and a storage medium based on a network model, which can avoid signal aliasing and improve the sound quality of the generated speech.

[0028] The network model-based speech generation method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the network model-based speech generation method in the embodiments of the present application is described.

[0029] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0030] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. Artificial intelligence software technologies mainly include computer vision technology, robotics technology, biometrics technology, module management technology of online meeting systems, natural language processing technology, and machine learning / deep learning.

[0031] The network model-based speech generation method provided in the embodiment of the present application relates to the field of data processing technology. The network model-based speech generation method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a network model-based speech generation method, etc., but is not limited to the above forms.

[0032] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0033] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0034] The development of speech generation models has a long history. Current speech generation models capture long-distance dependencies by autoregressively modeling speech waveforms in the time domain and using dilated convolutions to expand the receptive field. However, with the advancement of technology, existing models based on upsampling with transposed convolutions have also exposed some shortcomings.

[0035] The current model based on transposed convolution for upsampling will cause signal aliasing, which is a common problem in the process of signal sampling and reconstruction, causing errors and distortion. The main reason for aliasing is that the signal sampling frequency is not enough to meet the Nyquist-Shannon sampling theorem, that is, the sampling frequency should be at least twice the highest frequency of the signal. If the sampling frequency is lower than this critical value, frequency aliasing will occur. Aliasing will cause high-frequency components to be incorrectly mapped to low-frequency areas, causing signal distortion. This phenomenon will lead to a decrease in sound quality and unclear pronunciation in speech generation, seriously affecting the naturalness and intelligibility of speech.

[0036] In order to solve the above problems, the present embodiment provides a speech generation method, device, electronic device and storage medium based on a network model. First, a speech signal to be processed is obtained. Since the speech signal to be processed is non-stationary, the present embodiment of the application will perform a frame operation on the speech signal to be processed to obtain multiple speech frames, which can divide the continuous signal processing into multiple independent frames, thereby improving the accuracy of subsequent spectrum analysis. For each speech frame, the network model is used to extract features of the speech frame, so that the energy distribution of the speech frame at different time points and frequencies can be determined, and the spectral features can be obtained. It can capture the dynamic changes and rhythm information of the speech and determine the spectral features corresponding to the speech. feature dimension, then, the spectral feature is input into the convolution block, so that the convolution block performs feature transformation on the spectral feature based on the feature dimension, thereby ensuring that the feature dimension of the spectral feature of each input convolution block is consistent with the output dimension, further ensuring the time resolution of the input spectral feature, and outputting the short-time Fourier transform coefficient, so that the intensity and energy distribution of the speech signal to be processed at different frequencies can be obtained, and at the same time, the phase information of each frequency component in the speech signal to be processed can be obtained, and finally, the short-time Fourier transform coefficient is inversely Fourier transformed to obtain the target speech waveform, thereby avoiding unclear speech due to signal aliasing and improving the sound quality of the generated speech. The embodiment of the present application does not perform transformation in the time dimension, but restores the waveform by calculating the short-time Fourier transform coefficient of the input feature, which reduces a large number of upsampling calculations in the current waveform processing process and also reduces the parameters of the entire model. And it avoids the aliasing phenomenon that may occur during the upsampling process.

[0037] The following is a detailed description with reference to the accompanying drawings.

[0038] Figure 1 is an optional flow chart of a network model-based speech generation method provided in an embodiment of the present application. Figure 1The method may include but is not limited to steps S101 to S105.

[0039] It should be noted that the network model includes multiple convolution blocks.

[0040] Step S101, obtaining a speech signal to be processed.

[0041] In step S101 of some embodiments, a voice signal to be processed is obtained, wherein the voice signal to be processed may be a voice signal in a financial scenario, a voice signal in a medical scenario, etc., and the embodiments of the present application do not impose specific limitations.

[0042] Step S102, performing a frame operation on the speech signal to be processed to obtain a plurality of speech frames.

[0043] In step S102 of some embodiments, the speech signal to be processed is framed, so that the continuous speech signal can be divided into multiple speech frames, the discontinuity at the frame boundary is reduced, the edge of the frame is smoothed, and multiple speech frames are obtained, laying the foundation for subsequent speech processing tasks.

[0044] Step S103: For each speech frame, extract features of the speech frame through the network model to obtain spectral features, and determine the feature dimensions corresponding to the spectral features.

[0045] In step S103 of some embodiments, for each speech frame, feature extraction is performed on the speech frame through the network model, so that the energy distribution of the speech signal at different time points and frequencies can be obtained, which is convenient for subsequent capture of dynamic changes in the speech. In addition, the data volume of the original speech signal is usually large. Feature extraction can reduce the dimension of the data, extract the most useful information, obtain spectral features, and determine the feature dimensions corresponding to the spectral features, thereby reducing the computational complexity of subsequent processing. No additional upsampling operations are performed, so that the features of each input convolution block are consistent with the output feature dimensions, and the influence of noise can be further removed.

[0046] Step S104: input the spectral features into a convolution block, so that the convolution block performs feature transformation on the spectral features based on the feature dimension and outputs short-time Fourier transform coefficients.

[0047] In step S104 of some embodiments, the spectral features are input into the convolution block so that the convolution block performs feature transformation on the spectral features based on the feature dimension, thereby obtaining the intensity or energy distribution of the speech signal to be processed at different frequencies and the phase information of each frequency component in the signal, outputting short-time Fourier transform coefficients, and improving the quality of the speech signal.

[0048] Step S105, performing inverse Fourier transform on the short-time Fourier transform coefficients to obtain a target speech waveform.

[0049] In step S105 of some embodiments, the short-time Fourier transform coefficients are inversely transformed, that is, all reconstructed frames are superimposed to form a complete speech signal to obtain a target speech waveform. The embodiment of the present application uses an inverse short-time Fourier transform to restore the speech waveform through the amplitude spectrum and phase spectrum in the short-time Fourier transform coefficients, thereby reducing a large number of intermediate upsampling calculations and reducing the parameters of the entire model, while avoiding the aliasing phenomenon that may occur during the upsampling process.

[0050] See also Figure 2 In some embodiments, step S104 may also include but is not limited to steps S201 to S203.

[0051] Step S201: input the spectrum feature into the convolution block, so that the convolution block performs feature transformation on the spectrum feature based on the feature dimension and outputs the convolution feature.

[0052] Step S202, performing feature segmentation on the convolution feature through a preset convolution layer, and outputting an amplitude spectrum and a phase spectrum.

[0053] Step S203: generating short-time Fourier transform coefficients according to the amplitude spectrum and the phase spectrum.

[0054] In steps S201 to S203 of some embodiments, in the process of the convolution block performing feature transformation on the spectral feature based on the feature dimension, first, the spectral feature is input into the convolution block, so that the convolution block performs feature transformation on the spectral feature based on the feature dimension, and outputs the convolution feature, so that more complex speech features can be captured, so that the features of each input convolution block are consistent with the output feature dimension, and the influence of noise can be further removed, which is helpful to extract the most useful features, eliminate unimportant information, and make the features more prominent. The convolution feature is then feature segmented through a preset convolution layer, that is, the convolution feature is divided into two through the convolution layer, and the convolution feature is divided into an amplitude spectrum and a phase spectrum, and the amplitude spectrum and the phase spectrum are output, so that the intensity or energy distribution of the speech signal to be processed at different frequencies and the phase information of each frequency component in the signal can be obtained. Finally, the short-time Fourier transform coefficients are generated according to the amplitude spectrum and the phase spectrum, so that the frequency characteristics of the speech signal can be analyzed according to the amplitude spectrum and the phase spectrum, and the quality of the speech signal can be improved.

[0055] See also Figure 3 In some embodiments, step S201 may also include but is not limited to steps S301 to S303.

[0056] It should be noted that the convolution block includes a separation convolution layer, a convolution layer and an aggregation convolution layer.

[0057] Step S301: Input the spectral features into the convolution block to perform a separation convolution operation on the convolution features through a separation convolution layer to obtain a first feature map.

[0058] Step S302: Perform a feature capture operation on the first feature map through a convolutional layer to obtain a second feature map.

[0059] Step S303: Perform feature aggregation on the first feature map and the second feature map through an aggregate convolution layer, and output a convolution feature with the same feature dimension.

[0060] In steps S301 to S303 of some embodiments, in the process of the convolution block performing feature transformation on the spectral feature based on the feature dimension and outputting the convolution feature, the embodiment of the present application first inputs the spectral feature into the convolution block to perform a separate convolution operation on the convolution feature through a separate convolution layer. Specifically, the local features in the speech signal are extracted through the separate convolution layer, thereby reducing the computational complexity and the number of parameters, and obtaining a first feature map. The features of the speech signal at different scales and directions can be obtained through the first feature map. Then, the first feature map is subjected to a feature capture operation through the convolution layer to enhance the network's representation capability of the speech signal, and obtain a second feature map, thereby being able to capture more complex speech features and obtain a higher-level abstract representation of the spectral feature. The first feature map and the second feature map are then subjected to feature aggregation through an aggregate convolution layer, thereby being able to integrate the information in the feature map, reduce the feature dimension, and output a convolution feature with the same feature dimension, thereby ensuring the temporal resolution of the input feature, and no additional upsampling operation is performed, so that the features of each input convolution block are consistent with the output feature dimension, and the influence of noise can be further removed, which helps to extract the most useful features, eliminate unimportant information, and make the features more prominent.

[0061] See also Figure 4 In some embodiments, step S102 may also include but is not limited to steps S401 to S402.

[0062] Step S401 : performing a frame division operation on a speech signal to be processed according to a preset frame length and a preset frame shift to obtain a plurality of segmented speech frames.

[0063] Step S402: for each segmented speech frame, a windowing operation is performed on the frame boundary of the segmented speech frame by using a preset window function to obtain multiple speech frames.

[0064] In steps S401 to S402 of some embodiments, since the speech signal is non-stationary, it means that its statistical characteristics change over time, for example, frequency and amplitude. In order to clarify the changing relationship between speech frames, in the embodiment of the present application, in the process of performing a framing operation on the speech signal to be processed, the framing operation is first performed on the speech signal to be processed according to a preset frame length and a preset frame shift to obtain a plurality of segmented speech frames, so that the continuous speech signal can be segmented into a plurality of speech frames, and for each segmented speech frame, a windowing operation is performed on the frame boundary of the segmented speech frame by a preset window function to obtain a plurality of speech frames, so that the discontinuity at the frame boundary can be reduced, the edge of the frame can be smoothed, and the spectrum leakage can be reduced.

[0065] It should be noted that the frame length and frame shift in the embodiment of the present application can be set according to the needs of the user, wherein the frame length depends on the sampling rate and the required time resolution. Specifically, the frame length can be represented by the number of sampling points. For example, for a sampling rate of 16kHz, a frame length of 20ms is equal to 320 sampling points. The frame shift is used to characterize the degree of overlap between frames, and the frame shift in the embodiment of the present application is less than the frame length, thereby increasing the time resolution and reducing the time ambiguity.

[0066] It is worth noting that in order to reduce the discontinuity at the frame boundary, in the process of performing a frame division operation on the speech signal to be processed according to a preset frame length and a preset frame shift, the embodiment of the present application will multiply each frame by a window function, such as a Hanning window, a Hamming window or a rectangular window, etc. Then the speech signal is divided into a series of continuous frames, specifically, the first frame starts from the starting position of the signal, and the last frame ends at the end position of the signal. Each time the frame shift distance is moved forward by a part of the speech signal, a new frame is extracted, and a plurality of segmented speech frames are obtained. For each segmented speech frame, a windowing operation is performed on the frame boundary of the segmented speech frame by a preset window function, and the window function is applied to each frame to obtain a plurality of speech frames, wherein the window function can be symmetrical, such as a Hanning window or a Hamming window, or can be asymmetrical, such as a rectangular window, and the embodiment of the present application does not make specific restrictions.

[0067] In some embodiments, in the process of performing a frame operation on the speech signal to be processed according to a preset frame length and a preset frame shift, frame overlap may occur, that is, the starting point of a new frame may overlap with the starting point of a previous frame to a certain extent. In order to ensure that no information is lost in subsequent processing, the embodiment of the present application adds zero padding at the beginning and end of the signal to ensure that the last frame has enough data, thereby avoiding frame overlap.

[0068] In some embodiments, before performing the frame division operation on the processed speech signal according to the preset frame length and the preset frame shift, the embodiment of the present application will also perform endpoint detection on the processed speech signal, detect and cut off the non-speech segment in the speech signal, such as the silent segment, the noise segment, etc., so that the subsequent processing can focus on the speech segment containing effective information. In the process of performing endpoint detection on the processed speech signal, the embodiment of the present application will extract the characteristic information of each frame, such as the short-time energy, the zero-crossing rate, etc., so that the voiced sound in the processed speech signal can be detected by the short-time energy, and the unvoiced sound in the processed speech signal can be detected by the zero-crossing rate. After that, the characteristic information is endpoint detected by the preset double threshold method. Specifically, two high and low thresholds are set, the high threshold is used for preliminary detection of the speech segment, and the low threshold is used to determine the beginning and end of the speech segment. Finally, according to the result of the endpoint detection, the non-speech segment is identified and cut off from the speech signal, so that the useful speech part can be effectively separated from the speech signal, laying the foundation for the subsequent speech processing task.

[0069] It can be understood that the short-time energy in the embodiment of the present application is used to calculate the energy of each frame of the signal, and the zero-crossing rate is used to calculate the number of times the signal crosses zero in each frame.

[0070] See also Figure 5 , Figure 5 It is a specific flow chart of extracting features of speech frames through a network model provided by an embodiment of the present application. In some embodiments, the method includes but is not limited to steps S501 to S506.

[0071] Step S501: For each speech frame, Fourier transform is performed on the speech frame through a network model to obtain a spectrum representation.

[0072] Step S502: combine all frequency spectrum representations to obtain a spectrum graph.

[0073] Step S503: extract coefficients from the spectrum graph to obtain Mel-frequency cepstrum coefficients.

[0074] Step S504: extract energy from the spectrum graph to obtain spectrum energy.

[0075] Step S505 , performing a weighted average operation on the spectrum representation according to the spectrum graph to obtain the spectrum centroid.

[0076] Step S506, obtaining spectrum features according to the Mel-frequency cepstral coefficients, spectrum energy and spectrum centroid.

[0077] In steps S501 to S506 of some embodiments, in the process of extracting features from speech frames through a network model, for each speech frame, a Fourier transform is performed on the speech frame through the network model to convert the time domain signal into a frequency domain signal to obtain a spectrum representation of each speech frame, so that the frequency components of the speech frame at different time points can be analyzed, and then all the spectrum representations are combined to obtain a spectrogram, and then coefficients are extracted from the spectrogram to calculate the Mel filter group energy of each speech frame. Specifically, the spectrum is mapped to a Mel frequency scale, and then the logarithm of the Mel filter group energy is taken, and a discrete cosine transform is applied to calculate the Mel frequency cepstral coefficients to obtain the Mel frequency cepstral coefficients; the spectrum is energy extracted. The spectral energy of each frame is extracted, and the spectral energy is calculated to obtain the spectral energy; the spectral representation is weighted averaged according to the spectrogram. Specifically, the spectral centroid is calculated by summing the product of the frequency and its amplitude spectrum and dividing it by the total energy. Finally, the spectral features are obtained according to the Mel-frequency cepstral coefficients, the spectral energy and the spectral centroid. Specifically, the Mel-frequency cepstral coefficients, the spectral energy and the spectral centroid are sorted in time series to obtain the spectral features, so as to obtain the energy distribution of the speech signal at different time points and frequencies, which is convenient for subsequent capture of the dynamic changes of the speech. In addition, the data volume of the original speech signal is usually large. Feature extraction can reduce the dimension of the data, extract the most useful information, and reduce the computational complexity of subsequent processing.

[0078] See also Figure 6 In some embodiments, step S105 may also include but is not limited to steps S601 to S602.

[0079] Step S601: for each speech frame's short-time Fourier transform coefficient, perform waveform restoration on the short-time Fourier transform coefficient by inverse short-time Fourier transform to obtain a signal segment.

[0080] Step S602: superimpose all signal segments to obtain a target speech waveform.

[0081] In steps S601 to S602 of some embodiments, in the process of performing inverse Fourier transform on the short-time Fourier transform coefficients, for the short-time Fourier transform coefficients of each speech frame, the waveform of the short-time Fourier transform coefficients is restored by inverse short-time Fourier transform, that is, the short-time Fourier transform coefficients of each speech frame are inversely transformed to reconstruct the time domain signal to obtain signal segments, and then all signal segments are superimposed, that is, all reconstructed frames are superimposed to form a complete speech signal to obtain a target speech waveform. The embodiment of the present application uses an inverse short-time Fourier transform to recover the speech waveform through the amplitude spectrum and phase spectrum in the short-time Fourier transform coefficients, thereby reducing a large number of intermediate upsampling calculations, reducing the parameters of the entire model, and avoiding the aliasing phenomenon that may occur during the upsampling process.

[0082] See also Figure 7 , Figure 7 It is a specific flow chart of a speech generation method based on a network model provided by another embodiment of the present application. In some embodiments, the method includes but is not limited to steps S701 to S702.

[0083] It should be noted that step S701 to step S702 occur before performing a frame operation on the speech signal to be processed to obtain a plurality of speech frames.

[0084] Step S701, performing a denoising operation on the speech signal to be processed, and performing speech enhancement on the speech signal to be processed after the denoising operation.

[0085] Step S702, normalizing the speech signal to be processed after speech enhancement.

[0086] In steps S701 to S702 of some embodiments, before the processed speech signal is divided into frames to obtain multiple speech frames, the embodiments of the present application will also perform a denoising operation on the processed speech signal. Specifically, the embodiments of the present application will estimate the characteristics of the background noise of the processed speech signal, and then use a filtering algorithm, such as spectral subtraction, Wiener filtering, Kalman filtering, etc., to reduce or remove the noise component, and perform speech enhancement on the processed speech signal after the denoising operation, thereby improving the signal-to-noise ratio of the speech by highlighting the characteristics of the speech signal and suppressing the noise characteristics. Thereafter, the processed speech signal after the speech enhancement is normalized to adjust the dynamic range of the speech signal, so that the loudness of the speech is more uniform and the extreme changes in the volume are reduced.

[0087] Specifically, in the process of normalizing the speech signal to be processed after speech enhancement, the embodiment of the present application will perform mean normalization and amplitude normalization on the speech signal to be processed, wherein the mean normalization is to adjust the mean of the speech signal to zero to facilitate the subsequent feature extraction of the speech signal to be processed; the amplitude normalization is to scale the amplitude of the speech signal to a specific range, such as [-1,1] or [0,1], to prevent numerical overflow in subsequent processing.

[0088] See also Figure 8 The embodiment of the present application also provides a speech generation device based on a network model, wherein the network model includes a plurality of convolution blocks; the device includes: The signal acquisition module 801 is used to acquire the speech signal to be processed; A signal framing module 802, configured to perform a framing operation on the speech signal to be processed to obtain a plurality of speech frames; The feature extraction module 803 is used to extract features of each speech frame through the network model to obtain spectrum features and determine the feature dimensions corresponding to the spectrum features; A feature transformation module 804 is used to input the spectrum feature into the convolution block, so that the convolution block performs feature transformation on the spectrum feature based on the feature dimension and outputs a short-time Fourier transform coefficient; The inverse Fourier transform module 805 is used to perform an inverse Fourier transform on the short-time Fourier transform coefficients to obtain a target speech waveform.

[0089] In some embodiments, the network model-based speech generation device also includes a preprocessing module, which is used to perform denoising operations on the speech signal to be processed, perform speech enhancement on the speech signal to be processed after the denoising operation, and perform normalization processing on the speech signal to be processed after the speech enhancement.

[0090] The specific implementation of the network model-based speech generation device is basically the same as the specific implementation of the network model-based speech generation method described above, and will not be repeated here.

[0091] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned network model-based speech generation method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0092] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes: The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the network model-based speech generation method of the embodiment of this application; Input / output interface 903, used to implement information input and output; Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.); A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904); The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0093] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned network model-based speech generation method is implemented.

[0094] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0095] The network model-based speech generation method, device, electronic device and storage medium provided in the embodiments of the present application first obtain a speech signal to be processed. Since the speech signal to be processed is non-stationary, the embodiments of the present application perform a frame operation on the speech signal to be processed to obtain multiple speech frames, which can divide the continuous signal processing into multiple independent frames, thereby improving the accuracy of subsequent spectrum analysis. For each speech frame, the network model is used to extract features of the speech frame, so that the energy distribution of the speech frame at different time points and frequencies can be determined, and the spectrum feature can be obtained. It can capture the dynamic changes and rhythm information of the speech and determine the feature dimension corresponding to the spectrum feature. After that, the spectral features are input into the convolution block, so that the convolution block performs feature transformation on the spectral features based on the feature dimension, thereby ensuring that the feature dimension of the spectral features of each input convolution block is consistent with the output dimension, further ensuring the time resolution of the input spectral features, and outputting the short-time Fourier transform coefficients, so that the intensity and energy distribution of the speech signal to be processed at different frequencies can be obtained, and at the same time, the phase information of each frequency component in the speech signal to be processed can be obtained. Finally, the short-time Fourier transform coefficients are inversely Fourier transformed to obtain the target speech waveform, thereby avoiding unclear speech due to signal aliasing and improving the sound quality of the generated speech. The embodiment of the present application does not perform transformation in the time dimension, but restores the waveform by calculating the short-time Fourier transform coefficients of the input features, which reduces a large number of upsampling calculations in the current waveform processing process and also reduces the parameters of the entire model. And it avoids the aliasing phenomenon that may occur during the upsampling process.

[0096] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0097] It can be understood by those skilled in the art that Figure 1-9 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0098] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0099] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0100] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0101] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0102] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0103] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0104] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0106] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A speech generation method based on a network model, characterized in that: The network model includes a plurality of convolution blocks; the method includes: Obtaining a voice signal to be processed; Performing a frame operation on the speech signal to be processed to obtain a plurality of speech frames; For each of the speech frames, extract features of the speech frame through a network model to obtain spectral features, and determine feature dimensions corresponding to the spectral features; Inputting the spectral feature into the convolution block, so that the convolution block performs feature transformation on the spectral feature based on the feature dimension and outputs a short-time Fourier transform coefficient; The short-time Fourier transform coefficients are subjected to inverse Fourier transform to obtain a target speech waveform.

2. The method for generating speech based on a network model according to claim 1, characterized in that: The step of inputting the spectral feature into the convolution block so that the convolution block performs feature transformation on the spectral feature based on the feature dimension and outputs a short-time Fourier transform coefficient comprises: Inputting the spectral feature into the convolution block, so that the convolution block performs feature transformation on the spectral feature based on the feature dimension and outputs a convolution feature; Performing feature segmentation on the convolution feature through a preset convolution layer, and outputting an amplitude spectrum and a phase spectrum; Short-time Fourier transform coefficients are generated based on the magnitude spectrum and the phase spectrum.

3. The method for generating speech based on a network model according to claim 2, characterized in that: The convolution block includes a separation convolution layer, a convolution layer and an aggregation convolution layer; the spectral feature is input into the convolution block so that the convolution block performs feature transformation on the spectral feature based on the feature dimension and outputs the convolution feature, including: Inputting the spectral feature into the convolution block to perform a separation convolution operation on the convolution feature through the separation convolution layer to obtain a first feature map; Performing a feature capture operation on the first feature map through the convolution layer to obtain a second feature map; The first feature map and the second feature map are subjected to feature aggregation through the aggregation convolution layer, and convolution features having the same feature dimension as the feature are output.

4. The method for generating speech based on a network model according to claim 1, characterized in that: The step of performing a frame operation on the speech signal to be processed to obtain a plurality of speech frames comprises: Performing a frame division operation on the speech signal to be processed according to a preset frame length and a preset frame shift to obtain a plurality of segmented speech frames; For each of the segmented speech frames, a windowing operation is performed on the frame boundary of the segmented speech frame by using a preset window function to obtain multiple speech frames.

5. The method for generating speech based on a network model according to claim 1, characterized in that: The extracting features of the speech frame by using a network model to obtain spectral features includes: For each of the speech frames, Fourier transform is performed on the speech frame through a network model to obtain a spectrum representation; Combining all the frequency spectrum representations to obtain a frequency spectrum diagram; Extracting coefficients from the spectrum to obtain Mel-frequency cepstrum coefficients; Performing energy extraction on the spectrum graph to obtain spectrum energy; Performing a weighted average operation on the spectrum representation according to the spectrum graph to obtain a spectrum centroid; A spectrum feature is obtained according to the Mel-frequency cepstral coefficients, the spectrum energy and the spectrum centroid.

6. The method for generating speech based on a network model according to claim 1, characterized in that: The inverse Fourier transform of the short-time Fourier transform coefficients to obtain a target speech waveform includes: For each short-time Fourier transform coefficient of the speech frame, waveform recovery is performed on the short-time Fourier transform coefficient by inverse short-time Fourier transform to obtain a signal segment; All the signal segments are superimposed to obtain a target speech waveform.

7. The method for generating speech based on a network model according to claim 1, characterized in that: Before performing a frame operation on the speech signal to be processed to obtain a plurality of speech frames, the method further includes: Performing a denoising operation on the speech signal to be processed, and performing speech enhancement on the speech signal to be processed after the denoising operation; The speech signal to be processed after speech enhancement is normalized.

8. A speech generation device based on a network model, characterized in that: The network model includes a plurality of convolution blocks; the device includes: A signal acquisition module, used to acquire a speech signal to be processed; A signal framing module, used for performing a framing operation on the speech signal to be processed to obtain a plurality of speech frames; A feature extraction module, used for extracting features of each speech frame through a network model to obtain spectral features, and determining feature dimensions corresponding to the spectral features; A feature transformation module, used for inputting the spectral feature into the convolution block, so that the convolution block performs feature transformation on the spectral feature based on the feature dimension and outputs a short-time Fourier transform coefficient; The inverse Fourier transform module is used to perform an inverse Fourier transform on the short-time Fourier transform coefficients to obtain a target speech waveform.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the network model-based speech generation method described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the network model-based speech generation method according to any one of claims 1 to 7 is implemented.

Images