A speech enhancement method, device, terminal equipment and storage medium

Abstract

This invention discloses a speech enhancement method, apparatus, terminal device, and storage medium. The method includes: acquiring a plurality of speech data to be enhanced; performing windowing processing and short-time Fourier transform on each speech data to be enhanced to generate a plurality of first speech data; performing complex encoding on each of the first speech data through a preset forward convolutional network to generate a plurality of second speech data; performing prediction on each of the second speech data through a long short-term memory network to generate a plurality of third speech data; performing complex decoding on each of the third speech data through a preset transposed convolutional network to generate a plurality of fourth speech data; and sequentially performing masking processing, short-time inverse Fourier transform, and windowing processing on each of the fourth speech data to generate a plurality of enhanced speech data. By implementing this invention, the effect of enhancing speech data can be achieved.

Description

Technical Field

[0001] This invention relates to the field of speech enhancement, and more particularly to a speech enhancement method, apparatus, terminal device, and storage medium. Background Technology

[0002] Voice communication is a way of communicating through voice and transmission medium. It is one of the most convenient and natural ways for humans to communicate. Nowadays, people often use some communication devices to communicate by voice. Voice can be converted into voice data and transmitted through communication devices. However, voice data is usually affected by noise in the environment during transmission. Therefore, it is particularly important to study effective voice enhancement technology. Summary of the Invention

[0003] This invention provides a speech enhancement method, apparatus, terminal device, and storage medium, which can enhance speech data.

[0004] This invention provides a speech enhancement method, apparatus, terminal device, and storage medium, comprising: acquiring a plurality of speech data to be enhanced, performing windowing processing on each of the speech data to be enhanced, and performing short-time Fourier transform on each of the windowed speech data to be enhanced to generate a plurality of first speech data;

[0005] Each of the first speech data is input into a preset forward convolutional network, so that the forward convolutional network performs complex encoding on each of the first speech data to generate a plurality of second speech data.

[0006] Each of the second speech data is input into a long short-term memory network, which predicts the current speech data based on historical speech data to generate several third speech data.

[0007] Each of the third speech data is input into a preset transposed convolutional network, so that the transposed convolutional network performs complex decoding on each of the third speech data to generate several fourth speech data.

[0008] The fourth speech data is masked, and the masked fourth speech data is then subjected to short-time inverse Fourier transform and windowing to generate several enhanced speech data.

[0009] Furthermore, the preset forward convolutional network includes: four levels of forward convolutional layers; each of the forward convolutional layers includes: a two-dimensional forward convolutional layer, a normalization layer, and an activation function layer;

[0010] The forward convolutional network performs complex encoding on each of the first speech data to generate several second speech data, including:

[0011] Each level of forward convolutional layer performs two-dimensional forward convolution processing on each of the first speech data to generate several second speech data.

[0012] In the process of generating the second speech data, the output of the previous forward convolutional layer serves as the input of the next forward convolutional layer. When each forward convolutional layer performs two-dimensional forward convolution processing on each of the first speech data, for the first-level forward convolutional layer, the input of the two-dimensional forward convolutional layer is two frames of first speech data output after Fourier transform. For the second to fourth-level forward convolutional layers, the input of the two-dimensional forward convolutional layer is one frame of speech data currently output from the previous level and one frame of speech data previously output from the previous level, and the output is one frame of speech data. The speech data output by the two-dimensional forward convolutional layer within each forward convolutional layer sequentially passes through a first normalization layer and a first activation function layer. The first normalization layer is used to normalize the speech data, and the first activation function layer is used to activate the normalized speech data.

[0013] Furthermore, there are four levels of transposed convolutional layers; each of the transposed convolutional layers includes: a two-dimensional transposed convolutional layer, a normalization layer, and an activation function layer;

[0014] The transposed convolutional network performs complex decoding on each of the third speech data to generate several fourth speech data, including:

[0015] In the process of generating the fourth speech data, the output of the previous transposed convolutional layer serves as the input of the next transposed convolutional layer. When each two-dimensional convolutional layer performs two-dimensional transposed convolution processing on each of the third speech data, for the first-level transposed convolutional layer, the input of the two-dimensional transposed convolutional layer is two frames of third speech data output after Fourier transform. For the second to fourth-level transposed convolutional layers, the input of the two-dimensional transposed convolutional layer is one frame of speech data currently output from the previous level and one frame of speech data previously output from the previous level, and the output is one frame of speech data. The speech data output by the two-dimensional transposed convolutional layer within each transposed convolutional layer sequentially passes through a second normalization layer and a second activation function layer. The second normalization layer is used to normalize the speech data, and the second activation function layer is used to perform activation function processing on the normalized speech data.

[0016] Furthermore, when each of the two-dimensional forward convolutional layers acquires the input speech data, the input speech data is subjected to depthwise convolution or pointwise convolution.

[0017] When performing depthwise convolution, the kernel height and width are calculated in a loop, and the results of the loop calculation are used to calculate the number of layers and the output height layer in a loop. When performing pointwise convolution, the output channel layer, the output height layer and the input channel layer are calculated in a loop.

[0018] Furthermore, when each of the two-dimensional transposed convolutional layers acquires the input speech data, it performs depthwise transposed convolution or pointwise transposed convolution on the input speech data.

[0019] When performing depthwise transposed convolution, the preset input width layer, kernel height layer, and kernel width layer are cyclically unwound and calculated. Based on the results of the unwound cyclic calculation, the preset number of group layers and input height layer are cyclically calculated. When performing pointwise transposed convolution, the preset input height layer, input channel layer, and output channel layer are cyclically calculated.

[0020] Furthermore, the long short-term memory network includes: a first long short-term memory layer, a second long short-term memory layer, and a fully connected layer;

[0021] Each of the second speech data is input into a Long Short-Term Memory (LSTM) network, so that the LSM network predicts the current speech data based on historical speech data, generating several third speech data, including:

[0022] Each of the second speech data is input into the first long short-term memory layer, and the first long short-term memory layer makes a first prediction on each of the second speech data based on historical speech data;

[0023] The second speech data after the first prediction are input into the second long short-term memory layer, and the second prediction is performed through the second long short-term memory layer;

[0024] The second speech data after each prediction is input into the fully connected layer. The fully connected layer aggregates the information of the second speech data after each prediction to generate several third speech data.

[0025] Furthermore, the generation of the plurality of speech data to be enhanced includes: sampling the input speech in frames according to a preset frequency to generate a plurality of speech data to be enhanced.

[0026] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments;

[0027] The present invention provides a speech enhancement device, comprising: a short-time Fourier transform module, a complex encoding module, a long short-term memory module, a complex decoding module, and an enhanced speech data generation module.

[0028] The short-time Fourier transform module is used to acquire several speech data to be enhanced, perform windowing processing on each speech data to be enhanced, and perform short-time Fourier transform on each speech data to be enhanced after windowing processing to generate several first speech data.

[0029] The complex encoding module is used to input each of the first speech data into a preset forward convolutional network, so that the forward convolutional network performs complex encoding on each of the first speech data to generate a plurality of second speech data.

[0030] The long short-term memory module is used to input each of the second speech data into the long short-term memory network. The long short-term memory network predicts the current speech data based on historical speech data and generates several third speech data.

[0031] The complex decoding module is used to input each of the third speech data into a preset transposed convolutional network, so that the transposed convolutional network performs complex decoding on each of the third speech data to generate several fourth speech data.

[0032] The enhanced speech data generation module is used to perform masking processing on each of the fourth speech data, and to perform short-time inverse Fourier transform and windowing processing on each of the masked fourth speech data to generate several enhanced speech data.

[0033] Based on the above method embodiments, the present invention provides a corresponding terminal device embodiment;

[0034] The present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the speech enhancement method according to any one of the present invention.

[0035] Based on the above method embodiments, the present invention provides a corresponding storage medium embodiment;

[0036] The present invention provides a storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the storage medium is located to execute the speech enhancement method according to any one of the present invention.

[0037] The embodiments of the present invention have the following beneficial effects:

[0038] This invention provides a speech enhancement method, apparatus, terminal device, and storage medium. The method involves acquiring several speech data sets to be enhanced, performing windowing and short-time Fourier transform to generate several first speech data sets; encoding each first speech data set through a forward convolutional network to generate several second speech data sets; performing long short-term memory processing on each second speech data set to generate several third speech data sets; decoding each third speech data set through a transposed convolutional network to generate several fourth speech data sets; and sequentially performing masking, inverse short-time Fourier transform, and windowing processing on each fourth speech data set to generate several enhanced speech data sets. By implementing this invention, the effect of enhancing speech data can be achieved. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of a voice method provided in an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of a forward convolutional network structure provided in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of a long short-term memory network structure provided in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of a transposed convolutional network structure provided in an embodiment of the present invention;

[0043] Figure 5 This is a schematic diagram of a speech enhancement method provided in an embodiment of the present invention;

[0044] Figure 6 This is a schematic diagram of the structure of a speech enhancement device provided in an embodiment of the present invention. Detailed Implementation

[0045] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0046] like Figure 1 As shown, one embodiment provides a speech enhancement method, including:

[0047] Step S101: Obtain several speech data to be enhanced, perform windowing processing on each speech data to be enhanced, and perform short-time Fourier transform on each speech data to be enhanced after windowing processing to generate several first speech data.

[0048] Step S102: Input each of the first speech data into a preset forward convolutional network, so that the forward convolutional network performs complex encoding on each of the first speech data to generate a plurality of second speech data;

[0049] Step S103: Input each of the second speech data into the Long Short-Term Memory Network, wherein the Long Short-Term Memory Network predicts the current speech data based on historical speech data and generates several third speech data;

[0050] Step S104: Input each of the third speech data into a preset transposed convolutional network, so that the transposed convolutional network performs complex decoding on each of the third speech data to generate several fourth speech data;

[0051] Step S105: Perform masking processing on each of the fourth speech data, and perform short-time inverse Fourier transform and windowing processing on each of the masked fourth speech data to generate several enhanced speech data.

[0052] For step S101, in a preferred embodiment, after acquiring several speech data to be enhanced, a Hanning window is added to each speech data to be enhanced to prevent spectrum leakage. After windowing, a 512-point Fourier transform is performed to convert the speech data from the time domain to the frequency domain, and finally several first speech data are generated. The frequency domain signal is a complex number, including the amplitude of the real part and the phase of the imaginary part.

[0053] In an optional embodiment, the generation of the plurality of speech data to be enhanced includes: sampling the input speech in frames according to a preset frequency to generate a plurality of speech data to be enhanced;

[0054] Specifically, after acquiring the input speech, the input speech is first initialized, which mainly includes dividing the speech into frames. Each frame has a length of 128 sampling points, which corresponds to 8ms for a 16k sampling rate speech. After sampling, several first speech data are finally generated.

[0055] In a preferred embodiment, for step S102, each first speech data is input into a preset forward convolutional network, that is, the amplitude and phase of each first speech data pass through a forward convolutional network at the same time, and the first speech data is encoded by the forward convolutional network to generate a number of second speech data.

[0056] In an optional embodiment, the preset forward convolutional network includes: four levels of forward convolutional layers; each of the forward convolutional layers includes: a two-dimensional forward convolutional layer, a normalization layer, and an activation function layer;

[0057] The forward convolutional network performs complex encoding on each of the first speech data to generate several second speech data, including:

[0058] Each level of forward convolutional layer performs two-dimensional forward convolution processing on each of the first speech data to generate several second speech data.

[0059] In the process of generating the second speech data, the output of the previous forward convolutional layer serves as the input of the next forward convolutional layer. When each forward convolutional layer performs two-dimensional forward convolution processing on each of the first speech data, for the first-level forward convolutional layer, the input of the two-dimensional forward convolutional layer is two frames of first speech data output after Fourier transform. For the second to fourth-level forward convolutional layers, the input of the two-dimensional forward convolutional layer is one frame of speech data currently output from the previous level and one frame of speech data previously output from the previous level, and the output is one frame of speech data. The speech data output by the two-dimensional forward convolutional layer within each forward convolutional layer sequentially passes through a first normalization layer and a first activation function layer. The first normalization layer is used to normalize the speech data, and the first activation function layer is used to activate the normalized speech data.

[0060] Specifically, such as Figure 2 As shown, the system includes four levels of forward convolutional layers. For each two-dimensional forward convolutional layer, the kernel size is 5*2. To reduce data processing latency, the number of input and output frames per frame is minimized, which effectively reduces the memory usage of the buffer. For a kernel size of 5*2, the kernel width is 2, so the minimum input is two frames. For the first level of forward convolutional layers, the input of the two-dimensional forward convolutional layer is the two frames of the first speech data output after Fourier transform. For the second to fourth levels of forward convolutional layers, the input of the two-dimensional forward convolutional layer is the current output frame of speech data from the previous level, and the previous output frame of speech data from the previous level.

[0061] For each forward convolutional layer, after performing forward convolution on the speech data in the two-dimensional forward convolutional layer, the speech data is then normalized through a normalization layer to improve the algorithm accuracy and speed up the convergence. After normalization, the speech data is finally processed by the pReLU activation function through an activation function layer to generate the speech data output by each forward convolutional layer.

[0062] In an optional embodiment, when each of the two-dimensional transposed convolutional layers acquires the input speech data, it performs a depthwise transposed convolution or a pointwise transposed convolution on the input speech data.

[0063] When performing depthwise transposed convolution, the preset input width layer, kernel height layer, and kernel width layer are cyclically unwound and calculated. Based on the results of the unwound cyclic calculation, the preset number of group layers and input height layer are cyclically calculated. When performing pointwise transposed convolution, the preset input height layer, input channel layer, and output channel layer are cyclically calculated.

[0064] Specifically, a typical forward convolution (separable convolution is a special type of forward convolution) consists of 8 loops. The size of each loop is batch, group, output channel, output height, output weight, input channel, kernel height, and kernel weight, respectively. The batch size is usually 1. The dataset is divided into several batches only when it is impossible to pass the data through the neural network all at once.

[0065] For depthwise convolutions, since the batch size, output channel size, output weight size, and input channel size are all 1, the corresponding first, third, fifth, and sixth layers can be removed. The kernel height and kernel weight sizes are 5 and 2 respectively, which are small values ​​and can be directly expanded into loops for calculation. Ultimately, only two loops remain: one for the group size and one for the input height, corresponding to the group size and the other for the output height. It's worth noting that the layer with a size of 1 can be removed because this layer only has one element, eliminating the need to iterate through all elements.

[0066] For pointwise convolution, since the batch (number of parts), group (number of groups), output_weight (output width), kernel_height (kernel height), and kernel_weight (kernel width) are all 1, the corresponding first, second, fifth, seventh, and eighth layers can be removed. Only the output_channel (number of output channels), output_height (output height), and input_channel (number of input channels) correspond to the three loop layers of output channel layer, output height layer, and input channel layer, respectively.

[0067] For step S103, in a preferred embodiment, each second speech data is input into a long short-term memory network. Since speech is a serialized data, the long short-term memory network can predict the current frame data based on historical data and generate several third speech data.

[0068] In an optional embodiment, the long short-term memory network includes: a first long short-term memory layer, a second long short-term memory layer, and a fully connected layer;

[0069] Each of the second speech data is input into a Long Short-Term Memory (LSTM) network, so that the LSM network predicts the current speech data based on historical speech data, generating several third speech data, including:

[0070] Each of the second speech data is input into the first long short-term memory layer, and the first long short-term memory layer makes a first prediction on each of the second speech data based on historical speech data;

[0071] The second speech data after the first prediction are input into the second long short-term memory layer, and the second prediction is performed through the second long short-term memory layer;

[0072] The second speech data after each prediction is input into the fully connected layer. The fully connected layer aggregates the information of the second speech data after each prediction to generate several third speech data.

[0073] Indicative, such as Figure 3 As shown, the input to the first layer of the LSTM (Long Short-Term Memory) network is the encoded output, with a data length of 1024 and a weight size of 128*4*1024, plus the output of the previous frame's LSTM_L1 (first Long Short-Term Memory layer), with a length of 128. The output of the first LSTM layer is similar to that of a traditional LSTM, in complex form, with an output length of 128. The input to the second LSTM layer is the output of the first LSTM layer, with a data length of 128 and a weight size of 128*4*128, plus the output of the previous frame's LSTM_L2 (second Long Short-Term Memory layer), with a length of 128. The output of the second LSTM layer is similar to that of a traditional LSTM, in complex form, with an output length of 128. The input to the fully connected layer is the output of LSTM_L2. Each output data point is connected to each input data point through a weight network. Generally, fully connected layers are mainly used for classification or regression; the fully connected layer after LSTM is mainly used for regression, allowing the model to fit better. The complete formula is shown below:

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080] in, This represents the input gate. This represents the Sigmoid activation function; This represents the weight parameter. This represents the bias parameter. This represents a hidden state. This indicates the updated cell state. This indicates the Forget gate. Indicates cell state; This represents an intermediate variable used to calculate c, which is used to calculate information at the current time. This represents the updated hidden state. Tanh represents the output gate; Tanh represents the Tanh activation function.

[0081] The role of LSTM is to ensure that the current state can effectively learn useful information from the previous state and pass it on effectively. LSTM is a special type of RNN that, compared to ordinary RNNs, solves the problems of vanishing and exploding gradients in long sequences. Current speech enhancement models all use LSTM; without it, the performance will be poor. Its main role in speech enhancement is temporal modeling. If LSTM is not used, similar modules, such as GRU or TCN, can be used to achieve temporal modeling.

[0082] In a preferred embodiment, for step S104, each third speech data is input into a preset transposed convolutional network so that the transposed convolutional network performs complex decoding on each third speech data to generate several fourth speech data.

[0083] In an optional embodiment, the preset transposed convolutional network includes: four layers of transposed convolutional layers; each of the transposed convolutional layers includes: a two-dimensional transposed convolutional layer, a normalization layer, and an activation function layer.

[0084] The transposed convolutional network performs complex decoding on each of the third speech data to generate several fourth speech data, including:

[0085] In the process of generating the fourth speech data, the output of the previous transposed convolutional layer serves as the input of the next transposed convolutional layer. When each two-dimensional convolutional layer performs two-dimensional transposed convolution processing on each of the third speech data, for the first-level transposed convolutional layer, the input of the two-dimensional transposed convolutional layer is two frames of third speech data output after Fourier transform. For the second to fourth-level transposed convolutional layers, the input of the two-dimensional transposed convolutional layer is one frame of speech data currently output from the previous level and one frame of speech data previously output from the previous level, and the output is one frame of speech data. The speech data output by the two-dimensional transposed convolutional layer within each transposed convolutional layer sequentially passes through a second normalization layer and a second activation function layer. The second normalization layer is used to normalize the speech data, and the second activation function layer is used to perform activation function processing on the normalized speech data.

[0086] Specifically, such as Figure 4 As shown, the transposed convolutional layer consists of four levels. For each 2D transposed convolutional layer, the kernel size is 5*2. To reduce data processing latency, the number of input and output frames per frame is minimized, effectively reducing the memory usage of the buffer. Since the kernel size is 5*2 and the kernel width is 2, the minimum input is two frames. The output consists of three frames of speech data. Because the three output frames are identical, only one frame is actually used, so the other two frames can be removed without further processing. For the first level transposed convolutional layer, the input is the two frames of the third speech data output after Fourier transform. For the second to fourth levels, the input is the current output frame of speech data from the previous level and the previous output frame of speech data from the previous level.

[0087] For each transposed convolutional layer, after transposing the speech data in the two-dimensional transposed convolutional layer, the speech data is then normalized through a normalization layer to improve the algorithm's accuracy and accelerate convergence. After normalization, the pReLU activation function is applied through an activation function layer to introduce nonlinearity, delinearize the network, improve the neural network's ability to express the model, and solve problems that linear models cannot solve. The final generated speech data serves as the speech data output by the corresponding transposed convolutional layer.

[0088] In an optional embodiment, when each of the two-dimensional transposed convolutional layers acquires the input speech data, it performs a depthwise transposed convolution or a pointwise transposed convolution on the input speech data.

[0089] When performing depthwise transposed convolution, the preset input width layer, kernel height layer, and kernel width layer are cyclically unwound and calculated. Based on the results of the unwound cyclic calculation, the preset number of group layers and input height layer are cyclically calculated. When performing pointwise transposed convolution, the preset input height layer, input channel layer, and output channel layer are cyclically calculated.

[0090] Specifically, a typical transposed convolution (a separable transposed convolution is a special type of transposed convolution) also includes 8 loops, with the size of each loop being batch, group, input_height, input_weight, input_channel, output_channel, kernel_height, and kernel_weight.

[0091] For depthwise transposed convolutions, since the batch size, input channel size, and output channel size are all 1, the corresponding first, fifth, and sixth layers can be omitted. Because the input weight size is 2, the corresponding input width layer can be directly expanded. Since kernel height and kernel weight are 5 and 2 respectively, the corresponding kernel height and kernel width layers are directly expanded and looped for calculation. Ultimately, only the group size and input height corresponding to the group number and input height layers are looped. Furthermore, although output weight is 3 frames, only 1 frame is actually used; therefore, the other 2 frames do not need to be calculated when calculating the output, reducing computational load.

[0092] For pointwise transposed convolution, since the batch (number of parts), group (number of groups), input_weight (input width), kernel_height (kernel height), and kernel_weight (kernel width) are all 1, the corresponding first, second, fourth, seventh, and eighth layers can be removed. Only the input_height (input height), input_channel (number of input channels), and output_channel (number of output channels) layers are looped, forming the input height layer, input channel layer, and output channel layer, respectively. In step S105, each fourth speech data is masked, and the masked fourth speech data is subjected to short-time inverse Fourier transform and windowing (adding a synthesis window) to generate several enhanced speech data.

[0093] Specifically, the fourth speech data is masked using the following formula:

[0094]

[0095]

[0096]

[0097] in, This represents the phase mask. This represents the imaginary mask. This represents the real mask. This indicates the clean speech output after noise reduction; Indicates the magnitude of the speech before noise reduction. Indicates the magnitude mask; This indicates the phase of the speech before noise reduction.

[0098] The overall process of the above embodiments is illustrative as follows: Figure 5 As shown.

[0099] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments.

[0100] like Figure 6 As shown, an embodiment of the present invention provides a speech enhancement device, including: a short-time Fourier transform module, a complex encoding module, a long short-term memory module, a complex decoding module, and an enhanced speech data generation module.

[0101] The short-time Fourier transform module is used to acquire several speech data to be enhanced, perform windowing processing on each speech data to be enhanced, and perform short-time Fourier transform on each speech data to be enhanced after windowing processing to generate several first speech data.

[0102] The complex encoding module is used to input each of the first speech data into a preset forward convolutional network, so that the forward convolutional network performs complex encoding on each of the first speech data to generate a plurality of second speech data.

[0103] The long short-term memory module is used to input each of the second speech data into the long short-term memory network. The long short-term memory network predicts the current speech data based on historical speech data and generates several third speech data.

[0104] The complex decoding module is used to input each of the third speech data into a preset transposed convolutional network, so that the transposed convolutional network performs complex decoding on each of the third speech data to generate several fourth speech data.

[0105] The enhanced speech data generation module is used to perform masking processing on each of the fourth speech data, and to perform short-time inverse Fourier transform and windowing processing on each of the masked fourth speech data to generate several enhanced speech data.

[0106] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0107] Those skilled in the art will clearly understand that, for convenience and simplicity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0108] Based on the above method embodiments, the present invention provides corresponding terminal device embodiments.

[0109] Another embodiment of the present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor; when the processor executes the computer program, it implements the speech enhancement method of any embodiment of the present invention.

[0110] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the device.

[0111] The device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The device may include, but is not limited to, a processor and memory.

[0112] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the device, connecting various parts of the device via various interfaces and lines.

[0113] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0114] Based on the above method embodiments, the present invention provides corresponding storage medium embodiments.

[0115] Another embodiment of the present invention provides a storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the storage medium is located to execute the speech enhancement method of any embodiment of the present invention.

[0116] In this embodiment, the storage medium is a computer-readable storage medium, and the computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0117] By implementing the various embodiments of the present invention described above, it is possible to achieve the effect of enhancing voice data.

[0118] The above description represents the preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications are also considered to be protected by the present invention.

Claims

1. A speech enhancement method, characterized in that, include: Acquire several speech data to be enhanced, perform windowing processing on each speech data to be enhanced, and perform short-time Fourier transform on each speech data to be enhanced after windowing processing to generate several first speech data. Each of the first speech data is input into a preset forward convolutional network, so that the forward convolutional network performs complex encoding on each of the first speech data to generate a plurality of second speech data. Each of the second speech data is input into a long short-term memory network, which predicts the current speech data based on historical speech data to generate several third speech data. Each of the third speech data is input into a preset transposed convolutional network, so that the transposed convolutional network performs complex decoding on each of the third speech data to generate several fourth speech data. The fourth speech data is masked, and the masked fourth speech data is subjected to short-time inverse Fourier transform and windowing to generate several enhanced speech data. The preset forward convolutional network includes: four levels of forward convolutional layers; each of the forward convolutional layers includes: a two-dimensional forward convolutional layer, a normalization layer, and an activation function layer; The forward convolutional network performs complex encoding on each of the first speech data to generate several second speech data, including: Each level of forward convolutional layer performs two-dimensional forward convolution processing on each of the first speech data to generate several second speech data. In the process of generating the second speech data, the output of the previous forward convolutional layer serves as the input of the next forward convolutional layer. When each forward convolutional layer performs two-dimensional forward convolution processing on each of the first speech data, for the first-level forward convolutional layer, the input of the two-dimensional forward convolutional layer is two frames of first speech data output after Fourier transform. For the second to fourth-level forward convolutional layers, the input of the two-dimensional forward convolutional layer is one frame of speech data currently output from the previous level and one frame of speech data previously output from the previous level, and the output is one frame of speech data. The speech data output by the two-dimensional forward convolutional layer within each forward convolutional layer sequentially passes through a first normalization layer and a first activation function layer. The first normalization layer is used to normalize the speech data, and the first activation function layer is used to activate the normalized speech data.

2. The speech enhancement method according to claim 1, characterized in that, The preset transposed convolutional network includes: four layers of transposed convolutional layers; each transposed convolutional layer includes: a two-dimensional transposed convolutional layer, a normalization layer, and an activation function layer; The transposed convolutional network performs complex decoding on each of the third speech data to generate several fourth speech data, including: In the process of generating the fourth speech data, the output of the previous transposed convolutional layer serves as the input of the next transposed convolutional layer. When each two-dimensional convolutional layer performs two-dimensional transposed convolution processing on each of the third speech data, for the first-level transposed convolutional layer, the input of the two-dimensional transposed convolutional layer is two frames of third speech data output after Fourier transform. For the second to fourth-level transposed convolutional layers, the input of the two-dimensional transposed convolutional layer is one frame of speech data currently output from the previous level and one frame of speech data previously output from the previous level, and the output is one frame of speech data. The speech data output by the two-dimensional transposed convolutional layer within each transposed convolutional layer sequentially passes through a second normalization layer and a second activation function layer. The second normalization layer is used to normalize the speech data, and the second activation function layer is used to perform activation function processing on the normalized speech data.

3. The speech enhancement method according to claim 1, characterized in that, When each of the two-dimensional forward convolutional layers acquires the input speech data, it performs depthwise convolution or pointwise convolution on the input speech data. When performing depthwise convolution, the kernel height and width are calculated in a loop, and the results of the loop calculation are used to calculate the number of layers and the output height layer in a loop. When performing pointwise convolution, the output channel layer, the output height layer and the input channel layer are calculated in a loop.

4. The speech enhancement method according to claim 2, characterized in that, When each of the two-dimensional transposed convolutional layers acquires the input speech data, it performs a depthwise transposed convolution or a pointwise transposed convolution on the input speech data. When performing depthwise transposed convolution, the preset input width layer, kernel height layer, and kernel width layer are cyclically unwound and calculated. Based on the results of the unwound cyclic calculation, the preset number of group layers and input height layer are cyclically calculated. When performing pointwise transposed convolution, the preset input height layer, input channel layer, and output channel layer are cyclically calculated.

5. The speech enhancement method according to claim 1, characterized in that, The long short-term memory network includes: a first long short-term memory layer, a second long short-term memory layer, and a fully connected layer; Each of the second speech data is input into a Long Short-Term Memory (LSTM) network, so that the LSM network predicts the current speech data based on historical speech data, generating several third speech data, including: Each of the second speech data is input into the first long short-term memory layer, and the first long short-term memory layer makes a first prediction on each of the second speech data based on historical speech data; The second speech data after the first prediction are input into the second long short-term memory layer, and the second prediction is performed through the second long short-term memory layer; The second speech data after each prediction is input into the fully connected layer. The fully connected layer aggregates the information of the second speech data after each prediction to generate several third speech data.

6. The speech enhancement method according to claim 1, characterized in that, The generation of the several speech data to be enhanced includes: sampling the input speech in frames according to a preset frequency to generate several speech data to be enhanced.

7. A voice enhancement device, characterized in that, include: Short-time Fourier transform module, complex number encoding module, long short-term memory module, complex number decoding module, and enhanced speech data generation module; The short-time Fourier transform module is used to acquire several speech data to be enhanced, perform windowing processing on each speech data to be enhanced, and perform short-time Fourier transform on each speech data to be enhanced after windowing processing to generate several first speech data. The complex encoding module is used to input each of the first speech data into a preset forward convolutional network, so that the forward convolutional network performs complex encoding on each of the first speech data to generate a plurality of second speech data. The preset forward convolutional network includes: four levels of forward convolutional layers; each of the forward convolutional layers includes: a two-dimensional forward convolutional layer, a normalization layer, and an activation function layer; the forward convolutional network performs complex encoding on each of the first speech data to generate several second speech data, including: performing two-dimensional forward convolution processing on each of the first speech data through each level of the forward convolutional layer to generate several second speech data; wherein, in the process of generating second speech data, the output of the previous forward convolutional layer is used as the input of the next forward convolutional layer; when each forward convolutional layer performs two-dimensional forward convolution processing on each of the first speech data, for the first... The system consists of a first-level forward convolutional layer. The input to the two-dimensional forward convolutional layer is two frames of first speech data output after Fourier transform. For the second to fourth-level forward convolutional layers, the input to the two-dimensional forward convolutional layer is a frame of speech data currently output from the previous layer and a frame of speech data previously output from the previous layer, with an output of a single frame of speech data. The speech data output from the two-dimensional forward convolutional layer within each forward convolutional layer sequentially passes through a first normalization layer and a first activation function layer. The first normalization layer is used to normalize the speech data, and the first activation function layer is used to activate the normalized speech data. The long short-term memory module is used to input each of the second speech data into the long short-term memory network. The long short-term memory network predicts the current speech data based on historical speech data and generates several third speech data. The complex decoding module is used to input each of the third speech data into a preset transposed convolutional network, so that the transposed convolutional network performs complex decoding on each of the third speech data to generate several fourth speech data. The enhanced speech data generation module is used to perform masking processing on each of the fourth speech data, and to perform short-time inverse Fourier transform and windowing processing on each of the masked fourth speech data to generate several enhanced speech data.

8. A terminal device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the speech enhancement method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the storage medium is located to perform the speech enhancement method as described in any one of claims 1 to 6.

Images