Speech enhancement method, device, equipment and storage medium

Abstract

The present invention discloses a speech enhancement method, device, equipment and storage medium. Acquire speech sample data, and split the speech sample data into a training set and a verification set according to a set ratio; wherein the speech sample data includes noisy speech and corresponding pure speech; train a set speech enhancement model based on the training set; verify the trained set speech enhancement model based on the verification set; input the speech to be enhanced into the verified set speech enhancement model, and output the target speech. The speech enhancement method provided in an embodiment of the present invention inputs the speech to be enhanced into the verified set speech enhancement model, and outputs the target speech, which can improve the problem of speech loss and speech distortion that easily damage the original speech data during the speech enhancement process, and ensure the effect of speech enhancement.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of speech processing technology, and in particular to a speech enhancement method, apparatus, device and storage medium. Background Art

[0002] Traditional speech enhancement technologies are based on signal processing methods. These algorithms are based on a series of mathematical assumptions and ideal assumptions of scenarios. The noise reduction performance of the algorithm is not high, and the performance improvement of the speech recognition system is even more limited. With the development of computer technology and the maturity of deep learning algorithms, speech enhancement technology based on neural network models has gradually become mainstream. Most of these methods perform noise reduction processing on speech in the frequency domain, and their adaptability to non-stationary noise and diverse noise environments is much higher than that of signal processing methods. However, they also have higher requirements for the amount of model training data, and the stability of the model is not as good as that of signal processing methods. The service objects of speech enhancement methods are divided into voice communication and speech recognition systems. The algorithms are pre-processed at the front end of voice communication and speech recognition systems respectively. The optimization directions of the two are different. The algorithms must use different optimization goals to achieve their respective effects. It is difficult to improve the clarity of speech while significantly improving the accuracy of the speech recognition system. Summary of the invention

[0003] The embodiments of the present invention provide a speech enhancement method, apparatus, device and storage medium, which can improve the problem of speech loss and speech distortion that easily occur during speech enhancement and damage the original speech data, thereby ensuring the effect of speech enhancement.

[0004] In a first aspect, an embodiment of the present invention provides a speech enhancement method, comprising:

[0005] Acquire speech sample data, and split the speech sample data into a training set and a validation set according to a set ratio; wherein the speech sample data includes noisy speech and corresponding clean speech;

[0006] Training a set speech enhancement model based on the training set;

[0007] Verifying the trained set speech enhancement model based on the verification set;

[0008] The speech to be enhanced is input into the verified set speech enhancement model, and the target speech is output.

[0009] In a second aspect, an embodiment of the present invention further provides a speech enhancement device, comprising:

[0010] A sample data acquisition module is used to acquire speech sample data and split the speech sample data into a training set and a validation set according to a set ratio; wherein the speech sample data includes noisy speech and corresponding clean speech;

[0011] A speech enhancement model training module is set, which is used to train the speech enhancement model based on the training set;

[0012] A speech enhancement model verification module is set, which is used to verify the trained speech enhancement model based on the verification set;

[0013] The target speech acquisition module is used to input the speech to be enhanced into the set speech enhancement model after verification, and output the target speech.

[0014] In a third aspect, an embodiment of the present invention further provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the speech enhancement method as described in the embodiment of the present invention is implemented.

[0015] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processing device, implements the speech enhancement method as described in the embodiment of the present invention.

[0016] The embodiment of the present invention discloses a speech enhancement method, device, equipment and storage medium. Acquire speech sample data, and split the speech sample data into a training set and a verification set according to a set ratio; wherein the speech sample data includes noisy speech and corresponding pure speech; train a set speech enhancement model based on the training set; verify the trained set speech enhancement model based on the verification set; input the speech to be enhanced into the verified set speech enhancement model, and output the target speech. The speech enhancement method provided by the embodiment of the present invention inputs the speech to be enhanced into the verified set speech enhancement model, and outputs the target speech, which can improve the problem of speech loss and speech distortion that easily damage the original speech data during the speech enhancement process, and ensure the effect of speech enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of a speech enhancement method in Embodiment 1 of the present invention;

[0018] Figure 2 is an example diagram of a frequency domain feature extraction module in Embodiment 1 of the present invention;

[0019] Figure 3 is a structural principle diagram of a setting speech enhancement model in Embodiment 1 of the present invention;

[0020] Figure 4 is a structural schematic diagram of a speech enhancement device in Embodiment 2 of the present invention;

[0021] Figure 5It is a structural diagram of a computer device in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0022] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0023] Embodiment 1

[0024] Figure 1 This is a flow chart of a speech enhancement method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of enhancing speech with noise. The method can be executed by a speech enhancement device, such as Figure 1 As shown, the method specifically comprises the following steps:

[0025] Step 110, obtaining speech sample data, and splitting the speech sample data into a training set and a validation set according to a set ratio.

[0026] The speech sample data includes noisy speech and corresponding clean speech, wherein the noisy speech may be a mixture of clean speech and noise data according to a random signal-to-noise ratio. The training set is used to train the set speech enhancement model, and the validation set is used to validate the effect of the trained set speech enhancement model. The number of speech sample data included in the training set is greater than the number of speech sample data in the validation set. For example, the set ratio may be 2:1.

[0027] Specifically, the process of obtaining the speech sample data may be: obtaining pure speech; mixing the pure speech with noise data according to a random signal-to-noise ratio to obtain noisy speech; and combining the pure speech and the noisy speech into speech sample data.

[0028] Among them, the clean speech can be obtained from a set corpus (for example: LibriSpeech). The noise data can be obtained from the WHAM noise data set but is not limited to the speech data set. The range of the random signal-to-noise ratio can be a value between -5 and 25dB. Exemplarily, the process of obtaining the speech sample data can be: the clean speech of 4 different users is selected from the LibriSpeech corpus, and a total of 132 data are selected as the training set, and the total length of each speech is 10s. In the corpus, the clean speech of 4 speakers is selected as the verification set, of which there are 3 users who are the same as the users in the training set but have different speech content, with a total of 50 voices. There is 1 user who is different from the user in the training set, and the user has a total of 10 voices. 8 noise data are selected from the noise data set. The noise data and the clean speech are mixed in a random signal-to-noise ratio (-5 to 25dB). The final training set has a total of 1056 noisy voices and 1056 corresponding clean speech data, about 3h of data. The validation set contains 480 noisy speeches and their corresponding clean speeches, totaling about 1.5 hours of data.

[0029] Step 120: training a set speech enhancement model based on the training set.

[0030] Among them, the speech enhancement model is set to include from input to output: Fourier transform module, frequency domain feature extraction module, data fusion module, inverse Fourier transform module, normalization module, time domain feature extraction module and feature analysis module; the output of the Fourier transform module is jump-connected to the input of the data fusion module.

[0031] Among them, the Fourier transform module can be a short-time Fourier transform (STFT) module. In this embodiment, the speech sampling frequency can be 16kHz, and the short-time Fourier transform can be performed according to the frame length of 510 sampling points and the frame shift of 250 sampling points. The frequency domain feature extraction module includes a first frequency domain feature extraction unit and a second frequency domain feature extraction unit. Among them, the first frequency domain feature extraction unit includes a normalization layer, a discard layer and a multi-head attention mechanism layer in sequence; the second frequency domain feature extraction unit includes: a normalization layer, a discard layer and a feedforward layer network in sequence; wherein the feedforward layer network includes: a circulation layer, an activation layer, a discard layer and a linear processing layer in sequence. Figure 2 is an example diagram of the frequency domain feature extraction module in this embodiment. Figure 2 As shown in the figure, the data passes through the normalization layer, the discard layer and the multi-head attention mechanism layer in sequence, and the output data is multiplied by the input data of the normalization layer and then enters the circulation layer, activation layer, discard layer and linear processing layer in sequence. Among them, the normalization layer performs layer normalization on the data.

[0032] The inverse Fourier transform module can be an inverse short-time Fourier transform module (ISTFT). The normalization module includes: a normalization layer, a one-dimensional convolution layer, an activation layer and a segmentation layer. Among them, the normalization layer is used to group normalize the data; the activation layer can be a ReLU activation function; the segmentation layer is used to block the three-dimensional tensor with a block size of K (which can be 200, 250 or 300, etc.) and a block shift of P (P = K / 2, rounded). At the same time, in order to evenly divide the blocks, padding is used to fill the last block.

[0033] The time domain feature extraction module includes a first time domain feature extraction unit and a second time domain feature extraction unit; wherein the first time domain feature extraction unit and the first frequency domain feature extraction unit have the same structure, and the second time domain feature extraction unit and the second frequency domain feature extraction unit have the same structure.

[0034] Among them, the feature analysis module includes a two-dimensional convolution layer, an overlap-addition layer, a one-dimensional convolution unit and a one-dimensional deconvolution layer.

[0035] The processing process of the overlap-add layer is the inverse process of the segmentation process, which is used to overlap and add multiple data blocks according to the block shift P to restore the length of the original three-dimensional tensor when it is not segmented.

[0036] The one-dimensional convolutional layer unit includes three one-dimensional convolutional layers, two of which are connected in parallel. Each one-dimensional convolutional layer is connected to an activation function. The outputs of the two parallel one-dimensional convolutions are multiplied and input into the third one-dimensional convolutional layer. The three activation functions are Tanh activation function, Sigmoid activation function and ReLU activation function. The stride and convolution kernel size of the one-dimensional deconvolution layer are the same as those of the one-dimensional convolution layer.

[0037] For example, Figure 3 It is a structural principle diagram of setting the speech enhancement model in this embodiment. Figure 3 The two-stage Transformer in the figure represents the frequency domain feature extraction module and the time domain feature extraction module respectively, GroupNorm represents the group normalization layer, Conv1d and 1DConv both represent one-dimensional convolutional layers, 2-DConv represents a two-dimensional convolutional layer, and ConvTranspose1d represents a one-dimensional deconvolutional layer.

[0038] like Figure 3 As shown in FIG. 1 , the working principle of the speech enhancement model is divided into the frequency domain part and the time domain part. The processing process of the frequency domain part is as follows:

[0039] First, the noisy speech x is used as the input data of the model, and Fourier transform is performed first. As shown in the following formula, the output data h is obtained: h = STFT (x).

[0040] Then it goes through a two-stage Transformer structure. In the first stage of the Transformer structure, h is first layer-normalized (LayerNorm), then passed through a dropout layer (Dropout), and then passed through a multi-head attention mechanism (MultiHeadAttention) to obtain the output data h′: as shown in the following formula: h′=MultiHeadAttention(Dropout(LayerNorm(h))).

[0041] Next, h′ and h are added and input into the second-stage Transformer structure, and then pass through layer normalization, dropout layer, and feed forward network (FFN) to obtain the output data h″, as shown in the following formula: h″=Feed Forward(Dropout(LayerNorm(h′)))

[0042] Finally, the product of h and h″ is inverse Fourier transformed to restore the dimension of the original input data x. The final output result z of the frequency domain part is obtained, as shown in the following formula: z=ISTFT(h*h″).

[0043] The processing of the time domain part is as follows:

[0044] The output result z of the frequency domain part is taken as input, group normalized (GroupNorm), then passed through a one-dimensional convolution (conv1d), and transformed through a ReLU activation function to obtain the output result z′, as shown in the following formula: z′=ReLU(conv1d(GroupNorm(z))).

[0045] Next, z′ is segmented with a segment size of K and a segment shift of P (P = K / 2, rounded to the nearest integer). Each segment is then input into a two-stage Transformer structure, where each segment is processed in the first stage Transformer to extract the segment information. The segment information extracted in the first stage is used as the input of the second stage Transformer to extract contextual information between segments, thereby obtaining the output result z″.

[0046] Then z″ is input into a two-dimensional convolutional layer (conv2d) to obtain the output result s. As shown in the following formula: s=conv2d(z″), and then s is overlapped and added.

[0047] Next, perform a one-dimensional convolution operation on s, and then pass it through the Tanh activation function to get the result s1, as shown in the following formula: s1 = Tanh (conv1d (s)). At the same time, input s into another one-dimensional convolution, and then pass it through the Sigmoid activation function to get the result s2, as shown in the following formula: s2 = Sigmoid (conv1d (s)). Finally, after multiplying s1 and s2, pass it through a one-dimensional convolution and ReLU activation function to get s′, as shown in the following formula: s′ = ReLU (conv1d (s1*s2)).

[0048] Finally, s′ is input into a one-dimensional deconvolution layer (conv1d-transpose) to obtain the enhanced speech s″, as shown in the following formula: s″=conv1d-transpose(s′).

[0049] Specifically, the process of training the set speech enhancement model based on the training set can be: inputting the noisy speech in the training set into the set speech enhancement model to obtain the first enhanced speech; calculating the scale-invariant signal-to-noise ratio SI-SNR loss function between the first enhanced speech and the pure speech in the training set; training the set speech enhancement model based on the SI-SNR loss function and the set optimizer until the training termination condition is met.

[0050] The training termination condition may be that the number of iterations reaches a set value. The set optimizer may be an Adam optimizer. The calculation formula of the scale-invariant signal-to-noise ratio SI-SNR loss function is as follows: in, E noise =s″-S target , s″ represents enhanced speech, S represents pure speech, ||S|| 2 =<S,S> represents the power of the speech signal,<s″,S> =|s″|·|S|.

[0051] Step 130: verify the trained set speech enhancement model based on the verification set.

[0052] The process of verifying the trained set speech enhancement model based on the verification set can be: inputting the noisy speech in the verification set into the trained set speech enhancement model, and outputting a second enhanced speech; determining a first speech quality index and a second speech quality index of the second enhanced speech; if both the first speech quality index and the second speech quality index meet the conditions, the verification of the set speech enhancement model is completed; otherwise, retraining the set speech enhancement model based on the training set.

[0053] Among them, the first speech quality indicator can be a PESQ (Perceptual evaluation of speechquality) indicator, with a value range of -0.5 to 4.5, and the larger the value, the better the performance. The second speech quality indicator can be an STOI (Short Time Objective Intelligibility) indicator, ranging from 0 to 1. The relationship between speech intelligibility and speech distortion and speech loss is relatively large. The greater the distortion, the lower the STOI value. The first speech quality indicator and the second speech quality indicator both meet the conditions, which can be understood as the PESQ indicator is greater than the first set value, and the STOI is greater than the second set value. Exemplarily, the first setting value can be set to 4, and the second setting value can be set to 0.8.

[0054] Specifically, the specific principle of setting the speech enhancement model after training with the noisy speech input in the verification set can be found in the working principle of the set speech enhancement model in the above embodiment, which will not be repeated here.

[0055] Step 140, input the speech to be enhanced into the verified set speech enhancement model, and output the target speech.

[0056] Specifically, the specific principle of outputting the target speech by the set speech enhancement model after the speech to be enhanced is verified can be found in the working principle of the set speech enhancement model in the above embodiment, which will not be repeated here.

[0057] Exemplarily, Table 1 lists the PESQ and STOI values ​​of different enhancement algorithms at different signal-to-noise ratios, and Table 2 lists the PESQ and STOI values ​​of the time-frequency combined speech enhancement algorithm proposed in this embodiment at different signal-to-noise ratios.

[0058] Table 1

[0059]

[0060] Table 2

[0061]

[0062]

[0063] By comparing Table 1 and Table 2, it is found that the algorithm proposed in this patent has a PESQ improvement of approximately 0.8 under different signal-to-noise ratios, which is 0.4 higher than the DNN-DM algorithm and 0.3 higher than the LSTM-IRM algorithm. In addition, it can be found that the DNN-DM algorithm and the LSTM-IRM algorithm have a decrease in speech intelligibility while improving the speech listening quality. Compared with the algorithm proposed in this patent, this situation does not occur, which shows that the algorithm proposed in this embodiment can reduce speech loss and speech distortion to a certain extent.

[0064] The technical solution of this embodiment obtains speech sample data, and splits the speech sample data into a training set and a verification set according to a set ratio; wherein the speech sample data includes noisy speech and corresponding pure speech; a set speech enhancement model is trained based on the training set; the trained set speech enhancement model is verified based on the verification set; the speech to be enhanced is input into the verified set speech enhancement model, and the target speech is output. The speech enhancement method provided in the embodiment of the present invention inputs the speech to be enhanced into the verified set speech enhancement model, and outputs the target speech, which can improve the problem of speech loss and speech distortion that easily damage the original speech data during the speech enhancement process, and ensure the effect of speech enhancement.

[0065] Embodiment 2

[0066] Figure 4 Schematic diagram of a speech enhancement device provided by an embodiment of the present disclosure. Figure 4 As shown, the device comprises:

[0067] The sample data acquisition module 210 is used to acquire speech sample data and split the speech sample data into a training set and a validation set according to a set ratio; wherein the speech sample data includes noisy speech and corresponding clean speech;

[0068] A set speech enhancement model training module 220 is used to train a set speech enhancement model based on a training set;

[0069] A set speech enhancement model verification module 230 is used to verify the trained set speech enhancement model based on the verification set;

[0070] The target speech acquisition module 240 is used to input the speech to be enhanced into the set speech enhancement model after verification and output the target speech.

[0071] Optionally, the sample data acquisition module 210 is further used to:

[0072] Get pure audio;

[0073] The clean speech and the noise data are mixed according to the random signal-to-noise ratio to obtain the noisy speech;

[0074] The clean speech and the noisy speech are combined into speech sample data.

[0075] Optionally, the speech enhancement model training module 220 is set to be further used for:

[0076] Inputting the noisy speech in the training set into a speech enhancement model to obtain a first enhanced speech;

[0077] Calculate the scale-invariant signal-to-noise ratio SI-SNR loss function between the first enhanced speech and the clean speech in the training set;

[0078] The set speech enhancement model is trained based on the SI-SNR loss function and the set optimizer until the training termination condition is met.

[0079] Optionally, the speech enhancement model verification module 230 is set to be further used for:

[0080] Input the noisy speech in the validation set into the trained set speech enhancement model, and output the second enhanced speech;

[0081] Determining a first speech quality indicator and a second speech quality indicator of a second enhanced speech;

[0082] If both the first speech quality index and the second speech quality index meet the conditions, the verification of the set speech enhancement model is completed; otherwise, the set speech enhancement model is retrained based on the training set.

[0083] Optionally, the speech enhancement model is set to include, from input to output, in sequence: a Fourier transform module, a frequency domain feature extraction module, a data fusion module, an inverse Fourier transform module, a normalization module, a time domain feature extraction module and a feature analysis module; the output of the Fourier transform module is jump-connected to the input of the data fusion module.

[0084] Optionally, the frequency domain feature extraction module includes a first frequency domain feature extraction unit and a second frequency domain feature extraction unit;

[0085] The first frequency domain feature extraction unit includes a normalization layer, a discard layer and a multi-head attention mechanism layer in sequence; the second frequency domain feature extraction unit includes a normalization layer, a discard layer and a feedforward layer network in sequence; the input of the first frequency domain feature extraction unit is jump-connected with the second frequency domain feature extraction unit; wherein the feedforward layer network includes a recurrent layer, an activation layer, a discard layer and a linear processing layer in sequence.

[0086] Optionally, the normalization module includes: a normalization layer, a one-dimensional convolution layer, an activation layer, and a segmentation layer;

[0087] The time domain feature extraction module includes a first time domain feature extraction unit and a second time domain feature extraction unit; wherein the first time domain feature extraction unit has the same structure as the first frequency domain feature extraction unit, and the second time domain feature extraction unit has the same structure as the second frequency domain feature extraction unit;

[0088] The feature analysis module includes a two-dimensional convolution layer, an overlap-addition layer, a one-dimensional convolution unit and a one-dimensional deconvolution layer.

[0089] The above device can execute the methods provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not described in detail in this embodiment, please refer to the methods provided by all the above embodiments of the present invention.

[0090] Embodiment 4

[0091] Figure 5 A schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 5 A block diagram of a computer device 312 suitable for use in implementing embodiments of the present invention is shown. Figure 5 The computer device 312 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention. The device 312 is a typical computing device with speech enhancement functions.

[0092] like Figure 5 As shown, the computer device 312 is in the form of a general-purpose computing device. The components of the computer device 312 may include, but are not limited to: one or more processors 316, a storage device 328, and a bus 318 connecting different system components (including the storage device 328 and the processor 316).

[0093] Bus 318 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0094] The computer device 312 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 312, including volatile and non-volatile media, removable and non-removable media.

[0095] The storage device 328 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 330 and / or cache memory 332. The computer device 312 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 334 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a read-only optical disk (Compact Disc-Read Only Memory, CD-ROM), a digital video disk (Digital Video Disc-Read Only Memory, DVD-ROM) or other optical media) may be provided. In these cases, each drive may be connected to the bus 318 via one or more data medium interfaces. The storage device 328 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0096] A program 336 having a set (at least one) of program modules 326 may be stored, for example, in a storage device 328, such program modules 326 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 326 generally perform the functions and / or methods of the embodiments described herein.

[0097] The computer device 312 may also communicate with one or more external devices 314 (e.g., keyboard, pointing device, camera, display 324, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 312, and / or communicate with any device that enables the computer device 312 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 322. In addition, the computer device 312 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter 320. As shown, the network adapter 320 communicates with other modules of the computer device 312 through a bus 318. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in conjunction with the computer device 312, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) systems, tape drives, and data backup storage systems.

[0098] The processor 316 executes various functional applications and data processing by running the programs stored in the storage device 328, such as implementing the speech enhancement method provided in the above embodiment of the present invention.

[0099] Embodiment 4

[0100] The embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processing device, a speech enhancement method as in the embodiment of the present invention is implemented. The above-mentioned computer-readable medium of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0101] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0102] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0103] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains speech sample data, and splits the speech sample data into a training set and a verification set according to a set ratio; wherein the speech sample data includes noisy speech and corresponding clean speech; trains a set speech enhancement model based on the training set; verifies the trained set speech enhancement model based on the verification set; inputs the speech to be enhanced into the verified set speech enhancement model, and outputs the target speech.

[0104] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0105] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0106] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0107] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0108] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0109] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A speech enhancement method, characterized in that: include: Acquire speech sample data, and split the speech sample data into a training set and a validation set according to a set ratio; wherein the speech sample data includes noisy speech and corresponding clean speech; Training a set speech enhancement model based on the training set; Verifying the trained set speech enhancement model based on the verification set; Input the speech to be enhanced into the set speech enhancement model after verification, and output the target speech; The set speech enhancement model includes, from input to output, a Fourier transform module, a frequency domain feature extraction module, a data fusion module, an inverse Fourier transform module, a normalization module, a time domain feature extraction module and a feature analysis module; the output of the Fourier transform module is jump-connected to the input of the data fusion module; Wherein, the frequency domain feature extraction module includes a first frequency domain feature extraction unit and a second frequency domain feature extraction unit; The first frequency domain feature extraction unit includes a normalization layer, a discard layer and a multi-head attention mechanism layer in sequence; the second frequency domain feature extraction unit includes a normalization layer, a discard layer and a feedforward layer network in sequence; the input of the first frequency domain feature extraction unit is jump-connected with the second frequency domain feature extraction unit; wherein the feedforward layer network includes a recurrent layer, an activation layer, a discard layer and a linear processing layer in sequence.

2. The method according to claim 1, characterized in that: Get voice sample data, including: Get pure audio; Mixing the clean speech with the noise data according to a random signal-to-noise ratio to obtain noisy speech; The clean speech and the noisy speech are combined into speech sample data.

3. The method according to claim 1, characterized in that Training a set speech enhancement model based on the training set includes: Inputting the noisy speech in the training set into a speech enhancement model to obtain a first enhanced speech; Calculating a scale-invariant signal-to-noise ratio (SI-SNR) loss function between the first enhanced speech and the clean speech in the training set; The set speech enhancement model is trained based on the SI-SNR loss function and the set optimizer until a training termination condition is met.

4. The method according to claim 1, characterized in that Verifying the trained set speech enhancement model based on the verification set includes: Inputting the noisy speech in the verification set into the trained set speech enhancement model to output a second enhanced speech; Determining a first speech quality indicator and a second speech quality indicator of the second enhanced speech; If both the first speech quality index and the second speech quality index meet the conditions, the verification of the set speech enhancement model is completed; otherwise, the set speech enhancement model is retrained based on the training set.

5. The method according to claim 1, characterized in that The normalization module includes: normalization layer, one-dimensional convolution layer, activation layer and segmentation layer; The time domain feature extraction module includes a first time domain feature extraction unit and a second time domain feature extraction unit; wherein the first time domain feature extraction unit has the same structure as the first frequency domain feature extraction unit, and the second time domain feature extraction unit has the same structure as the second frequency domain feature extraction unit; The feature analysis module includes a two-dimensional convolution layer, an overlap-addition layer, a one-dimensional convolution unit and a one-dimensional deconvolution layer.

6. A speech enhancement device, characterized in that: include: A sample data acquisition module is used to acquire speech sample data and split the speech sample data into a training set and a validation set according to a set ratio; wherein the speech sample data includes noisy speech and corresponding clean speech; A speech enhancement model training module is set, which is used to train the speech enhancement model based on the training set; A speech enhancement model verification module is set, which is used to verify the trained speech enhancement model based on the verification set; A target speech acquisition module, used for inputting the speech to be enhanced into the set speech enhancement model after verification, and outputting the target speech; The set speech enhancement model includes, from input to output, a Fourier transform module, a frequency domain feature extraction module, a data fusion module, an inverse Fourier transform module, a normalization module, a time domain feature extraction module and a feature analysis module; the output of the Fourier transform module is jump-connected to the input of the data fusion module; Wherein, the frequency domain feature extraction module includes a first frequency domain feature extraction unit and a second frequency domain feature extraction unit; The first frequency domain feature extraction unit includes a normalization layer, a discard layer and a multi-head attention mechanism layer in sequence; the second frequency domain feature extraction unit includes a normalization layer, a discard layer and a feedforward layer network in sequence; the input of the first frequency domain feature extraction unit is jump-connected with the second frequency domain feature extraction unit; wherein the feedforward layer network includes a recurrent layer, an activation layer, a discard layer and a linear processing layer in sequence.

7. A computer device, characterized in that: include: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the speech enhancement method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the speech enhancement method as described in any one of claims 1 to 5 is implemented.

Images