Speech enhancement method and device based on deep learning, equipment and medium

Abstract

The invention relates to the technical field of speech recognition, and discloses a speech enhancement method and device based on deep learning, equipment and a medium. The method comprises the following steps: acquiring a noisy voice signal, and performing feature extraction on the noisy voice signal through one-dimensional convolution to obtain a noisy voice feature; inputting the noisy speech features into a mask prediction model to obtain a ratio mask, and calculating and predicting pure speech features according to the ratio mask; inputting the predicted pure voice features into a full connection layer to obtain predicted pure voice signals; and obtaining an actual pure voice signal, training the one-dimensional convolution, the mask prediction model and the full connection layer according to the actual pure voice signal and the predicted pure voice signal, and obtaining the trained one-dimensional convolution, the trained mask prediction model and the trained full connection layer. The method can adapt to speech enhancement of different sampling rates, and avoids the problem that artificial feature extraction is difficult to be compatible with different sampling rates.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a speech enhancement method, device, equipment and medium based on deep learning. Background Art

[0002] Speech enhancement is to separate the noise and human voice in mixed speech, while retaining the human voice as completely as possible and removing as much noise as possible. This can effectively improve the quality of voice communication or voice interaction, so that people or machines can hear clear and clean speech in noisy environments.

[0003] The current noise reduction method based on deep learning technology can effectively remove fast-changing noise and achieve good speech noise reduction effects, but the noise reduction method based on deep learning is heavily dependent on data. Summary of the invention

[0004] In view of this, the purpose of the present invention is to overcome the deficiencies in the prior art and provide a speech enhancement method, device, equipment and medium based on deep learning.

[0005] The present invention provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present disclosure provides a method for speech enhancement based on deep learning, the method comprising: obtaining a noisy speech signal, performing feature extraction on the noisy speech signal through one-dimensional convolution, and obtaining a noisy speech feature;

[0007] Inputting the noisy speech features into a mask prediction model to obtain a ratio mask, and calculating and predicting the clean speech features based on the ratio mask;

[0008] Inputting the predicted clean speech feature into a fully connected layer to obtain a predicted clean speech signal;

[0009] An actual clean speech signal is obtained, and according to the actual clean speech signal and the predicted clean speech signal, the one-dimensional convolution, the mask prediction model and the fully connected layer are trained to obtain a trained one-dimensional convolution, a trained mask prediction model and a trained fully connected layer.

[0010] Optionally, the noisy speech signal includes a first noisy speech signal and a second noisy speech signal, the one-dimensional convolution includes a first one-dimensional convolution and a second one-dimensional convolution, the noisy speech feature includes a first noisy speech feature and a second noisy speech feature, and the extracting features of the noisy speech signal by one-dimensional convolution to obtain the noisy speech feature includes:

[0011] Obtaining a preset window length and a preset window shift, calculating a first convolution kernel according to the preset window length and the first noisy speech signal, and calculating a first convolution step length according to the preset window shift and the first noisy speech signal;

[0012] Calculating a second convolution kernel according to the preset window length and the second noisy speech signal, and calculating a second convolution step length according to the preset window shift and the second noisy speech signal;

[0013] Performing feature extraction on the first noisy speech signal according to the first convolution kernel and the first convolution step size to obtain the first noisy speech feature;

[0014] Feature extraction is performed on the second noisy speech signal according to the second convolution kernel and the second convolution step size to obtain the second noisy speech feature.

[0015] Optionally, inputting the noisy speech feature into a mask prediction model to obtain a ratio mask includes:

[0016] The mask prediction model performs forward propagation on the noisy speech feature according to a preset network structure, and outputs the ratio mask, wherein the ratio mask represents the ratio relationship between the noisy speech feature and its corresponding clean speech feature.

[0017] Optionally, the calculating and predicting the clean speech feature according to the ratio mask comprises:

[0018] The ratio mask is multiplied element-by-element by the noisy speech feature to obtain the predicted clean speech feature.

[0019] Optionally, the fully connected layer includes a first fully connected layer and a second fully connected layer, the predicted clean speech signal includes a first predicted clean speech signal and a second predicted clean speech signal, and the step of inputting the predicted clean speech feature into the fully connected layer to obtain the predicted clean speech signal includes:

[0020] Mapping the predicted clean speech feature to a time domain signal through the first fully connected layer to obtain the first predicted clean speech signal;

[0021] The predicted clean speech feature is mapped to a time domain signal through the second fully connected layer to obtain the second predicted clean speech signal.

[0022] Optionally, the actual clean speech signal includes a first actual clean speech signal and a second actual clean speech signal, and the training of the one-dimensional convolution, the mask prediction model and the fully connected layer according to the actual clean speech signal and the predicted clean speech signal to obtain the trained one-dimensional convolution, the trained mask prediction model and the trained fully connected layer includes:

[0023] Calculating a first error between the first actual clean speech signal and the first predicted clean speech signal, and calculating a second error between the second actual clean speech signal and the second predicted clean speech signal;

[0024] Using a back propagation algorithm to train the one-dimensional convolution, the mask prediction model, and the fully connected layer according to the first error and the second error;

[0025] The training process is continuously iterated until the error is less than a preset threshold, thereby obtaining the trained one-dimensional convolution, the trained mask prediction model and the trained fully connected layer.

[0026] Optionally, after obtaining the trained one-dimensional convolution, the trained mask prediction model and the trained fully connected layer, the method further includes:

[0027] The trained one-dimensional convolution, the trained mask prediction model, and the trained fully connected layer are evaluated to obtain evaluation indicators, wherein the evaluation indicators include at least one of a signal-to-noise ratio, an accuracy rate, a recall rate, and an F1 score.

[0028] In a second aspect, an embodiment of the present disclosure provides a speech enhancement device based on deep learning, the device comprising:

[0029] An extraction module, used to obtain a noisy speech signal, and perform feature extraction on the noisy speech signal through one-dimensional convolution to obtain a noisy speech feature;

[0030] A calculation module, used for inputting the noisy speech feature into a mask prediction model to obtain a ratio mask, and calculating and predicting the clean speech feature according to the ratio mask;

[0031] A generation module, used for inputting the predicted clean speech features into a fully connected layer to obtain a predicted clean speech signal;

[0032] A training module is used to obtain an actual clean speech signal, and train the one-dimensional convolution, the mask prediction model and the fully connected layer according to the actual clean speech signal and the predicted clean speech signal to obtain a trained one-dimensional convolution, a trained mask prediction model and a trained fully connected layer.

[0033] In a third aspect, a computer device is provided in an embodiment of the present disclosure, wherein the computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the deep learning-based speech enhancement method described in the first aspect are implemented.

[0034] In a fourth aspect, a computer-readable storage medium is provided in an embodiment of the present disclosure, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the deep learning-based speech enhancement method described in the first aspect are implemented.

[0035] Beneficial effects of this application:

[0036] The deep learning-based speech enhancement method provided in the embodiment of the present application includes: obtaining a noisy speech signal, extracting features of the noisy speech signal through one-dimensional convolution to obtain noisy speech features; inputting the noisy speech features into a mask prediction model to obtain a ratio mask, and calculating and predicting clean speech features based on the ratio mask; inputting the predicted clean speech features into a fully connected layer to obtain a predicted clean speech signal; obtaining an actual clean speech signal, and training the one-dimensional convolution, the mask prediction model, and the fully connected layer based on the actual clean speech signal and the predicted clean speech signal to obtain a trained one-dimensional convolution, a trained mask prediction model, and a trained fully connected layer. The present application can adapt to speech enhancement at different sampling rates, avoiding the problem that artificial feature extraction is difficult to be compatible with different sampling rates.

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work. In each of the drawings, similar components are numbered similarly.

[0039] Figure 1 A flowchart of a deep learning-based speech enhancement method provided in an embodiment of the present application is shown;

[0040] Figure 2 A network structure diagram of a speech enhancement method based on deep learning provided in an embodiment of the present application is shown;

[0041] Figure 3 A schematic diagram of the structure of a speech enhancement device based on deep learning provided in an embodiment of the present application is shown;

[0042] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0044] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. In contrast, when an element is referred to as being "directly on" another element, there is no intermediate element. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0045] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0046] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of the template are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0048] Example 1

[0049] like Figure 1 , which is a flow chart of a deep learning-based speech enhancement method in an embodiment of the present application. The deep learning-based speech enhancement method provided in an embodiment of the present application includes the following steps:

[0050] Step S110, obtaining a noisy speech signal, performing feature extraction on the noisy speech signal through one-dimensional convolution, and obtaining noisy speech features.

[0051] In this embodiment, if Figure 2 As shown, the noisy speech signal includes a first noisy speech signal and a second noisy speech signal, the one-dimensional convolution includes a first one-dimensional convolution and a second one-dimensional convolution, and the noisy speech feature includes a first noisy speech feature and a second noisy speech feature. Among them, the first noisy speech signal is a noisy speech signal of 8KHz, and the second noisy speech signal is a noisy speech signal of 16KHz.

[0052] Next, obtain the preset window length and the preset window shift, calculate the first convolution kernel according to the preset window length and the first noisy speech signal, and calculate the first convolution step length according to the preset window shift and the first noisy speech signal; calculate the second convolution kernel according to the preset window length and the second noisy speech signal, and calculate the second convolution step length according to the preset window shift and the second noisy speech signal.

[0053] For example, taking the commonly used framing method with a window length of 20ms and a window shift of 10ms as an example, for an 8KHz noisy speech signal, a convolution kernel size of 8000×0.02=160 and a convolution step size of 8000×0.01=80 are used, while for a 16KHz noisy speech signal, a convolution kernel size of 16000×0.02=320 and a convolution step size of 16000×0.01=160 are used.

[0054] Furthermore, feature extraction is performed on the first noisy speech signal according to the first convolution kernel and the first convolution step size to obtain a first noisy speech feature; and feature extraction is performed on the second noisy speech signal according to the second convolution kernel and the second convolution step size to obtain a second noisy speech feature.

[0055] It should be noted that in the above process, the noisy speech signal is used as the input of the one-dimensional convolution, and the number of channels of the convolution kernel matches the number of channels of the input signal. For a single microphone signal, the number of input channels is 1, and the number of output channels of the convolution kernel determines the dimension of the feature vector. This embodiment can take 256, but in practice it can be adaptively adjusted according to the computing power situation, and this application does not limit this. In this way, after the convolution operation, noisy speech signals of different sampling rates will be uniformly converted into noisy speech feature vectors with a frame rate of 100 and a single frame dimension of 256. For example, a 10-second noisy speech signal will be converted into a 1000-frame 256-dimensional noisy speech feature vector.

[0056] The one-dimensional convolution in the above method can effectively extract local structure and timing information from noisy speech signals, providing key features for subsequent processing. After the convolution operation, noisy speech signals with different sampling rates are uniformly converted into feature vectors with the same frame rate and single frame dimension, which facilitates subsequent processing.

[0057] Step S120: input the noisy speech feature into a mask prediction model to obtain a ratio mask, and calculate and predict the clean speech feature based on the ratio mask.

[0058] Specifically, Figure 2 As shown in FIG, the noisy speech features extracted from the noisy speech signal are used as the input of the mask prediction model. These features are usually obtained from the original noisy speech signal through a one-dimensional convolution operation, and they contain the local structure and temporal information of the noisy speech signal.

[0059] After the mask prediction model receives the input noisy speech features, it will perform forward propagation according to the preset network structure. During this process, the noisy speech features will pass through each layer of the mask prediction model (such as convolution layer, pooling layer, etc., depending on the model design) in turn, and perform nonlinear transformation and feature extraction layer by layer.

[0060] During the forward propagation of the mask prediction model, a ratio mask with the same dimension as the input noisy speech feature is finally output. This ratio mask represents the ratio relationship between the noisy speech feature and its corresponding clean speech feature. Each element in the mask corresponds to an element in the input noisy speech feature vector, indicating the relative importance of the element in the clean speech feature.

[0061] Furthermore, the ratio mask is element-wise multiplied with the noisy speech feature to obtain the predicted clean speech feature. This step is actually filtering and enhancing the noisy speech feature using the ratio mask to extract features that are closer to clean speech.

[0062] The above method can effectively suppress noise and extract features that are closer to pure speech. The process of mask prediction and noise suppression can significantly improve the quality of predicted pure speech signals, making them closer to actual pure speech signals. In addition, it can also adapt to speech denoising at different sampling rates. Since there are very few parameters in the branch part, most of the parameters are shared by the mask prediction model, and the model parameters will not be significantly increased due to the addition of one more sampling rate. Speech denoising training at different sampling rates belongs to the training of different tasks. The tasks can assist each other, which is beneficial to suppress the overfitting of the mask prediction model and enhance the representation and generalization capabilities of the model.

[0063] Step S130: input the predicted clean speech features into a fully connected layer to obtain a predicted clean speech signal.

[0064] In this embodiment, if Figure 2 As shown, the fully connected layer includes a first fully connected layer and a second fully connected layer, and the predicted clean speech signal includes a first predicted clean speech signal and a second predicted clean speech signal. The first predicted clean speech signal is a predicted clean speech signal of 8KHz, and the second predicted clean speech signal is a predicted clean speech signal of 16KHz.

[0065] Understandably, the feature vector of a single frame is extracted from the predicted pure speech features output by the mask prediction model. These feature vectors are usually obtained in the convolution and mask application process of the previous step, and they contain the local structure and timing information of the speech signal, and some noise has been removed.

[0066] The extracted single-frame feature vector is input into the fully connected layer. Each node (or neuron) of the fully connected layer is connected to each element in the input predicted pure speech feature, so that global interaction between features can be achieved.

[0067] Inside the fully connected layer, each node calculates the weighted sum of the input predicted clean speech features and adds a bias term. This weighted sum is obtained by performing a dot product operation on the input predicted clean speech features and the node's weight vector.

[0068] Then, the weighted sum is transformed nonlinearly through an activation function to introduce the nonlinear expression ability of the model. Common activation functions include ReLU, sigmoid, and tanh. In this scenario, ReLU or similar activation functions are usually selected because they can effectively process sparse features and accelerate the training process. The specific activation function can be determined according to the actual situation, and this application does not limit this.

[0069] After the nonlinear transformation of the fully connected layer, each node will output a time domain signal sample. These samples constitute the predicted pure speech signal of a single frame. For speech signals with different sampling rates, the number of output nodes of the fully connected layer will be different. For example, for an 8KHz speech signal, each frame contains 80 sampling points, so the number of output nodes of the fully connected layer should be set to 80. Similarly, for a 16KHz signal, the number of output nodes of the fully connected layer should be set to 160.

[0070] By concatenating the output nodes of all frames, we can get the complete predicted pure speech signal. This process is achieved by concatenating single frame signals in the time domain.

[0071] In the above method, the fully connected layer can realize global interaction between features, so that the model can learn more complex feature relationships. Through the nonlinear transformation of the fully connected layer and the splicing of the output nodes, the predicted pure speech features can be converted into time domain signals to achieve the reconstruction of speech signals. 3. Through end-to-end learning, the one-dimensional convolution and the fully connected layer learn the common features between different sampling rates, avoiding the problem that artificial feature extraction is difficult to be compatible with different sampling rates.

[0072] Step S140, obtaining an actual clean speech signal, and training the one-dimensional convolution, the mask prediction model, and the fully connected layer according to the actual clean speech signal and the predicted clean speech signal to obtain a trained one-dimensional convolution, a trained mask prediction model, and a trained fully connected layer.

[0073] In this embodiment, if Figure 2 As shown, the actual clean voice signal includes a first actual clean voice signal and a second actual clean voice signal, wherein the first actual clean voice signal is an 8KHz actual clean voice signal, and the second actual clean voice signal is an 16KHz actual clean voice signal.

[0074] Understandably, the first error between the first actual clean speech signal and the first predicted clean speech signal is calculated, and the second error between the second actual clean speech signal and the second predicted clean speech signal is calculated, and the mean square error or cross entropy loss is usually used as the loss function. The loss function measures the accuracy of the model prediction, and the goal is to minimize this loss.

[0075] The gradient is calculated based on the loss function, and the gradient is propagated from the output layer to the input layer through the back-propagation algorithm. The parameters of the one-dimensional convolution, mask prediction model, and fully connected layer are updated according to the gradient using an optimization algorithm (such as SGD, Adam, etc.). The training process is continuously iterated until the error between the actual clean speech signal and the predicted clean speech signal is less than the preset threshold (or until the preset number of iterations is reached), and the trained one-dimensional convolution, the trained mask prediction model, and the trained fully connected layer are obtained for subsequent deployment and application.

[0076] In the above method, the model performance can be continuously optimized by calculating the error between the actual clean speech signal and the predicted clean speech signal, and using the back propagation algorithm and the optimization algorithm to update the model parameters. After multiple iterations of training, the prediction accuracy of the model will continue to improve, making the predicted clean speech signal closer to the actual clean speech signal. The trained model can better handle unseen noisy speech signals, improving the generalization ability of the model.

[0077] In an optional implementation, during and after training, the one-dimensional convolution, mask prediction model and fully connected layer need to be evaluated to verify their performance and effect. Common evaluation indicators include signal-to-noise ratio, accuracy, recall rate and F1 score, etc., which are not limited in this application.

[0078] In the above method, the trained model is evaluated through evaluation indicators, which can verify the performance and effect of the model and provide a reliable basis for subsequent deployment and application. The evaluation results can guide the further improvement and optimization of the model to improve the performance and accuracy of the model.

[0079] The deep learning-based speech enhancement method provided in the embodiment of the present application obtains a noisy speech signal, extracts features of the noisy speech signal through one-dimensional convolution, and obtains noisy speech features; inputs the noisy speech features into a mask prediction model to obtain a ratio mask, and calculates and predicts clean speech features based on the ratio mask; inputs the predicted clean speech features into a fully connected layer to obtain a predicted clean speech signal; obtains an actual clean speech signal, and trains the one-dimensional convolution, the mask prediction model, and the fully connected layer based on the actual clean speech signal and the predicted clean speech signal to obtain a trained one-dimensional convolution, a trained mask prediction model, and a trained fully connected layer. The present application can adapt to speech enhancement at different sampling rates, avoiding the problem that artificial feature extraction is difficult to be compatible with different sampling rates.

[0080] Example 2

[0081] like Figure 3 , which is a structural diagram of a speech enhancement device 300 based on deep learning in an embodiment of the present application, and the device includes:

[0082] The extraction module 310 is used to obtain a noisy speech signal, and perform feature extraction on the noisy speech signal through one-dimensional convolution to obtain a noisy speech feature;

[0083] A calculation module 320, configured to input the noisy speech feature into a mask prediction model to obtain a ratio mask, and calculate and predict the clean speech feature based on the ratio mask;

[0084] A generating module 330, configured to input the predicted clean speech feature into a fully connected layer to obtain a predicted clean speech signal;

[0085] The training module 340 is used to obtain an actual clean speech signal, and train the one-dimensional convolution, the mask prediction model and the fully connected layer according to the actual clean speech signal and the predicted clean speech signal to obtain a trained one-dimensional convolution, a trained mask prediction model and a trained fully connected layer.

[0086] The deep learning-based speech enhancement device provided in the embodiment of the present application can adapt to speech enhancement at different sampling rates, avoiding the problem that artificial feature extraction is difficult to be compatible with different sampling rates.

[0087] Example 3

[0088] The present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0089] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0090] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0091] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D slot compatibility test memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the slot compatibility test method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0092] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other deep learning-based voice enhancement chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions or process data stored in the memory 41, such as computer-readable instructions for running the slot compatibility test method.

[0093] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0094] The computer device provided in this embodiment can execute the above-mentioned speech enhancement method based on deep learning. The speech enhancement method based on deep learning here can be the speech enhancement method based on deep learning in each of the above-mentioned embodiments.

[0095] Example 4

[0096] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the deep learning-based speech enhancement method in the embodiment are implemented.

[0097] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, referred to as SMC), a secure digital (Secure Digital, referred to as SD) card, a flash card, etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0098] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow chart, and the combination of boxes in the structure diagram and / or the flow chart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0099] In addition, the functional modules or units in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0100] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or partly contributed to the prior art or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium can be a non-volatile storage medium or a volatile storage medium. For example, the storage medium can be: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and other media that can store program codes.

[0101] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A speech enhancement method based on deep learning, characterized in that: The method comprises: Acquire a noisy speech signal, and perform feature extraction on the noisy speech signal through one-dimensional convolution to obtain noisy speech features; Inputting the noisy speech features into a mask prediction model to obtain a ratio mask, and calculating and predicting the clean speech features based on the ratio mask; Inputting the predicted clean speech feature into a fully connected layer to obtain a predicted clean speech signal; An actual clean speech signal is obtained, and according to the actual clean speech signal and the predicted clean speech signal, the one-dimensional convolution, the mask prediction model and the fully connected layer are trained to obtain a trained one-dimensional convolution, a trained mask prediction model and a trained fully connected layer.

2. The deep learning-based speech enhancement method according to claim 1, characterized in that: The noisy speech signal includes a first noisy speech signal and a second noisy speech signal, the one-dimensional convolution includes a first one-dimensional convolution and a second one-dimensional convolution, the noisy speech feature includes a first noisy speech feature and a second noisy speech feature, and the feature extraction of the noisy speech signal by the one-dimensional convolution to obtain the noisy speech feature includes: Obtaining a preset window length and a preset window shift, calculating a first convolution kernel according to the preset window length and the first noisy speech signal, and calculating a first convolution step length according to the preset window shift and the first noisy speech signal; Calculating a second convolution kernel according to the preset window length and the second noisy speech signal, and calculating a second convolution step length according to the preset window shift and the second noisy speech signal; Performing feature extraction on the first noisy speech signal according to the first convolution kernel and the first convolution step size to obtain the first noisy speech feature; Feature extraction is performed on the second noisy speech signal according to the second convolution kernel and the second convolution step size to obtain the second noisy speech feature.

3. The deep learning-based speech enhancement method according to claim 1, characterized in that: The step of inputting the noisy speech feature into a mask prediction model to obtain a ratio mask comprises: The mask prediction model performs forward propagation on the noisy speech feature according to a preset network structure, and outputs the ratio mask, wherein the ratio mask represents the ratio relationship between the noisy speech feature and its corresponding clean speech feature.

4. The deep learning-based speech enhancement method according to claim 1, characterized in that: The step of calculating and predicting the clean speech feature according to the ratio mask comprises: The ratio mask is multiplied element-by-element by the noisy speech feature to obtain the predicted clean speech feature.

5. The deep learning-based speech enhancement method according to claim 1, characterized in that: The fully connected layer includes a first fully connected layer and a second fully connected layer, the predicted clean speech signal includes a first predicted clean speech signal and a second predicted clean speech signal, and the predicted clean speech feature is input into the fully connected layer to obtain the predicted clean speech signal, including: Mapping the predicted clean speech feature to a time domain signal through the first fully connected layer to obtain the first predicted clean speech signal; The predicted clean speech feature is mapped to a time domain signal through the second fully connected layer to obtain the second predicted clean speech signal.

6. The method for speech enhancement based on deep learning according to claim 5, characterized in that: The actual clean speech signal includes a first actual clean speech signal and a second actual clean speech signal, and the one-dimensional convolution, the mask prediction model and the fully connected layer are trained according to the actual clean speech signal and the predicted clean speech signal to obtain the trained one-dimensional convolution, the trained mask prediction model and the trained fully connected layer, including: Calculating a first error between the first actual clean speech signal and the first predicted clean speech signal, and calculating a second error between the second actual clean speech signal and the second predicted clean speech signal; Using a back propagation algorithm to train the one-dimensional convolution, the mask prediction model, and the fully connected layer according to the first error and the second error; The training process is continuously iterated until the error is less than a preset threshold, thereby obtaining the trained one-dimensional convolution, the trained mask prediction model and the trained fully connected layer.

7. The method for speech enhancement based on deep learning according to claim 1, characterized in that: After obtaining the trained one-dimensional convolution, the trained mask prediction model and the trained fully connected layer, the method further includes: The trained one-dimensional convolution, the trained mask prediction model, and the trained fully connected layer are evaluated to obtain evaluation indicators, wherein the evaluation indicators include at least one of a signal-to-noise ratio, an accuracy rate, a recall rate, and an F1 score.

8. A speech enhancement device based on deep learning, characterized in that: The device comprises: An extraction module, used to obtain a noisy speech signal, and perform feature extraction on the noisy speech signal through one-dimensional convolution to obtain a noisy speech feature; A calculation module, used for inputting the noisy speech feature into a mask prediction model to obtain a ratio mask, and calculating and predicting the clean speech feature according to the ratio mask; A generation module, used for inputting the predicted clean speech features into a fully connected layer to obtain a predicted clean speech signal; A training module is used to obtain an actual clean speech signal, and train the one-dimensional convolution, the mask prediction model and the fully connected layer according to the actual clean speech signal and the predicted clean speech signal to obtain a trained one-dimensional convolution, a trained mask prediction model and a trained fully connected layer.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the deep learning-based speech enhancement method described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the deep learning-based speech enhancement method described in any one of claims 1 to 7 are implemented.

Images