A speech enhancement method, device, equipment and medium based on deep learning

Abstract

This invention relates to the field of speech recognition technology, and discloses a speech enhancement method, apparatus, device, and medium based on deep learning. The method includes: acquiring a noisy speech signal; extracting features from the noisy speech signal through one-dimensional convolution to obtain noisy speech features; inputting the noisy speech features into a mask prediction model to obtain a ratio mask; calculating predicted clean speech features based on the ratio mask; inputting the predicted clean speech features into a fully connected layer to obtain a predicted clean speech signal; acquiring an actual clean speech signal; and training the one-dimensional convolution, mask prediction model, and fully connected layer based on the actual clean speech signal and the predicted clean speech signal to obtain trained one-dimensional convolution, trained mask prediction model, and trained fully connected layer. This application can adapt to speech enhancement at different sampling rates, avoiding the problem of manual feature extraction being incompatible with different sampling rates.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a speech enhancement method, apparatus, device, and medium based on deep learning. Background Technology

[0002] Speech enhancement separates noise from human voice in mixed speech, removing as much noise as possible while preserving the human voice as completely as possible. This effectively improves the quality of voice communication or voice interaction, allowing people or machines to hear clear and clean speech even in noisy environments.

[0003] Current deep learning-based noise reduction methods can effectively remove rapidly changing noise and achieve good speech noise reduction results. However, deep learning-based noise reduction methods are heavily dependent on data. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to overcome the shortcomings of the prior art and provide a speech enhancement method, apparatus, device and medium based on deep learning.

[0005] This invention provides the following technical solution:

[0006] In a first aspect, this disclosure provides a speech enhancement method based on deep learning, the method comprising: acquiring a noisy speech signal, and extracting features from the noisy speech signal through one-dimensional convolution to obtain noisy speech features;

[0007] The noisy speech features are input into the mask prediction model to obtain a ratio mask, and the clean speech features are predicted based on the ratio mask.

[0008] The predicted clean speech features are input into the fully connected layer to obtain the predicted clean speech signal;

[0009] Obtain the actual clean speech signal, and train the one-dimensional convolution, the mask prediction model, and the fully connected layer based on the actual clean speech signal and the predicted clean speech signal to obtain the trained one-dimensional convolution, the trained mask prediction model, and the trained fully connected layer.

[0010] Optionally, the noisy speech signal includes a first noisy speech signal and a second noisy speech signal, the one-dimensional convolution includes a first one-dimensional convolution and a second one-dimensional convolution, the noisy speech features include a first noisy speech feature and a second noisy speech feature, and the step of extracting features from the noisy speech signal through one-dimensional convolution to obtain noisy speech features includes:

[0011] Obtain a preset window length and a preset window shift; calculate a first convolution kernel based on the preset window length and the first noisy speech signal; calculate a first convolution stride based on the preset window shift and the first noisy speech signal.

[0012] The second convolution kernel is calculated based on the preset window length and the second noisy speech signal, and the second convolution stride is calculated based on the preset window shift and the second noisy speech signal.

[0013] Based on the first convolution kernel and the first convolution stride, feature extraction is performed on the first noisy speech signal to obtain the first noisy speech features;

[0014] Based on the second convolution kernel and the second convolution stride, feature extraction is performed on the second noisy speech signal to obtain the second noisy speech features.

[0015] Optionally, the step of inputting the noisy speech features into the mask prediction model to obtain the ratio mask includes:

[0016] The mask prediction model performs forward propagation on the noisy speech features according to a preset network structure and outputs the ratio mask, wherein the ratio mask represents the ratio relationship between the noisy speech features and their corresponding clean speech features.

[0017] Optionally, calculating the predicted clean speech features based on the ratio mask includes:

[0018] The ratio mask is multiplied element-wise with the noisy speech features to obtain the predicted clean speech features.

[0019] Optionally, the fully connected layer includes a first fully connected layer and a second fully connected layer, the predicted clean speech signal includes a first predicted clean speech signal and a second predicted clean speech signal, and the step of inputting the predicted clean speech features into the fully connected layer to obtain the predicted clean speech signal includes:

[0020] The predicted clean speech features are mapped onto the time-domain signal through the first fully connected layer to obtain the first predicted clean speech signal.

[0021] The predicted clean speech features are mapped onto the time-domain signal through the second fully connected layer to obtain the second predicted clean speech signal.

[0022] Optionally, the actual clean speech signal includes a first actual clean speech signal and a second actual clean speech signal. The step of training the one-dimensional convolution, the mask prediction model, and the fully connected layer based on the actual clean speech signal and the predicted clean speech signal to obtain the trained one-dimensional convolution, the trained mask prediction model, and the trained fully connected layer includes:

[0023] Calculate the first error between the first actual clean speech signal and the first predicted clean speech signal, and calculate the second error between the second actual clean speech signal and the second predicted clean speech signal;

[0024] The one-dimensional convolution, the mask prediction model, and the fully connected layer are trained using the backpropagation algorithm based on the first error and the second error.

[0025] The training process is iterated until the error is less than a preset threshold, resulting in the trained one-dimensional convolution, the trained mask prediction model, and the trained fully connected layer.

[0026] Optionally, after obtaining the trained one-dimensional convolution, the trained mask prediction model, and the trained fully connected layer, the method further includes:

[0027] The trained one-dimensional convolution, the trained mask prediction model, and the trained fully connected layer are evaluated to obtain evaluation metrics, wherein the evaluation metrics include at least one of signal-to-noise ratio, accuracy, recall, and F1 score.

[0028] Secondly, this disclosure provides a deep learning-based speech enhancement device, the device comprising:

[0029] The extraction module is used to acquire noisy speech signals and extract features from the noisy speech signals through one-dimensional convolution to obtain noisy speech features.

[0030] The calculation module is used to input the noisy speech features into the mask prediction model to obtain a ratio mask, and calculate and predict clean speech features based on the ratio mask.

[0031] The generation module is used to input the predicted clean speech features into the fully connected layer to obtain the predicted clean speech signal;

[0032] The training module is used to acquire the actual clean speech signal, and train the one-dimensional convolution, the mask prediction model and the fully connected layer based on the actual clean speech signal and the predicted clean speech signal, so as to obtain the trained one-dimensional convolution, the trained mask prediction model and the trained fully connected layer.

[0033] Thirdly, this disclosure provides a computer device including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the deep learning-based speech enhancement method described in the first aspect.

[0034] Fourthly, this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the deep learning-based speech enhancement method described in the first aspect.

[0035] The beneficial effects of this application are:

[0036] This application provides a deep learning-based speech enhancement method, which includes: acquiring a noisy speech signal; extracting features from the noisy speech signal using a one-dimensional convolution to obtain noisy speech features; inputting the noisy speech features into a mask prediction model to obtain a ratio mask; calculating predicted clean speech features based on the ratio mask; inputting the predicted clean speech features into a fully connected layer to obtain a predicted clean speech signal; acquiring an actual clean speech signal; and training the one-dimensional convolution, the mask prediction model, and the fully connected layer based on the actual clean speech signal and the predicted clean speech signal to obtain trained one-dimensional convolution, trained mask prediction model, and trained fully connected layer. This application can adapt to speech enhancement at different sampling rates, avoiding the problem of manual feature extraction being incompatible with different sampling rates.

[0037] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the various drawings, similar components are numbered similarly.

[0039] Figure 1 A flowchart of a deep learning-based speech enhancement method provided in an embodiment of this application is shown;

[0040] Figure 2 The diagram illustrates the network structure of a deep learning-based speech enhancement method provided in an embodiment of this application.

[0041] Figure 3 A schematic diagram of the structure of a deep learning-based speech enhancement device provided in an embodiment of this application is shown;

[0042] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0043] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0044] It should be noted that when an element is said to be "fixed" to another element, it can be directly on the other element or there may be an intervening element. When an element is said to be "connected" to another element, it can be directly connected to the other element or there may be an intervening element. Conversely, when an element is said to be "directly" on another element, there is no intervening element. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0045] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0046] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the template description is for the purpose of describing particular embodiments only and is not intended to limit the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0048] Example 1

[0049] like Figure 1 The diagram shown is a flowchart of a deep learning-based speech enhancement method according to an embodiment of this application. The deep learning-based speech enhancement method provided in this embodiment includes the following steps:

[0050] Step S110: Obtain the noisy speech signal, and extract features from the noisy speech signal through one-dimensional convolution to obtain noisy speech features.

[0051] In this embodiment, as Figure 2 As shown, the noisy speech signal includes a first noisy speech signal and a second noisy speech signal; the one-dimensional convolution includes a first one-dimensional convolution and a second one-dimensional convolution; and the noisy speech features include a first noisy speech feature and a second noisy speech feature. Specifically, the first noisy speech signal is an 8kHz noisy speech signal, and the second noisy speech signal is a 16kHz noisy speech signal.

[0052] Next, a preset window length and a preset window shift are obtained. A first convolution kernel is calculated based on the preset window length and the first noisy speech signal. A first convolution step is calculated based on the preset window shift and the first noisy speech signal. A second convolution kernel is calculated based on the preset window length and the second noisy speech signal. A second convolution step is calculated based on the preset window shift and the second noisy speech signal.

[0053] For example, taking the commonly used framing method with a window length of 20ms and a window shift of 10ms, for an 8kHz noisy speech signal, a convolution kernel size of 8000×0.02=160 and a convolution stride of 8000×0.01=80 are used, while for a 16kHz noisy speech signal, a convolution kernel size of 16000×0.02=320 and a convolution stride of 16000×0.01=160 are used.

[0054] Furthermore, feature extraction is performed on the first noisy speech signal based on the first convolution kernel and the first convolution stride to obtain the first noisy speech features; feature extraction is performed on the second noisy speech signal based on the second convolution kernel and the second convolution stride to obtain the second noisy speech features.

[0055] It should be noted that in the above process, the noisy speech signal is used as the input to a one-dimensional convolution. The number of channels in the convolution kernel matches the number of channels in the input signal. For a single microphone signal, the number of input channels is 1. The number of output channels in the convolution kernel determines the dimension of the feature vector. In this embodiment, it can be 256, but in practice, it can be adaptively adjusted according to the computing power. This application does not limit this. After the convolution operation, noisy speech signals with different sampling rates will be uniformly converted into noisy speech feature vectors with a frame rate of 100 and a single frame dimension of 256. For example, a 10-second noisy speech signal will be converted into 1000 frames of 256-dimensional noisy speech feature vectors.

[0056] In the above method, one-dimensional convolution can effectively extract local structure and temporal information from noisy speech signals, providing key features for subsequent processing. After convolution, noisy speech signals with different sampling rates are uniformly converted into feature vectors with the same frame rate and single-frame dimension, facilitating subsequent processing.

[0057] Step S120: Input the noisy speech features into the mask prediction model to obtain a ratio mask, and calculate the predicted clean speech features based on the ratio mask.

[0058] Specifically, such as Figure 2 As shown, the noisy speech features extracted from the noisy speech signal are used as input to the mask prediction model. These features are typically obtained from the original noisy speech signal through one-dimensional convolution operations, and they contain the local structure and temporal information of the noisy speech signal.

[0059] After receiving the input noisy speech features, the mask prediction model performs forward propagation according to the preset network structure. During this process, the noisy speech features pass through each layer of the mask prediction model (such as convolutional layers, pooling layers, etc., depending on the model design) in sequence, and undergo nonlinear transformation and feature extraction layer by layer.

[0060] During the forward propagation of the mask prediction model, a ratio mask with the same dimension as the input noisy speech features is ultimately output. This ratio mask represents the ratio between the noisy speech features and their corresponding clean speech features. Each element in the mask corresponds to an element in the input noisy speech feature vector, representing the relative importance of that element in the clean speech features.

[0061] Furthermore, the ratio mask is multiplied element-wise with the noisy speech features to obtain the predicted clean speech features. This step essentially uses the ratio mask to filter and enhance the noisy speech features, extracting features that are closer to clean speech.

[0062] The above method can effectively suppress noise and extract features that are closer to clean speech. The mask prediction and noise suppression processes can significantly improve the quality of the predicted clean speech signal, making it closer to the actual clean speech signal. Furthermore, it can adapt to speech denoising at different sampling rates. Since the parameters in the branch parts are few, most parameters are shared by the mask prediction model, and adding another sampling rate does not significantly increase the model parameters. Training speech denoising at different sampling rates belongs to training different tasks, and the tasks can assist each other, which helps to suppress overfitting of the mask prediction model and strengthens the model's representation and generalization ability.

[0063] Step S130: Input the predicted clean speech features into the fully connected layer to obtain the predicted clean speech signal.

[0064] In this embodiment, as Figure 2 As shown, the fully connected layer includes a first fully connected layer and a second fully connected layer, and the predicted clean speech signal includes a first predicted clean speech signal and a second predicted clean speech signal. The first predicted clean speech signal is an 8kHz predicted clean speech signal, and the second predicted clean speech signal is a 16kHz predicted clean speech signal.

[0065] Intuitively, feature vectors for a single frame are extracted from the predicted clean speech features output by the mask prediction model. These feature vectors are typically obtained during the previous convolution and masking process; they contain local structural and temporal information of the speech signal and have already had some noise removed.

[0066] The extracted single-frame feature vectors are input into a fully connected layer. Each node (or neuron) in the fully connected layer is connected to every element in the input predicted clean speech features, thus enabling global interaction between features.

[0067] Inside the fully connected layer, each node computes a weighted sum of the input predicted clean speech features, plus a bias term. This weighted sum is obtained by performing a dot product between the input predicted clean speech features and the node's weight vector.

[0068] Then, the weighted sum is nonlinearly transformed through an activation function to introduce the model's nonlinear expressive power. Commonly used activation functions include ReLU, sigmoid, and tanh. In this scenario, ReLU or similar activation functions are usually chosen because they can effectively handle sparse features and accelerate the training process. The specific activation function can be determined according to the actual situation, and this application does not impose any restrictions on it.

[0069] After the nonlinear transformation by the fully connected layer, each node outputs a time-domain signal sample. These samples constitute a single frame of the predicted clean speech signal. The number of output nodes of the fully connected layer will vary depending on the sampling rate of the speech signal. For example, for an 8kHz speech signal, each frame contains 80 sampling points, so the number of output nodes of the fully connected layer should be set to 80. Similarly, for a 16kHz signal, the number of output nodes of the fully connected layer should be set to 160.

[0070] By concatenating the output nodes of all frames, a complete, predictable, clean speech signal can be obtained. This process is achieved by concatenating the single-frame signals in the time domain.

[0071] In the above method, the fully connected layer enables global interaction between features, allowing the model to learn more complex feature relationships. Through the nonlinear transformation of the fully connected layer and the concatenation of output nodes, the predicted clean speech features can be converted into time-domain signals, achieving speech signal reconstruction. 3. By using an end-to-end learning approach, one-dimensional convolutional and fully connected layers learn common features across different sampling rates, avoiding the problem of manual feature extraction being incompatible with different sampling rates.

[0072] Step S140: Obtain the actual clean speech signal; based on the actual clean speech signal and the predicted clean speech signal, train the one-dimensional convolution, the mask prediction model, and the fully connected layer to obtain the trained one-dimensional convolution, the trained mask prediction model, and the trained fully connected layer.

[0073] In this embodiment, as Figure 2 As shown, the actual clean speech signal includes a first actual clean speech signal and a second actual clean speech signal, wherein the first actual clean speech signal is an 8kHz actual clean speech signal and the second actual clean speech signal is a 16kHz actual clean speech signal.

[0074] Understandably, a first error is calculated between the first actual clean speech signal and the first predicted clean speech signal, and a second error is calculated between the second actual clean speech signal and the second predicted clean speech signal. Mean squared error or cross-entropy loss is typically used as the loss function. The loss function measures the accuracy of the model's predictions, and the goal is to minimize this loss.

[0075] The gradient is calculated based on the loss function and propagated from the output layer to the input layer using the backpropagation algorithm. Optimization algorithms (such as SGD, Adam, etc.) are then used to update the parameters of the one-dimensional convolutional layer, the mask prediction model, and the fully connected layer based on the gradient. The training process is iterated until the error between the actual clean speech signal and the predicted clean speech signal is less than a preset threshold (or until a preset number of iterations is reached). This yields the trained one-dimensional convolutional layer, the trained mask prediction model, and the trained fully connected layer, which are then used for subsequent deployment and application.

[0076] The above method calculates the error between the actual clean speech signal and the predicted clean speech signal, and uses backpropagation and optimization algorithms to update the model parameters, continuously optimizing model performance. After multiple iterations of training, the model's prediction accuracy continuously improves, making the predicted clean speech signal closer to the actual clean speech signal. The trained model can better handle unseen noisy speech signals, improving the model's generalization ability.

[0077] In one alternative implementation, the one-dimensional convolution, mask prediction model, and fully connected layer need to be evaluated during and after training to verify their performance and effectiveness. Commonly used evaluation metrics include signal-to-noise ratio, accuracy, recall, and F1 score, which are not limited in this application.

[0078] The above method evaluates the trained model using assessment metrics, which verifies the model's performance and effectiveness, providing a reliable basis for subsequent deployment and application. The evaluation results can guide further improvement and optimization of the model to enhance its performance and accuracy.

[0079] The deep learning-based speech enhancement method provided in this application involves acquiring a noisy speech signal, extracting features from the noisy speech signal using a one-dimensional convolution to obtain noisy speech features, inputting the noisy speech features into a mask prediction model to obtain a ratio mask, calculating predicted clean speech features based on the ratio mask, inputting the predicted clean speech features into a fully connected layer to obtain a predicted clean speech signal, acquiring an actual clean speech signal, and training the one-dimensional convolution, the mask prediction model, and the fully connected layer based on the actual clean speech signal and the predicted clean speech signal to obtain trained one-dimensional convolution, trained mask prediction model, and trained fully connected layer. This application can adapt to speech enhancement at different sampling rates, avoiding the problem of manual feature extraction being difficult to be compatible with different sampling rates.

[0080] Example 2

[0081] like Figure 3 The diagram shown is a structural schematic of a deep learning-based speech enhancement device 300 according to an embodiment of this application. The device includes:

[0082] Extraction module 310 is used to acquire noisy speech signals and extract features from the noisy speech signals by one-dimensional convolution to obtain noisy speech features;

[0083] The calculation module 320 is used to input the noisy speech features into the mask prediction model to obtain a ratio mask, and calculate and predict clean speech features based on the ratio mask.

[0084] The generation module 330 is used to input the predicted clean speech features into the fully connected layer to obtain the predicted clean speech signal;

[0085] The training module 340 is used to acquire the actual clean speech signal, and train the one-dimensional convolution, the mask prediction model and the fully connected layer based on the actual clean speech signal and the predicted clean speech signal, so as to obtain the trained one-dimensional convolution, the trained mask prediction model and the trained fully connected layer.

[0086] The deep learning-based speech enhancement device provided in this application can adapt to speech enhancement at different sampling rates, avoiding the problem that manual feature extraction is difficult to be compatible with different sampling rates.

[0087] Example 3

[0088] This application also provides a computer device. Please refer to the following for details. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0089] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with memory 41, processor 42, and network interface 43 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0090] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0091] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or D slot compatibility test memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for slot compatibility testing methods. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0092] In some embodiments, the processor 42 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other deep learning-based speech enhancement chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or process data, such as executing computer-readable instructions for the slot compatibility testing method.

[0093] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0094] The computer device provided in this embodiment can execute the deep learning-based speech enhancement method described above. Here, the deep learning-based speech enhancement method can be any of the deep learning-based speech enhancement methods described in the various embodiments above.

[0095] Example 4

[0096] This embodiment also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the deep learning-based speech enhancement method in this embodiment.

[0097] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, as an alternative implementation, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0099] In addition, the functional modules or units in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0100] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium can be a non-volatile storage medium or a volatile storage medium. For example, the storage medium can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code.

[0101] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A speech enhancement method based on deep learning, characterized in that, The method includes: Noisy speech signals are acquired, and features are extracted from the noisy speech signals through one-dimensional convolution to obtain noisy speech features; The noisy speech features are input into the mask prediction model to obtain a ratio mask, and the clean speech features are predicted based on the ratio mask. The predicted clean speech features are input into the fully connected layer to obtain the predicted clean speech signal; Obtain the actual clean speech signal, and train the one-dimensional convolution, the mask prediction model, and the fully connected layer based on the actual clean speech signal and the predicted clean speech signal to obtain the trained one-dimensional convolution, the trained mask prediction model, and the trained fully connected layer. The noisy speech signal includes a first noisy speech signal and a second noisy speech signal. The first noisy speech signal is an 8kHz noisy speech signal, and the second noisy speech signal is a 16kHz noisy speech signal. The one-dimensional convolution includes a first one-dimensional convolution and a second one-dimensional convolution. The noisy speech features include a first noisy speech feature and a second noisy speech feature. The step of extracting features from the noisy speech signal using one-dimensional convolution to obtain noisy speech features includes: Obtain a preset window length and a preset window shift; calculate a first convolution kernel based on the preset window length and the first noisy speech signal; calculate a first convolution stride based on the preset window shift and the first noisy speech signal. The second convolution kernel is calculated based on the preset window length and the second noisy speech signal, and the second convolution stride is calculated based on the preset window shift and the second noisy speech signal. Based on the first convolution kernel and the first convolution stride, feature extraction is performed on the first noisy speech signal to obtain the first noisy speech features; Based on the second convolution kernel and the second convolution stride, feature extraction is performed on the second noisy speech signal to obtain the second noisy speech features.

2. The speech enhancement method based on deep learning according to claim 1, characterized in that, The step of inputting the noisy speech features into the mask prediction model to obtain the ratio mask includes: The mask prediction model performs forward propagation on the noisy speech features according to a preset network structure and outputs the ratio mask, wherein the ratio mask represents the ratio relationship between the noisy speech features and their corresponding clean speech features.

3. The speech enhancement method based on deep learning according to claim 1, characterized in that, The step of calculating and predicting clean speech features based on the ratio mask includes: The ratio mask is multiplied element-wise with the noisy speech features to obtain the predicted clean speech features.

4. The speech enhancement method based on deep learning according to claim 1, characterized in that, The fully connected layer includes a first fully connected layer and a second fully connected layer. The predicted clean speech signal includes a first predicted clean speech signal and a second predicted clean speech signal. The step of inputting the predicted clean speech features into the fully connected layer to obtain the predicted clean speech signal includes: The predicted clean speech features are mapped onto the time-domain signal through the first fully connected layer to obtain the first predicted clean speech signal. The predicted clean speech features are mapped onto the time-domain signal through the second fully connected layer to obtain the second predicted clean speech signal.

5. The deep learning-based speech enhancement method according to claim 4, characterized in that, The actual clean speech signal includes a first actual clean speech signal and a second actual clean speech signal. The step of training the one-dimensional convolution, the mask prediction model, and the fully connected layer based on the actual clean speech signal and the predicted clean speech signal to obtain the trained one-dimensional convolution, the trained mask prediction model, and the trained fully connected layer includes: Calculate the first error between the first actual clean speech signal and the first predicted clean speech signal, and calculate the second error between the second actual clean speech signal and the second predicted clean speech signal; The one-dimensional convolution, the mask prediction model, and the fully connected layer are trained using the backpropagation algorithm based on the first error and the second error. The training process is iterated until the error is less than a preset threshold, resulting in the trained one-dimensional convolution, the trained mask prediction model, and the trained fully connected layer.

6. The speech enhancement method based on deep learning according to claim 1, characterized in that, After obtaining the trained one-dimensional convolution, the trained mask prediction model, and the trained fully connected layer, the process further includes: The trained one-dimensional convolution, the trained mask prediction model, and the trained fully connected layer are evaluated to obtain evaluation metrics, wherein the evaluation metrics include at least one of signal-to-noise ratio, accuracy, recall, and F1 score.

7. A speech enhancement device based on deep learning, characterized in that, The device includes: The extraction module is used to acquire noisy speech signals and extract features from the noisy speech signals through one-dimensional convolution to obtain noisy speech features. The calculation module is used to input the noisy speech features into the mask prediction model to obtain a ratio mask, and calculate and predict clean speech features based on the ratio mask. The generation module is used to input the predicted clean speech features into the fully connected layer to obtain the predicted clean speech signal; The training module is used to acquire the actual clean speech signal, and train the one-dimensional convolution, the mask prediction model and the fully connected layer based on the actual clean speech signal and the predicted clean speech signal, so as to obtain the trained one-dimensional convolution, the trained mask prediction model and the trained fully connected layer. The noisy speech signal includes a first noisy speech signal and a second noisy speech signal. The first noisy speech signal is an 8kHz noisy speech signal, and the second noisy speech signal is a 16kHz noisy speech signal. The one-dimensional convolution includes a first one-dimensional convolution and a second one-dimensional convolution. The noisy speech features include a first noisy speech feature and a second noisy speech feature. The step of extracting features from the noisy speech signal using one-dimensional convolution to obtain noisy speech features includes: Obtain a preset window length and a preset window shift; calculate a first convolution kernel based on the preset window length and the first noisy speech signal; calculate a first convolution stride based on the preset window shift and the first noisy speech signal. The second convolution kernel is calculated based on the preset window length and the second noisy speech signal, and the second convolution stride is calculated based on the preset window shift and the second noisy speech signal. Based on the first convolution kernel and the first convolution stride, feature extraction is performed on the first noisy speech signal to obtain the first noisy speech features; Based on the second convolution kernel and the second convolution stride, feature extraction is performed on the second noisy speech signal to obtain the second noisy speech features.

8. A computer device, characterized in that, The device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the deep learning-based speech enhancement method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the deep learning-based speech enhancement method according to any one of claims 1-6.

Images