System and method for processing an audio input signal
By combining a microphone, controller, and communication link system with linear noise reduction, nonlinear post-filtering, and deep neural network feature recovery algorithms, the problem of environmental noise when the distance between the speaker and the microphone is far is solved, thereby improving speech intelligibility and signal-to-noise ratio.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GM GLOBAL TECHNOLOGY OPERATIONS LLC
- Filing Date
- 2022-10-17
- Publication Date
- 2026-07-14
AI Technical Summary
Existing speech processing systems struggle to effectively reduce ambient noise and improve speech intelligibility when speakers are located in enclosed rooms and far from microphones. Furthermore, existing noise reduction strategies require precise tuning, making it difficult to achieve the desired signal-to-noise ratio.
The system employs a microphone, controller, and communication link, combining linear noise reduction filtering, nonlinear post-filtering, and feature recovery algorithms based on deep neural networks, including STFT, multiple convolutional layers, LSTM layers, transposed convolutional layers, and ISTFT, to process audio input signals to generate clear audio output.
It effectively reduces audible ambient noise, improves speech intelligibility, simplifies the need for system tuning, and enhances speech quality and signal-to-noise ratio.
Smart Images
Figure CN116597850B_ABST
Abstract
Description
Background Technology
[0001] Voice processing systems include hands-free, speakerphone-like systems such as smartphones, video conferencing systems, laptops, and tablets. In some systems, the speaker may be located in an enclosed room and at a relatively large distance from the microphone. This arrangement can introduce ambient noise, including surrounding noise, interference, and reverberation. This arrangement can lead to acoustic signal processing challenges that affect sound quality and the associated signal-to-noise ratio (SNR).
[0002] Speech processing techniques such as Automatic Speech Recognition (ASR) and teleconferencing typically incorporate noise reduction strategies and systems to reduce audible ambient noise levels and improve speech intelligibility. Noise reduction systems can include linear noise reduction algorithms, nonlinear post-filtering algorithms, and others. The performance of linear noise reduction algorithms may be insufficient to achieve the desired signal-to-noise ratio (SNR) target. Nonlinear post-filtering (PF) algorithms arranged in series with linear noise reduction algorithms can enhance noise reduction levels, but there is a trade-off between residual noise and speech distortion levels. Because spectral subtraction algorithms can be used in the PF module, removing speech features from the signal can cause sound distortion. Such systems require precise tuning to achieve the target SNR with minimal speech distortion, which can be difficult to achieve.
[0003] Therefore, there is a need for improved methods and systems for speech processing, which include noise reduction strategies that reduce the level of audible ambient noise, improve speech intelligibility, and reduce the need for precise tuning. Summary of the Invention
[0004] The concepts described herein provide methods, apparatus, and systems for speech processing, including noise reduction strategies to reduce audible ambient noise levels and improve speech intelligibility.
[0005] The concept includes a system for processing audio input signals, comprising a microphone, a controller, and a communication link connectable to a remote speaker. The microphone is configured to capture and generate the audio input signal and transmit it to the controller, which is connected to the communication link. The controller includes executable code to generate a first result based on the audio input signal via a linear noise reduction filtering algorithm, and a second result based on the first result via a nonlinear post-filtering algorithm. An audio output signal is generated based on the second result using a feature recovery algorithm. The audio output signal is transmitted via the communication link to a speaker, possibly at a remote location.
[0006] One aspect of this disclosure includes a feature recovery algorithm based on a deep neural network (DNN) module comprising: an STFT (Short Time Fourier Transform) layer; multiple convolutional layers; a first LSTM (Long Short Short Time Memory) layer; a second LSTM layer; a dense layer; multiple transposed convolutional layers; and an ISTFT (Inverse Short Time Fourier Transform) layer.
[0007] Another aspect of this disclosure includes STFT transforming an audio input signal from the amplitude domain to the frequency domain.
[0008] Another aspect of this disclosure includes STFT transforming an audio input signal into the frequency domain as a 2-channel sequence with real and imaginary parts.
[0009] Another aspect of this disclosure includes a plurality of convolutional layers: a first convolutional layer having a 2-channel input with 256 features and a 32-channel output with 128 features; a second convolutional layer having a 32-channel input with 128 features and a 64-channel output with 64 features; a third convolutional layer having a 64-channel input with 64 features and a 128-channel output with 32 features; a fourth convolutional layer having a 128-channel input with 32 features and a 128-channel output with 16 features; a fifth convolutional layer having a 128-channel input with 16 features and a 256-channel output with 8 features; and a sixth convolutional layer having a 256-channel input with 8 features and a 256-channel output with 4 features.
[0010] Another aspect of this disclosure includes providing a 256-channel output with four features from the sixth convolutional layer as input to the first LSTM layer.
[0011] Another aspect of this disclosure includes each of a plurality of convolutional layers having a convolutional kernel of size (2, 9) and a stride of size (1, 2).
[0012] Another aspect of this disclosure includes the input of the first convolutional layer being provided as input to ISTFT.
[0013] Another aspect of this disclosure includes the output of the sixth convolutional layer being provided as the input to the first LSTM layer.
[0014] Another aspect of this disclosure includes a first LSTM layer with 256 states.
[0015] Another aspect of this disclosure includes a second LSTM layer with 256 states.
[0016] Another aspect of this disclosure includes the output of the second LSTM layer being provided as input to the dense layer.
[0017] Another aspect of this disclosure includes a plurality of transposed convolutional layers having a sixth transposed convolutional layer having a 512-channel input with 4 features and a 256-channel output with 8 features; a fifth transposed convolutional layer having a 512-channel input with 8 features and a 128-channel output with 16 features; a fourth transposed convolutional layer having a 256-channel input with 16 features and a 128-channel output with 32 features; a third transposed convolutional layer having a 256-channel input with 32 features and a 64-channel output with 64 features; a second transposed convolutional layer having a 128-channel input with 64 features and a 32-channel output with 128 features; and a first transposed convolutional layer having a 64-channel input with 128 features and a 2-channel output with 256 features.
[0018] Another aspect of this disclosure includes the output of the dense layer being provided as the input to the sixth transposed convolutional layer.
[0019] Another aspect of this disclosure includes each of a plurality of transposed convolutional layers having a convolutional kernel of size (2, 9) and a stride of size (1, 2).
[0020] Another aspect of this disclosure includes providing the output of the first transposed convolutional layer as input to ISTFT to achieve feature recovery.
[0021] Another aspect of this disclosure includes the output of the first convolutional layer being provided as the input of the first transposed convolutional layer.
[0022] Another aspect of this disclosure includes providing the output of the second convolutional layer as the input of the second transposed convolutional layer.
[0023] Another aspect of this disclosure includes the output of the third convolutional layer being provided as the input of the third transposed convolutional layer.
[0024] Another aspect of this disclosure includes the output of the fourth convolutional layer being provided as the input of the fourth transposed convolutional layer.
[0025] Another aspect of this disclosure includes the output of the fifth convolutional layer being provided as the input of the fifth transposed convolutional layer.
[0026] Another aspect of this disclosure includes the output of the sixth convolutional layer being provided as the input of the sixth transposed convolutional layer.
[0027] Another aspect of this disclosure includes ISTFT transforming a transposed audio input signal, which is combined with the output of a first transposed convolutional layer, from the frequency domain to the amplitude domain to generate an audio output signal.
[0028] Another aspect of this disclosure includes a method for processing an audio input signal, the method comprising: capturing the audio input signal via a microphone; subjecting the audio input signal to a linear noise reduction filtering algorithm to generate a first result; subjecting the first result to a nonlinear post-filtering algorithm to generate a second result; generating an audio output signal by subjecting the second result to a feature recovery algorithm; and controlling a speaker in response to the audio output signal.
[0029] Another aspect of this disclosure includes a system for processing voice input, comprising a microphone, a controller, and a speaker, wherein the microphone is configured to capture a voice input signal and transmit the voice input signal to the controller; and wherein the controller is operatively connected to the speaker. The controller includes executable code to subject the voice input signal to a linear noise reduction filtering algorithm to generate a first result; to subject the first result to a nonlinear post-filtering algorithm to generate a second result; to generate an audio output signal by subjecting the second result to a feature recovery algorithm; and to control the speaker in response to the voice output signal.
[0030] This invention includes the following technical solutions:
[0031] Option 1. A system for processing audio input signals, the system comprising:
[0032] Microphone, controller, data storage, and communication link to remote audio speakers;
[0033] The microphone is configured to capture and generate the audio input signal and transmit the audio input signal to the controller;
[0034] The controller is operably connected to the communication link; and
[0035] The controller includes executable code to:
[0036] The data storage includes instructions executable by the controller, the instructions including:
[0037] A first result is generated based on the audio input signal using a linear noise reduction filtering algorithm;
[0038] A second result is generated based on the first result using a nonlinear post-filtering algorithm;
[0039] An audio output signal is generated based on the second result using a feature recovery algorithm; and
[0040] The audio output signal is transmitted to the remote audio speaker via the communication link.
[0041] Solution 2. According to the system described in Solution 1, wherein the feature recovery algorithm includes a module based on a deep neural network (DNN), the module including: STFT (Short Time Fourier Transform); multiple convolutional layers; a first LSTM (Long Short Time Memory) layer; a second LSTM layer; a dense layer; multiple transposed convolutional layers; and inverse STFT (ISTFT).
[0042] Option 3. The system according to Option 2, wherein the STFT transforms the audio input signal from the amplitude domain to the frequency domain.
[0043] Option 4. The system according to Option 3, wherein the STFT transforms the audio input signal into a frequency domain having a 2-channel sequence having a real part and an imaginary part.
[0044] Option 5. The system according to Option 2, wherein the plurality of convolutional layers comprises:
[0045] The first convolutional layer has a 2-channel input with 256 features and a 32-channel output with 128 features;
[0046] The second convolutional layer has a 32-channel input with 128 features and a 64-channel output with 64 features;
[0047] The third convolutional layer has a 64-channel input with 64 features and a 128-channel output with 32 features;
[0048] The fourth convolutional layer has a 128-channel input with 32 features and a 128-channel output with 16 features;
[0049] The fifth convolutional layer has a 128-channel input with 16 features and a 258-channel output with 8 features; and
[0050] The sixth convolutional layer has a 256-channel input with 8 features and a 256-channel output with 4 features.
[0051] Option 6. The system according to Option 5, wherein a 256-channel output with 4 features from the sixth convolutional layer is provided as the input to the first LSTM layer.
[0052] Option 7. The system according to Option 5, wherein each of the plurality of convolutional layers has a convolutional kernel of size (2, 9) and a stride of size (1, 2).
[0053] Solution 8. The system according to Solution 5, wherein the output of the first convolutional layer is provided as the input of the ISTFT.
[0054] Solution 9. The system according to Solution 5, wherein the output of the sixth convolutional layer is provided as the input of the first LSTM layer.
[0055] Option 10. The system according to Option 2, wherein the first LSTM layer has 256 states.
[0056] Option 11. The system according to Option 2, wherein the second LSTM layer has 256 states.
[0057] Option 12. The system implemented according to Option 2, wherein the plurality of transposed convolutional layers include a sixth transposed convolutional layer having a 512-channel input with 4 features and a 256-channel output with 8 features; a fifth transposed convolutional layer having a 512-channel input with 8 features and a 128-channel output with 16 features; a fourth transposed convolutional layer having a 256-channel input with 16 features and a 128-channel output with 32 features; a third transposed convolutional layer having a 256-channel input with 32 features and a 64-channel output with 64 features; a second transposed convolutional layer having a 128-channel input with 64 features and a 32-channel output with 128 features; and a first transposed convolutional layer having a 64-channel input with 128 features and a 2-channel output with 256 features.
[0058] Option 13. The system according to Option 12, wherein the output of the dense layer is provided as the input of the sixth transposed convolutional layer.
[0059] Option 14. The system according to Option 12, wherein each of the plurality of transposed convolutional layers has a convolutional kernel of size (2, 9) and a stride of size (1, 2).
[0060] Solution 15. The system according to Solution 12, wherein the output of the first transposed convolutional layer is provided as input to the ISTFT to achieve feature recovery.
[0061] Solution 16. The system according to Solution 15, wherein the ISTFT transforms the audio input signal transposed to the frequency domain in combination with the output of the first transposed convolutional layer from the frequency domain to the amplitude domain to generate the audio output signal.
[0062] Option 17. A method for processing an audio input signal, the method comprising:
[0063] Audio input signals are captured via a microphone;
[0064] The audio input signal is subjected to a linear noise reduction filtering algorithm to generate a first result;
[0065] The first result is subjected to a nonlinear post-filtering algorithm to generate the second result;
[0066] An audio output signal is generated by subjecting the second result to a feature recovery algorithm; and
[0067] The speaker is controlled in response to the audio output signal.
[0068] Scheme 18. The method according to Scheme 17, wherein the feature recovery algorithm includes a module based on a deep neural network (DNN), the module including: an STFT (Short Time Fourier Transform) layer, multiple convolutional layers, a first Long Short-Term Memory (LSTM) layer, a second LSTM layer, a dense layer, multiple transposed convolutional layers, and an inverse STFT (ISTFT) layer.
[0069] Option 19. A system for processing voice input, the system comprising:
[0070] Microphone, controller, and speaker;
[0071] The microphone is configured to capture a voice input signal and transmit the voice input signal to the controller; and the controller is operatively connected to the speaker.
[0072] The controller includes executable code to:
[0073] The voice input signal is subjected to a linear noise reduction filtering algorithm to generate a first result;
[0074] The first result is subjected to a nonlinear post-filtering algorithm to generate the second result;
[0075] The speech output signal is generated by subjecting the second result to a feature recovery algorithm; and
[0076] The speaker is controlled in response to the voice output signal.
[0077] Solution 20. The system according to Solution 19, wherein the feature recovery algorithm includes a module based on a deep neural network (DNN), the module including: an STFT (Short Time Fourier Transform) layer, multiple convolutional layers, a first Long Short Time Memory (LSTM) layer, a second LSTM layer, a dense layer, multiple transposed convolutional layers, and an inverse STFT (ISTFT) layer.
[0078] The foregoing summary is not intended to represent every possible implementation or aspect of this disclosure. Instead, the foregoing summary is intended to illustrate some novel aspects and features disclosed herein. The foregoing features and advantages, as well as other features and advantages of this disclosure, will readily become apparent from the following detailed description of representative embodiments and modes for carrying out this disclosure, when taken in conjunction with the accompanying drawings and claims. Attached Figure Description
[0079] One or more embodiments will now be described by way of example with reference to the accompanying drawings, wherein:
[0080] Figure 1 The microphone, controller, and communication link that can be connected to a remote speaker are schematically shown according to this disclosure;
[0081] Figure 2 Elements of a noise reduction routine for processing audio input signals according to this disclosure are illustrated schematically.
[0082] Figure 3 Elements of a feature recovery algorithm according to this disclosure are schematically shown, including a deep neural network (DNN) module for processing audio input signals as part of a noise reduction routine.
[0083] Figure 4 Elements associated with a training module for training a deep neural network (DNN) module to process audio input signals, according to this disclosure, are illustrated schematically.
[0084] The accompanying drawings are not necessarily drawn to scale and may present slightly simplified representations of various preferred elements of the present disclosure as disclosed herein, including, for example, specific dimensions, orientations, positions, and shapes. Details relating to these elements will be determined in part by the specific intended application and environment of use. Detailed Implementation
[0085] As described and illustrated herein, the components of the disclosed embodiments can be arranged and designed in a variety of different configurations. Therefore, the following detailed description is not intended to limit the scope of this disclosure as claimed, but merely to represent possible embodiments thereof. Furthermore, while numerous specific details are set forth in the following description to provide a thorough understanding of the embodiments disclosed herein, some embodiments may be practiced without some of these details. Moreover, for clarity, certain technical materials understood in the related art have not been described in detail to avoid unnecessarily obscuring this disclosure. Throughout the drawings, corresponding reference numerals denote the same or corresponding parts and elements. Furthermore, the contents of this disclosure as illustrated and described herein can be practiced without the presence of elements not specifically disclosed herein. Furthermore, it is not intended to be bound by any express or implied theory presented herein.
[0086] As used herein, the term "system" can refer to one or a combination of mechanical and electrical actuators, sensors, controllers, application-specific integrated circuits (ASICs), combinational logic circuits, software, firmware, and / or other components arranged to provide the described functions. Implementations can be described herein based on functional and / or logical block components and various processing steps. It should be understood that such block components can be implemented by any number, combination, or collection of mechanical and electrical hardware, software, and / or firmware components configured to perform specified functions and / or routines. For the sake of brevity, conventional components and techniques, as well as other functional aspects of the system (and its individual operating components), may not be described in detail herein. Furthermore, the connecting lines shown in the figures included herein are intended to represent exemplary functional relationships and / or physical connections between elements. It should be noted that many alternative or additional functional relationships or physical connections may alternatively exist.
[0087] The use of ordinal numbers such as first, second, and third does not necessarily imply a sequential order, but rather can distinguish between multiple instances of an action or structure.
[0088] The accompanying drawings are now provided for the purpose of illustrating certain exemplary embodiments and not for the purpose of limiting these embodiments. Figure 1 System 100 is schematically shown, including a microphone 20 and a controller 10 capable of communicating with a remote audio speaker 70 via a communication link 60. In one embodiment, the remote audio speaker 70 is located outside system 100. System 100 includes noise reduction routines 200 for managing audio input signals 15 to reduce audible ambient noise levels and improve speech intelligibility. The term "speech intelligibility" refers to speech clarity, that is, the degree to which spoken sounds can be correctly identified and understood by a listener.
[0089] Microphone 20 can be any device including a transducer capable of converting audible sound into an electrical signal in the form of audio input signal 15. Communication link 60 can be a direct wired point-to-point link, a network communication bus link, a wireless link, or another communication link.
[0090] The controller 10 includes a receiver 30, a processor 40, and a memory 50, wherein the memory 50 includes an implementation of the noise reduction routine 200 and provides data storage.
[0091] The term "controller" and related terms refer to one or various combinations thereof of associated transient and non-transitory memory components, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), electronic circuits, central processing units (e.g., microprocessors), and memory and data storage devices (read-only, programmable read-only, random access, hard disk drives, etc.). Non-transitory memory components are capable of storing machine-readable instructions in the form of one or more software or firmware programs or routines, combinational logic circuits, input / output circuits and devices, signal conditioning, buffering circuits, and other components, which can be accessed and executed by one or more processors to provide the described function. Input / output circuits and devices include analog-to-digital converters and associated devices for monitoring inputs from sensors, wherein such inputs are monitored at a preset sampling frequency or in response to a triggering event. Software, firmware, programs, instructions, control routines, code, algorithms, and similar terms refer to the set of instructions executable by the controller, including calibration and lookup tables. Each controller executes multiple control routines to provide the desired function. Routines can be executed at regular intervals, such as once every 100 microseconds during ongoing operation. Alternatively, routines can be executed in response to the occurrence of a triggering event. Communication between the controller, actuators and / or sensors, and the remote audio speaker 70 can be achieved using a direct wired point-to-point link, a networked communication bus link, a wireless link, or another communication link. Communication includes the exchange of data signals, including, for example, electrical signals via a conductive medium; electromagnetic signals via air; optical signals via an optical waveguide, etc. Data signals can include discrete, analog, and / or digitized analog signals representing inputs from sensors, actuator commands, and communication between the controller.
[0092] The term "signal" refers to a physically identifiable indicator that conveys information and can be a suitable waveform (e.g., electrical, optical, magnetic, mechanical, or electromagnetic) that can travel through a medium, such as DC, AC, sine wave, triangle wave, square wave, vibration, etc.
[0093] Figure 2 Elements of a noise reduction routine 200 for processing audio input signal 15 are schematically shown, including a linear noise reduction algorithm 210, a nonlinear post-filtering algorithm 240, and a feature recovery algorithm 300.
[0094] The linear noise reduction algorithm 210 includes acoustic echo cancellation (AEC) 220 and beamforming (BF) 230. AEC 220 is a digital signal processing technique for identifying and eliminating acoustic echoes, which has been simplified for implementation as an algorithm. BF 230 is a digital signal processing technique that uses spatial information to reduce the power of ambient noise, thereby improving the power ratio between the desired signal and noise. In one implementation, as shown, AEC 220 is positioned before BF 230. Alternatively, BF 230 may be positioned before AEC 220. Acoustic echo cancellation and beamforming are acoustic signal processing techniques known to those skilled in the art.
[0095] Linear noise reduction algorithm 210 generates a first result signal 235, which is provided as input to nonlinear post-filtering (NLP) algorithm 240. NLP algorithm 240 enhances the noise reduction level by employing nonlinear filtering to reduce residual noise and echo. NLP is an acoustic signal processing technique known to skilled technicians.
[0096] NLP algorithm 240 generates a second result signal 245, which is provided as input to feature recovery algorithm 300. Feature recovery algorithm 300 generates an audio output signal 55 based on the second result signal 245. The DNN-based feature recovery algorithm 300 is placed after the post-filtering module to simplify tuning and improve speech quality.
[0097] Figure 3 Elements of a feature recovery algorithm 300 for processing the audio input signal 15, which is part of a noise reduction routine 200, are schematically shown. The feature recovery algorithm 300 consists of a deep neural network (DNN) module, which includes a short-time Fourier transform (STFT) layer 310, multiple convolutional layers 320, a first long short-term memory (LSTM) layer 330, a second LSTM layer 332, a dense layer 340, multiple transposed convolutional layers 350, and an ISTFT layer 370.
[0098] Each of the STFT layer 310 and ISTFT layer 370 is a Fourier transform sequence of the windowed signal, providing time-localized frequency information about how the signal's frequency components change over time. RNNs (Recurrent Neural Networks) are time-series versions of artificial neural networks, or ANNs, arranged to process data sequences such as sound. RNN-based DNNs utilize the strong correlation between speech time and frequency in speech processing for noise reduction and blind source separation. This capability can be used for recovery problems, leading to simplified tuning of post-filtering modules at lower ambient noise levels to achieve improved speech quality in terms of speech intelligibility.
[0099] The first Long Short-Term Memory (LSTM) layer 330 and the second LSTM layer 332 are a class of recurrent neural networks commonly used for tasks such as text-to-speech or natural language processing. They have a cyclic state that is updated each time new data is fed through the network. Thus, the LSTM layer has a memory.
[0100] STFT layer 310 transforms the audio input signal 15 from the amplitude domain to the frequency domain, which is in the form of a 2-channel sequence with real and imaginary parts.
[0101] In one embodiment, the plurality of convolutional layers 320 include a first convolutional layer 321 having a 2-channel input with 256 features and a 32-channel output with 128 features; a second convolutional layer 322 having a 32-channel input with 128 features and a 64-channel output with 64 features; a third convolutional layer 323 having a 64-channel input with 64 features and a 128-channel output with 32 features; a fourth convolutional layer 324 having a 128-channel input with 32 features and a 128-channel output with 16 features; a fifth convolutional layer 325 having a 128-channel input with 16 features and a 256-channel output with 8 features; and a sixth convolutional layer 326 having a 256-channel input with 8 features and a 256-channel output with 4 features.
[0102] In one implementation, each of the plurality of convolutional layers 320 has a convolutional kernel of size (2, 9) and a stride of size (1, 2). The convolutional kernel is a filter used to extract features from the data and is a matrix that moves over the input data, performs a dot product with a subregion of the input data, and has an output as a dot product matrix. The stride controls how the filter convolves around the input volume.
[0103] The 256-channel output (327) with 4 features from the sixth convolutional layer 326 is provided as the input to the first LSTM layer 330 with 256 states.
[0104] The input of the first convolutional layer 321 is provided as the input of the ISTFT layer 370.
[0105] The output of the first LSTM layer 330 is provided as the input of the second LSTM layer 332, and the output of the second LSTM layer 332 is provided as the input of the dense layer 340.
[0106] The output of the dense layer 340 is provided as the input (357) to multiple transposed convolutional layers 350, particularly the sixth convolutional layer 326.
[0107] The multiple transposed convolutional layers 350 include a sixth transposed convolutional layer 356 having a 512-channel input with 4 features and a 256-channel output with 8 features; a fifth transposed convolutional layer 355 having a 512-channel input with 8 features and a 128-channel output with 16 features; a fourth transposed convolutional layer 354 having a 256-channel input with 16 features and a 128-channel output with 32 features; a third transposed convolutional layer 353 having a 256-channel input with 32 features and a 64-channel output with 64 features; a second transposed convolutional layer 352 having a 128-channel input with 64 features and a 32-channel output with 128 features; and a first transposed convolutional layer 351 having a 64-channel input with 128 features and a 2-channel output with 256 features.
[0108] In one implementation, each of the plurality of transposed convolutional layers 350 has a convolutional kernel of size (2, 9) and a stride of size (1, 2).
[0109] The output of the first convolutional layer 321 is provided as the input of the first transposed convolutional layer 351.
[0110] The output of the second convolutional layer 322 is provided as the input of the second transposed convolutional layer 352.
[0111] The output of the third convolutional layer 323 is provided as the input of the third transposed convolutional layer 353.
[0112] The output of the fourth convolutional layer 324 is provided as the input of the fourth transposed convolutional layer 354.
[0113] The output of the fifth convolutional layer 325 is provided as the input of the fifth transposed convolutional layer 355.
[0114] The output of the sixth convolutional layer 326 is provided as the input of the sixth transposed convolutional layer 356.
[0115] The output of the first transposed convolutional layer 251 is added to the input of the first convolutional layer 321, and the sum is provided as the input of the ISTFT layer 370 to achieve feature recovery when generating the audio output signal 55.
[0116] It should be understood that the number of convolutional layers 320, the number of features and channels associated with each convolutional layer 320, the number of transposed convolutional layers 350, the number of features and channels associated with each transposed convolutional layer 350, the kernel size and stride size, the number, type and size of RNN layers (330, 332) and the number and size of dense layers (340) are application-specific and are selected based on factors related to computing speed, processor capabilities, sound quality, etc.
[0117] Figure 4 The elements associated with training module 400, which is used to train the reference module, are illustrated schematically. Figure 3 The described feature recovery algorithm 300 is implemented using a deep neural network (DNN) module to process the audio input signal 15. The input to the training module 400 includes an audio input signal in the form of clear speech 411 and an audio input signal in the form of noise 412, such as white noise, road noise, baby noise, etc., both provided in the amplitude domain. The clear speech 411 and noise 412 are input to an STFT layer 410, which transforms them to the frequency domain, resulting in transformed clear speech 411' and transformed noise 412'.
[0118] The transformed clear speech 411' and the transformed noise 412' are added together to form noisy speech 415. Noisy speech 415 and transformed noise 412' are input to NLP 420, which enhances the noise reduction level by employing nonlinear filtering to attenuate the noise level. The output of NLP 420 includes residual noise 422 and a combination of distorted speech and residual noise 424. Residual noise 422 is added to the transposed clear speech 411' to form a first input 426. The first input 426, in the form of residual noise 422 added to the transformed clear speech 411', and the combination of distorted speech and residual noise 424 are provided as a reference. Figure 3 The input of the described feature recovery algorithm 300 is used to achieve training.
[0119] This arrangement of inputs to training module 400 is used to train feature recovery algorithm 300 to recover lost speech features without affecting noise levels. Residual noise signals are generated by processing the noise signal according to noisy speech processing. The deep learning method described in this paper unifies the feature extraction process through several layers of neural networks. During the training process, the parameters of the neural network are learned, and then real-time audio is fed into the trained neural network in real time to achieve speech feature recovery.
[0120] The concept described in this paper provides a system that employs a speech feature recovery module to replace a perfectly tuned power phonograph (PF). The feature recovery module supervises the recovery of the original speech quality, allowing for better noise reduction and speech quality in ways that would otherwise be unattainable with known methods. In the case of perfect recovery, the PF can be configured to output the desired noise level regardless of the added desired speech distortion.
[0121] The embodiments of this disclosure can be implemented as an apparatus, method, or computer program product. Therefore, this disclosure can take the form of a completely hardware implementation, a completely software implementation (including firmware, resident software, microcode, etc.), or a combination of software and hardware aspects, which may be generally referred to herein as a "module" or "system." Additionally, this disclosure can take the form of a computer program product implemented in a tangible medium, the computer program product having computer-usable program code implemented in the medium.
[0122] The flowcharts and block diagrams in the flowcharts illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code comprising one or more executable instructions for implementing one or more specified logical functions. It will also be noted that each block in a block diagram and / or flowchart illustration, and combinations of blocks in block diagrams and / or flowchart illustrations, may be implemented by a system based on special-function hardware, or a combination of special-function hardware and computer instructions, that performs the specified functions or actions. These computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture comprising a set of instructions that implement the functions / actions specified in one or more blocks of the flowcharts and / or block diagrams.
[0123] The detailed description and accompanying drawings support and describe this teaching, but the scope of this teaching is defined only by the claims. While some best modes and other embodiments for carrying out this teaching have been described in detail, various alternative designs and embodiments exist for practicing the teaching as defined in the claims.
Claims
1. A system for processing audio input signals, the system comprising: Microphone, controller, data storage, and communication link to remote audio speakers; The microphone is configured to capture and generate the audio input signal and transmit the audio input signal to the controller; The controller is operably connected to the communication link; and The controller includes executable code: The data storage includes instructions executable by the controller, the instructions including: A first result is generated based on the audio input signal using a linear noise reduction filtering algorithm; A second result is generated based on the first result using a nonlinear post-filtering algorithm; An audio output signal is generated based on the second result using a feature recovery algorithm; and The audio output signal is transmitted to the remote audio speaker via the communication link; The feature recovery algorithm includes a module based on a deep neural network (DNN), which includes: STFT (Short Time Fourier Transform); multiple convolutional layers; a first LSTM (Long Short-Term Memory) layer; a second LSTM layer; a dense layer; multiple transposed convolutional layers; and inverse STFT (ISTFT). The plurality of convolutional layers include: The first convolutional layer has a 2-channel input with 256 features and a 32-channel output with 128 features; The second convolutional layer has a 32-channel input with 128 features and a 64-channel output with 64 features; The third convolutional layer has a 64-channel input with 64 features and a 128-channel output with 32 features; The fourth convolutional layer has a 128-channel input with 32 features and a 128-channel output with 16 features; The fifth convolutional layer has a 128-channel input with 16 features and a 258-channel output with 8 features; and The sixth convolutional layer has a 256-channel input with 8 features and a 256-channel output with 4 features; The output of the sixth convolutional layer is used as the input of the first LSTM layer, and the output of the first convolutional layer is used as the input of the inverse STFT.
2. The system according to claim 1, wherein, The STFT transforms the audio input signal from the amplitude domain to the frequency domain.
3. The system according to claim 2, wherein, The STFT transforms the audio input signal into a frequency domain with a 2-channel sequence having a real part and an imaginary part.
4. The system according to claim 1, wherein, Each of the plurality of convolutional layers has a convolutional kernel of size (2, 9) and a stride of size (1, 2).
5. The system according to claim 1, wherein, The first LSTM layer has 256 states.
6. The system according to claim 1, wherein, The second LSTM layer has 256 states.
7. The system according to claim 1, wherein, The plurality of transposed convolutional layers include a sixth transposed convolutional layer having a 512-channel input with 4 features and a 256-channel output with 8 features; a fifth transposed convolutional layer having a 512-channel input with 8 features and a 128-channel output with 16 features; a fourth transposed convolutional layer having a 256-channel input with 16 features and a 128-channel output with 32 features; a third transposed convolutional layer having a 256-channel input with 32 features and a 64-channel output with 64 features; a second transposed convolutional layer having a 128-channel input with 64 features and a 32-channel output with 128 features; and a first transposed convolutional layer having a 64-channel input with 128 features and a 2-channel output with 256 features.
8. The system according to claim 7, wherein, The output of the dense layer is provided as the input of the sixth transposed convolutional layer.
9. The system according to claim 7, wherein, Each of the plurality of transposed convolutional layers has a convolutional kernel of size (2, 9) and a stride of size (1, 2).
10. The system according to claim 7, wherein, The output of the first transposed convolutional layer is used as the input to the inverse STFT to achieve feature recovery.
11. The system according to claim 10, wherein, The inverse STFT combines the audio input signal transposed to the frequency domain with the output of the first transposed convolutional layer to transform it from the frequency domain to the amplitude domain to generate the audio output signal.
12. A method for processing an audio input signal, the method comprising: Audio input signals are captured via a microphone; The audio input signal is subjected to a linear noise reduction filtering algorithm to generate a first result; The first result is subjected to a nonlinear post-filtering algorithm to generate the second result; An audio output signal is generated by subjecting the second result to a feature recovery algorithm. as well as The speaker is controlled in response to the audio output signal; The feature recovery algorithm includes a module based on a deep neural network (DNN), which includes: an STFT (Short Time Fourier Transform) layer, multiple convolutional layers, a first LSTM (Long Short Time Memory) layer, a second LSTM layer, a dense layer, multiple transposed convolutional layers, and an inverse STFT (ISTFT) layer. The plurality of convolutional layers include: The first convolutional layer has a 2-channel input with 256 features and a 32-channel output with 128 features; The second convolutional layer has a 32-channel input with 128 features and a 64-channel output with 64 features; The third convolutional layer has a 64-channel input with 64 features and a 128-channel output with 32 features; The fourth convolutional layer has a 128-channel input with 32 features and a 128-channel output with 16 features; The fifth convolutional layer has a 128-channel input with 16 features and a 258-channel output with 8 features; and The sixth convolutional layer has a 256-channel input with 8 features and a 256-channel output with 4 features; The output of the sixth convolutional layer is provided as the input of the first LSTM layer, and the output of the first convolutional layer is used as the input of the inverse STFT.
13. A system for processing voice input, the system comprising: Microphone, controller, and speaker; The microphone is configured to capture a voice input signal and transmit the voice input signal to the controller; and the controller is operatively connected to the speaker. The controller includes executable code: The voice input signal is subjected to a linear noise reduction filtering algorithm to generate a first result; The first result is subjected to a nonlinear post-filtering algorithm to generate the second result; The speech output signal is generated by subjecting the second result to a feature recovery algorithm; and The speaker is controlled in response to the voice output signal; The feature recovery algorithm includes a module based on a deep neural network (DNN), which includes: an STFT (Short Time Fourier Transform) layer, multiple convolutional layers, a first LSTM (Long Short Time Memory) layer, a second LSTM layer, a dense layer, multiple transposed convolutional layers, and an inverse STFT (ISTFT) layer. The plurality of convolutional layers include: The first convolutional layer has a 2-channel input with 256 features and a 32-channel output with 128 features; The second convolutional layer has a 32-channel input with 128 features and a 64-channel output with 64 features; The third convolutional layer has a 64-channel input with 64 features and a 128-channel output with 32 features; The fourth convolutional layer has a 128-channel input with 32 features and a 128-channel output with 16 features; The fifth convolutional layer has a 128-channel input with 16 features and a 258-channel output with 8 features; and The sixth convolutional layer has a 256-channel input with 8 features and a 256-channel output with 4 features; The output of the sixth convolutional layer is provided as the input of the first LSTM layer, and the output of the first convolutional layer is used as the input of the inverse STFT.