A speech enhancement method based on air-bone conduction dual-mode deep learning

Abstract

This invention discloses a speech enhancement method based on air-bone dual-mode deep learning, comprising the following steps: Step 1, synchronously recording air-guided and bone-guided speech in a noise-free environment, adding environmental noise to the air-guided speech to construct a dataset, and dividing it into a training set, a test set, and a validation set; Step 2, segmenting the training set by cutting the speech data of the training set into multiple short speech segments of a fixed length; Step 3, constructing a neural network, including a high-order encoder, zero-order terms, high-order terms, and auxiliary post-processing filters; Step 4, model training; Step 5, model testing. The algorithm of this invention has low computational complexity, meets real-time requirements, has stable noise reduction performance, and provides high speech quality and intelligibility, making it applicable to portable devices.

Description

Technical Field

[0001] This invention belongs to the field of speech processing technology, and in particular relates to a speech enhancement method based on air-bone dual-mode deep learning. Background Technology

[0002] Air conduction microphones capture vibration signals transmitted through the air, while bone conduction microphones capture vibration signals transmitted through the jawbone and other body tissues. Both capture speech signals from the same sound source but with different characteristics. In environments with significant noise interference, bone conduction speech signals can effectively avoid background noise interference, but its high-frequency energy attenuation characteristics result in a dull timbre and low intelligibility. In environments with low noise interference, the advantages of bone conduction signals over air conduction signals become less significant, and air conduction signals offer higher speech quality and intelligibility.

[0003] Current research combines the two types of signals to further improve noise reduction performance. To address the problem of poor speech quality and intelligibility caused by using only bone conduction or air conduction speech enhancement methods in high-noise environments, Chinese patent document CN112786064B describes an end-to-end bone and air conduction speech joint enhancement method. This method uses an encoder-mask estimator-decoder neural network model to jointly enhance noisy air conduction speech and bone conduction speech, resulting in a significant improvement in both speech quality and intelligibility.

[0004] Chinese patent application CN115966218A discloses a bone-guided assisted air-guided speech processing method, device, medium, and equipment. By inputting bone-guided features and noisy air-guided features into a neural network to obtain enhancement spectral coefficients, speech signal enhancement can be achieved with a small time delay.

[0005] In summary, both time-domain and time-frequency-domain speech enhancement methods lack mechanisms for separately processing the two signals. This leads to redundant extracted features, increasing the number of network parameters and computational complexity. Currently, achieving real-time, low-complexity, deep learning-based air-bone guided speech enhancement is a pressing technical challenge. Summary of the Invention

[0006] The purpose of this invention is to provide a speech enhancement method based on air-bone-guided dual-mode deep learning, which is based on a deep learning network structure that combines air and bone guidance, has low computational cost, low complexity, and meets real-time requirements.

[0007] To achieve the objective of this invention, a speech enhancement method based on air-bone conduction dual-mode deep learning is provided, comprising the following steps:

[0008] Step 1: Construct a dataset and divide it into a training set, a validation set, and a test set;

[0009] Step 2: Segment the speech data of the training set to obtain the training set data;

[0010] Step 3: Construct the Taylor Lite neural network model, which includes a high-order encoder, a zero-order term, a high-order term, and an auxiliary post-filter;

[0011] Step 4: Train the Taylor Lite neural network model using the training set data;

[0012] Step 5: Input the speech data of the test set into the trained Taylor Lite neural network model and output the target complex value spectrum.

[0013] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements a speech enhancement method based on air-bone dual-mode deep learning.

[0014] A non-transitory computer-readable storage medium storing computer instructions for a speech enhancement method based on air-bone dual-mode deep learning.

[0015] A computer program product includes computer program instructions that, when executed on a computer, cause the computer to perform a speech enhancement method based on air-bone dual-mode deep learning.

[0016] Compared with the prior art, the significant progress of the present invention is as follows: (1) The present invention utilizes both noisy air-guided speech and bone-guided speech to achieve speech enhancement based on air-bone dual-mode deep learning for portable devices; effectively integrates the signal characteristics of air and bone guides to improve noise resistance; (2) The algorithm of the present invention has low computational load, low complexity, meets real-time requirements, has stable noise reduction performance, and has high speech quality and intelligibility, and can be applied to portable devices.

[0017] To more clearly illustrate the functional characteristics and structural parameters of the present invention, further explanation is provided below in conjunction with the accompanying drawings and specific embodiments. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0019] Figure 1 This is a flowchart of the speech enhancement method of the present invention;

[0020] Figure 2This is a schematic diagram of a specific embodiment of the present invention;

[0021] Figure 3 This invention relates to a low-level deep neural network Taylor Lite model structure;

[0022] Figure 4 (a)-(f) are the simulation results of speech quality and speech intelligibility under different signal-to-noise ratios and different noise scenarios of the present invention. Detailed Implementation

[0023] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Combination Figure 1 The present invention provides a speech enhancement method based on air-bone dual-mode deep learning, comprising the following steps:

[0025] Step 1: Synchronously record air-guided speech in a noise-free environment. a Bone conduction speech b 、Give voice guidance a Adding ambient noise d, we obtain noisy air-guided speech x. a =s a +d, construct the dataset[(x a s b ), s a ], and the dataset [(x a s b ), s a The dataset is divided into training set, validation set, and test set.

[0026] Step 2: Cut the speech data of the training set into multiple small speech data segments of a fixed length to obtain the training set data;

[0027] Step 3: Construct the Taylor Lite neural network model, which includes a high-order encoder, a zero-order term, a high-order term, and an auxiliary post-filter;

[0028] Step 3-1: The higher-order encoder consists of a two-dimensional gating unit, normalization, and activation function; the input to the higher-order encoder is the training set data (x... a s b The output is processed by a high-order encoder to convert noisy, air-guided speech. a and bone conduction speech bThe amplitude spectrum is transformed into abstract features R1, and the noisy air-guided speech complex value spectrum and bone-guided speech complex value spectrum are transformed into abstract features R2.

[0029] Step 3-2: The zero-order term is composed of an equivalent rectangular bandwidth filter - linear layer - normalization - two-dimensional gated unit - linear layer; input the abstract feature R1, output the amplitude filter coefficients, and obtain the zero-order amplitude domain term in the Taylor expansion of the target complex spectrum;

[0030] Step 3-3: The higher-order term is composed of a linear layer-normalization-two-dimensional gated unit-linear layer-activation function-linear layer; the abstract feature R2 is input, and the output is the real and imaginary parts of the higher-order complex domain term in the Taylor expansion of the target complex spectrum;

[0031] Steps 3-4: The auxiliary post-filter consists of a normalized-gated cyclic unit-linear layer; the zero-order term output and the higher-order term output are superimposed and input according to the Taylor formula. The auxiliary post-filter suppresses noise residues that may occur in non-speech segments and outputs the jointly enhanced complex-valued spectrum.

[0032] Step 4: Train the Taylor Lite neural network model using the training set data;

[0033] Step 5: Input the speech data of the test set into the trained Taylor Lite neural network model and output the jointly enhanced target complex value spectrum.

[0034] Furthermore, the dataset is divided into 80% training set, 10% validation set, and 10% test set.

[0035] Furthermore, the speech data of the training set is divided into training set data with a fixed length of 20ms.

[0036] Furthermore, the Taylor Lite neural network model is constructed in a 4th-order Taylor expansion form, which includes a 0th-order amplitude spectrum estimation module and higher-order 1st, 2nd, and 3rd-order complex residual estimation modules.

[0037] Furthermore, for the scenario of air-bone guided speech denoising, the input channel of the high-order encoder in step 3-1 is 4; for the scenario of air-guided speech denoising, the input channel of the high-order encoder in step 3-1 is 2.

[0038] Example

[0039] Example 1

[0040] like Figure 2 A speech enhancement method based on air-bone dual-mode deep learning includes the following steps:

[0041] Step S 101: Synchronously acquire air conduction and bone conduction audio, perform short-time Fourier transform on bone conduction audio and noisy air conduction audio to obtain complex-valued spectra of bone conduction audio and noisy air conduction audio;

[0042] Step S102: Input the complex-valued spectra of bone conduction audio and noisy air conduction audio into a pre-trained neural network to obtain the target complex-valued spectrum estimate;

[0043] Step S103: Perform an inverse Fourier transform on the jointly enhanced target complex-valued spectrum to obtain the jointly enhanced target speech time-domain signal.

[0044] The method described above can simultaneously utilize noisy air-guided speech and bone-guided speech to achieve speech enhancement based on air-bone dual-mode deep learning for portable devices under low signal-to-noise ratio conditions and with low computational cost.

[0045] Example 2:

[0046] Step 1: Select air-guided speech s synchronously recorded in a low-noise environment from the open-source ABCS Chinese dataset. a and bone conduction speech b , for air-guided speech s a Noise datasets from the DNS competition were added as air conduction signal noise, with the signal-to-noise ratio set to [-15, -10, -5, 0, 5] dB to obtain noisy air conduction speech. Training and validation sets were constructed, and 8 speakers not seen in the training and validation sets were selected as test set samples. The NOISE92X noise dataset was selected as the air conduction signal noise for the test set, with each signal sampling rate of 16 kHz.

[0047] Step 2: Divide the speech data in the training set into multiple short speech segments of a fixed length;

[0048] Step 3: Construct the Taylor Lite neural network, including a high-order encoder, zero-order terms, high-order terms, and auxiliary post-processing filters;

[0049] Step 3-1: The high-order encoder consists of a two-dimensional gating unit, normalization, and activation function; the input to the high-order encoder is the data (x) cut in step 2. a s b The output is the noisy, air-guided speech encoded by a high-order encoder. a and bone conduction speech b The amplitude spectrum is transformed into an abstract feature R1, and the noisy gas conduction complex spectrum and bone conduction complex spectrum are transformed into an abstract feature R2.

[0050] Step 3-2: As Figure 3(a) The zero-order term is composed of an equivalent rectangular bandwidth filter, a linear layer, normalization, a two-dimensional gating unit, and a linear layer. The abstract feature R1 is input, and the output is the amplitude filtering coefficient, thereby obtaining the zero-order amplitude domain term in the Taylor expansion of the target complex spectrum. The equivalent rectangular bandwidth filter merges adjacent frequency bands according to the characteristics of human hearing, thereby significantly reducing the number of frequency bands. The traditional "encoder-temporal modeling-decoder" structure is replaced with a two-dimensional gating unit, and the number of parameters is further reduced through a grouping algorithm.

[0051] Step 3-3: As Figure 3 (b) The higher-order term is composed of a linear layer-normalization-two-dimensional gated unit-linear layer-activation function-linear layer; the abstract feature R2 is input and the output is the real part and imaginary part of the higher-order complex domain term in the Taylor expansion of the target complex spectrum;

[0052] In this embodiment, the higher-order terms are set as the first, second, and third-order complex residual estimation modules, which respectively estimate the first, second, and third-order Taylor expansion terms of the target complex-valued spectrum;

[0053] Steps 3-4: As Figure 3 (c) The auxiliary post-filter is composed of a normalized-gated cyclic unit-linear layer; the zero-order term output and the higher-order term output are superimposed and input according to the Taylor formula, and the output is the jointly enhanced complex value spectrum;

[0054] Since both the zero-order and higher-order terms are designed with lightweight features, the model lacks long-term memory. The processed speech has limited ability to suppress non-stationary noise and may retain a lot of nonlinear noise. Therefore, an auxiliary post-filter is designed to suppress the noise residue that may appear in non-speech segments.

[0055] Step 4: Model Training: Train the Taylor Lite neural network using the training set data. The input audio signal window length is 20ms, the frame shift is 10ms, the number of discrete Fourier transform points is 320, the training epochs are 50, and the learning rate is 1×10⁻⁶. -4 ;

[0056] Step 5: Model testing: Input the test set data into the Taylor Lite neural network model obtained in step 4, and output the jointly enhanced target complex value spectrum. The window length of the test data is 20ms, the frame shift is 10ms, and the number of discrete Fourier transform points is 320.

[0057] Compared with the classic baseline network GCCRN, this embodiment can achieve higher speech enhancement effect, while also having the advantages of low parameter count and low computational complexity.

[0058] Table 1 shows the number of parameters and computational cost of the GCCRN network and the Taylor Lite network in this embodiment. Under different signal-to-noise ratios, this embodiment has a lower number of network parameters and computational cost.

[0059] Table 1. Number of network parameters and computational complexity

[0060]

[0061] Tables 2 and 3 show the processing results of the GCCRN network and the Taylor Lite network of this embodiment for air-bone guided signals in a Babble noise scenario. The Taylor Lite network of this embodiment performs better in terms of signal-to-noise ratio gain (SSNR), especially under -5dB and 0dB conditions. Therefore, the Taylor Lite network of this embodiment has more stable noise reduction performance, and the processed signals all have high speech quality and speech intelligibility scores.

[0062] Table 2. Babble noise scenario. Air conduction signal processing results in this embodiment.

[0063]

[0064]

[0065] Table 3. GCCRN air conduction signal processing results in the Babble noise scenario.

[0066]

[0067] Combination Figure 4 (a) PESQ score in a Babble noise scene. Figure 4 (b) STOI score in the Babble noise scene. Figure 4 (c) PESQ score in a white noise scene. Figure 4 (d) STOI score in a white noise scene. Figure 4 (e) PESQ score in Factoryl noise scene. Figure 4 (f) STOI score in Factoryl noise scenario; In the above three noise scenarios with different signal-to-noise ratios, the speech quality PESQ and speech intelligibility STOI evaluation scores of the two models, GCCRN network and Taylor Lite network in this embodiment, are as follows: When the bone conduction signal is not introduced, the speech quality and speech intelligibility processed by the GCCRN model at low signal-to-noise ratio are worse than those without the bone conduction signal, but when the bone conduction signal is introduced, the model score is dramatically improved.

[0068] Furthermore, the Taylor Lite network in this embodiment also showed a slight improvement in score after the additional bone conduction signal was introduced. This demonstrates that under low signal-to-noise ratio conditions, the bone conduction signal can serve as an additional information input, and can obtain higher quality speech quality and speech intelligibility processing results compared to using only a single-channel air conduction signal. The Taylor Lite network in this embodiment has relatively stable processing effects for both input conditions, thus possessing higher speech enhancement effects, and also has the advantages of low parameter quantity and low computational complexity.

[0069] It should be noted that, in this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0070] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A speech enhancement method based on air-bone dual-mode deep learning, characterized in that, Includes the following steps: Step 1: Construct a dataset and divide it into a training set, a validation set, and a test set; Step 2: Segment the speech data of the training set to obtain the training set data; Step 3: Construct the Taylor Lite neural network model, which includes a high-order encoder, a zero-order term, a high-order term, and an auxiliary post-filter; Step 3-1: The higher-order encoder consists of a two-dimensional gating unit, normalization, and activation function; the input to the higher-order encoder is the training set data. The output is processed by a high-order encoder to convert noisy air-guided speech. and bone conduction speech The amplitude spectrum is transformed into abstract features And the abstract features that transform the noisy air-guided speech complex value spectrum and bone-guided speech complex value spectrum into ; Step 3-2: The zero-order term is composed of an equivalent rectangular bandwidth filter - linear layer - normalization - two-dimensional gated unit - linear layer; the abstract feature Input and output amplitude filter coefficients to obtain the zeroth-order amplitude domain term in the Taylor expansion of the target complex spectrum; Step 3-3: The higher-order term consists of a linear layer - normalization - two-dimensional gating unit - linear layer - activation function - linear layer; the abstract feature The input and output are the real and imaginary parts of the higher-order complex-valued field terms in the Taylor expansion of the target complex-valued spectrum; Steps 3-4: The auxiliary post-filter consists of a normalization-gated cyclic unit-linear layer; the zero-order term output and the higher-order term output are superimposed and input according to the Taylor formula, and the auxiliary post-filter suppresses noise residue that may occur in non-speech segments and outputs a complex-valued spectrum. Step 4: Train the Taylor Lite neural network model using the training set data; Step 5: Input the speech data of the test set into the trained Taylor Lite neural network model and output the target complex value spectrum.

2. The speech enhancement method based on air-bone conduction dual-mode deep learning according to claim 1, characterized in that, Step 1 involves synchronously recording air conduction speech in a noise-free environment. Bone conduction speech , Give air guidance voice Add environmental noise Noisy air-guided speech was obtained. Construct the dataset .

3. The speech enhancement method based on air-bone conduction dual-mode deep learning according to claim 1, characterized in that, The dataset is divided into 80% training set, 10% validation set, and 10% test set.

4. The speech enhancement method based on air-bone conduction dual-mode deep learning according to claim 1, characterized in that, The speech data in the training set is divided into training set data with a fixed length of 20ms.

5. The speech enhancement method based on air-bone conduction dual-mode deep learning according to claim 1, characterized in that, The Taylor Lite neural network model is constructed using a 4th-order Taylor expansion, which includes a 0th-order amplitude spectrum estimation module and higher-order 1st, 2nd, and 3rd-order complex residual estimation modules.

6. The speech enhancement method based on air-bone conduction dual-mode deep learning according to claim 1, characterized in that, For the scenario of air-bone guided speech denoising, the input channel of the high-order encoder in step 3-1 is 4; for the scenario of air-guided speech denoising, the input channel of the high-order encoder in step 3-1 is 2.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1 to 6.

9. A computer program product comprising computer program instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 6.

Images