Speech enhancement method based on air bone conduction dual-mode deep learning

Abstract

The invention discloses a speech enhancement method based on air and bone conduction dual-mode deep learning, and the method comprises the following steps: 1, synchronously recording air conduction speech and bone conduction speech in a noiseless environment, adding environment noise for the air conduction speech to construct a data set, and dividing the data set into a training set, a test set and a verification set; step 2, cutting the training set, and cutting the voice data of the training set into a plurality of small voice segments according to a fixed length; 3, constructing a neural network which comprises a high-order encoder, a zero-order term, a high-order term and an auxiliary post-processing filter; 4, performing model training; and 5, testing the model. The method is low in algorithm operand, low in complexity, capable of meeting the real-time requirement, stable in noise reduction performance, high in voice quality and intelligibility and capable of being applied to portable equipment.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech processing, and in particular relates to a speech enhancement method based on air-bone conduction dual-mode deep learning. Background Art

[0002] The air conduction microphone collects vibration signals transmitted through the air, while the bone conduction microphone collects vibration signals transmitted through the jawbone, human tissue, etc. Both collect voice signals with different characteristics from the same sound source. When the environmental noise interference is large, the bone conduction voice signal can effectively avoid the interference of background noise, but its characteristics such as high-frequency energy attenuation lead to dull timbre and low intelligibility. When the environmental noise interference is small, the advantage of bone conduction signal over air conduction signal becomes less obvious, and air conduction signal has higher voice quality and intelligibility.

[0003] At present, there are studies that combine the two types of signals to further improve the noise reduction performance. In order to solve the problem of poor speech quality and intelligibility caused by speech enhancement methods that only use bone conduction or air conduction in high-noise environments, Chinese patent document CN112786064B records an end-to-end bone-air conduction speech joint enhancement method, which jointly enhances noisy air conduction speech and bone conduction speech through an "encoder-mask estimator-decoder" neural network model, and the joint enhanced speech quality and intelligibility are greatly improved.

[0004] Chinese patent application document CN115966218A describes a bone conduction-assisted air conduction speech processing method, device, medium and equipment, which inputs bone conduction characteristics and noisy air conduction characteristics into a neural network to obtain enhanced spectral coefficients, and can achieve speech signal enhancement with a smaller time delay.

[0005] In summary, whether speech enhancement is performed in the time domain or the time-frequency domain, there is a lack of a mechanism for processing the two signals separately, which will lead to redundant extracted features and increase the number of network parameters and computation. At present, low-complexity, deep learning-based air-bone-conducted speech enhancement that meets real-time requirements has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The present invention aims to provide a speech enhancement method based on air-bone conduction dual-mode deep learning, which is based on a deep learning network structure that combines air-bone conduction, has low computational complexity, and meets real-time requirements.

[0007] In order to achieve the purpose of the present invention, the present invention provides a speech enhancement method based on air-bone conduction dual-mode deep learning, comprising the following steps:

[0008] Step 1: construct a data set and divide the data set into a training set, a validation set, and a test set;

[0009] Step 2: cutting the speech data of the training set to obtain training set data;

[0010] Step 3, constructing a Taylor Lite neural network model, which includes a high-order encoder, a zero-order term, a high-order term, and an auxiliary post-filter;

[0011] Step 4: training the Taylor Lite neural network model using the training set data;

[0012] Step 5: Send the speech data of the test set to the trained Taylor Lite neural network model, and output the target complex-valued spectrum.

[0013] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a speech enhancement method based on air-bone conduction dual-mode deep learning is implemented.

[0014] A non-transitory computer-readable storage medium storing computer instructions for a speech enhancement method based on air-bone conduction dual-mode deep learning.

[0015] A computer program product includes computer program instructions. When the computer program instructions are executed on a computer, the computer is caused to execute a speech enhancement method based on air-bone conduction dual-mode deep learning.

[0016] Compared with the prior art, the significant progress of the present invention is that: (1) the present invention simultaneously utilizes noisy air-conducted speech and bone-conducted speech to achieve speech enhancement based on air-bone dual-mode deep learning for use in portable devices; effectively integrates the signal characteristics of air and bone conduction to improve noise resistance performance; (2) the algorithm of the present invention has low computational complexity, low complexity, meets real-time requirements, has stable noise reduction performance, and has high speech quality and intelligibility, and can be used in portable devices.

[0017] In order to more clearly illustrate the functional characteristics and structural parameters of the present invention, further description is given below in conjunction with the accompanying drawings and specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1 is a flow chart of the speech enhancement method of the present invention;

[0020] Figure 2It is a schematic diagram of a specific embodiment of the present invention;

[0021] Figure 3 It is a low-level deep neural network Taylor Lite model structure of the present invention;

[0022] Figure 4 (a)-(f) are the simulation results of speech quality and speech intelligibility of the present invention under different signal-to-noise ratios and different noise scenarios. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0024] Combination Figure 1 A speech enhancement method based on air-bone conduction dual-mode deep learning of the present invention comprises the following steps:

[0025] Step 1: Record air-conduction speech synchronously in a noise-free environment a , bone conduction speech b , give air conduction speech a Add environmental noise d to get noisy air-conducted speech x a =s a +d, construct the data set [(x a ,s b ), s a ], and the data set [(x a ,s b ), s a ] is divided into training set, validation set and test set;

[0026] Step 2: Cut the speech data of the training set into multiple small speech data segments according to a fixed length to obtain training set data;

[0027] Step 3, constructing a Taylor Lite neural network model, which includes a high-order encoder, a zero-order term, a high-order term, and an auxiliary post-filter;

[0028] Step 3-1: The high-order encoder is composed of a two-dimensional gating unit-normalization-activation function; the input of the high-order encoder is the training set data (x a ,s b ), the output is converted into noisy air-conducted speech x by a high-order encoder a and bone conduction speech bThe amplitude spectrum of is converted into the abstract feature R1, and the complex spectrum of noisy air-conduction speech and the complex spectrum of bone-conduction speech are converted into the abstract feature R2;

[0029] Step 3-2: The zero-order term is composed of an equivalent rectangular bandwidth filter-linear layer-normalization-two-dimensional gating unit-linear layer; the abstract feature R1 is input, and the amplitude filter coefficient is output to obtain the zero-order amplitude domain term in the target complex-valued spectrum Taylor expansion;

[0030] Step 3-3: The high-order term is composed of a linear layer-normalization-two-dimensional gating unit-linear layer-activation function-linear layer; the abstract feature R2 is input, and the output is the real part and imaginary part of the high-order complex-valued domain term in the target complex-valued spectrum Taylor expansion;

[0031] Step 3-4: The auxiliary post-filter is composed of a normalization-gated recurrent unit-linear layer; the zero-order item output and the high-order item output are superimposed and input according to the Taylor formula, and the auxiliary post-filter suppresses the noise residue that may appear in the non-speech segment and outputs a jointly enhanced complex-valued spectrum.

[0032] Step 4: training the Taylor Lite neural network model using the training set data;

[0033] Step 5: Send the speech data of the test set to the trained Taylor Lite neural network model, and output a jointly enhanced target complex-valued spectrum.

[0034] Furthermore, the data set is divided into 80% training set, 10% validation set, and 10% test set.

[0035] Furthermore, the speech data of the training set is cut into training set data according to a fixed length of 20 ms.

[0036] Furthermore, when constructing the Taylor Lite neural network model, a 4th-order Taylor expansion form is set, and the 4th-order Taylor expansion form includes a 0th-order amplitude spectrum estimation module and high-order 1st, 2nd, and 3rd-order complex residual estimation modules.

[0037] Furthermore, for the scenario of air-bone conduction speech noise reduction, the input channel of the high-order encoder in step 3-1 is 4; for the scenario of air-conduction speech noise reduction, the input channel of the high-order encoder in step 3-1 is 2.

[0038] Example

[0039] Example 1

[0040] like Figure 2 A speech enhancement method based on air-bone conduction dual-mode deep learning, comprising the following steps:

[0041] Step S101: synchronously collecting air conduction audio and bone conduction audio, performing short-time Fourier transform on the bone conduction audio and the noisy air conduction audio, and obtaining complex value spectra of the bone conduction audio and the noisy air conduction audio;

[0042] Step S102: inputting the complex-valued spectra of the bone conduction audio and the noisy air conduction audio into a pre-trained neural network to obtain a target complex-valued spectrum estimate;

[0043] Step S103: performing an inverse Fourier transform on the jointly enhanced target complex-valued spectrum to obtain a jointly enhanced target speech time domain signal.

[0044] The method can simultaneously utilize noisy air-conduction speech and bone-conduction speech to achieve speech enhancement based on air-bone conduction dual-mode deep learning for use in portable devices under low signal-to-noise ratio conditions with a low algorithm computational load.

[0045] Embodiment 2:

[0046] Step 1: Select the air-conducted speech s recorded synchronously in a low-noise environment from the open-source ABCS Chinese dataset a and bone conduction speech b , for air-conducted speech a Add the noise dataset in the DNS competition as the air-conduction signal noise, set the signal-to-noise ratio to [-15, -10, -5, 0, 5] dB, obtain noisy air-conduction speech, construct the training set and validation set, and select 8 speakers not seen in the training set and validation set as the test set expectations. Select the NOISE92X noise dataset as the test set air-conduction signal noise, and the sampling rate of each signal is 16kHz;

[0047] Step 2: Cut the speech data of the training set into multiple short speech segments according to fixed length;

[0048] Step 3: Construct a Taylor Lite neural network, including a high-order encoder, zero-order terms, high-order terms, and auxiliary post-processing filters;

[0049] Step 3-1: The high-order encoder is composed of a two-dimensional gating unit-normalization-activation function; the input of the high-order encoder is the data cut in step 2 (x a ,s b ), the output is the noisy air-conducted speech x through a high-order encoder a and bone conduction speech b The amplitude spectrum is converted into an abstract feature R1, and the noisy air conduction complex value spectrum and bone conduction complex value spectrum are converted into an abstract feature R2;

[0050] Step 3-2: If Figure 3(a), the zero-order term is composed of an equivalent rectangular bandwidth filter-linear layer-normalization-two-dimensional gating unit-linear layer; the abstract feature R1 is input and output as an amplitude filter coefficient, thereby obtaining the zero-order amplitude domain term in the target complex-valued spectrum Taylor expansion; wherein the equivalent rectangular bandwidth filter merges adjacent frequency bands according to the auditory characteristics of the human ear, thereby significantly reducing the number of frequency bands, and replaces the traditional "encoder-series modeling-decoder" structure with a two-bit gating unit, and further reduces the number of parameters through a grouping algorithm;

[0051] Step 3-3: If Figure 3 (b), the high-order term is composed of a linear layer-normalization-two-dimensional gating unit-linear layer-activation function-linear layer; the abstract feature R2 is input, and the output is the real part and imaginary part of the high-order complex-valued domain term in the target complex-valued spectrum Taylor expansion;

[0052] In this embodiment, the high-order term is set as the 1st, 2nd, and 3rd order complex residual estimation modules, which respectively estimate the first-order, second-order, and third-order expansion terms of the Taylor expansion of the target complex-valued spectrum;

[0053] Step 3-4: If Figure 3 (c), the auxiliary post-filter is composed of a normalization-gated recurrent unit-linear layer; the zero-order term output and the high-order term output are superimposed and input according to the Taylor formula, and a jointly enhanced complex value spectrum is output;

[0054] Since both the zero-order and high-order terms are lightweight in design, the model lacks long-term memory, and the processed speech has limited ability to suppress non-stationary noise, and more nonlinear noise may remain. Therefore, an auxiliary post-filter is designed to suppress the noise residue that may appear in the non-speech segment.

[0055] Step 4: Model training: The Taylor Lite neural network is trained using the data from the training set. The input audio signal window length is 20 ms, the frame shift is 10 ms, the number of discrete Fourier transform points is 320, the training rounds are 50, and the learning rate is 1×10 -4 ;

[0056] Step 5: Model testing: The test set data is sent to the Taylor Lite neural network model obtained in step 4, and the output is a jointly enhanced target complex value spectrum. The window length of the test data is 20ms, the frame shift is 10ms, and the number of discrete Fourier transform points is 320.

[0057] Compared with the classic method baseline network GCCRN, this embodiment can have a higher speech enhancement effect and has the advantages of low parameter amount and low computational complexity.

[0058] Table 1 shows the parameters and computational complexity of the GCCRN network and the Taylor Lite network of this embodiment. Under different signal-to-noise ratios, this embodiment has lower network parameters and computational complexity.

[0059] Table 1. Network parameters and computational complexity

[0060]

[0061] Tables 2 and 3 show the processing results of the GCCRN network and the Taylor Lite network of this embodiment for air-bone conduction signal input in the Babble noise scenario. The Taylor Lite network of this embodiment performs better in terms of signal-to-noise ratio gain (SSNR), especially under -5dB and 0dB conditions. Therefore, the Taylor Lite network of this embodiment has a more stable noise reduction performance, and the processed signals have higher voice quality and voice intelligibility scores.

[0062] Table 2. Babble noise scenario, air-bone conduction signal processing results of this embodiment

[0063]

[0064]

[0065] Table 3. Babble noise scene, GCCRN air-bone conduction signal processing results

[0066]

[0067] Combination Figure 4 (a) PESQ score in Babble noise scenario, Figure 4 (b) STOI score in Babble noise scenario, Figure 4 (c) PESQ score in white noise scenario, Figure 4 (d) STOI score in white noise scenario, Figure 4 (e) PESQ score under Factoryl noise scenario, Figure 4 (f) STOI score in Factoryl noise scenario; In the above three noise scenarios and scenarios with different signal-to-noise ratios, the evaluation scores of speech quality PESQ and speech intelligibility STOI corresponding to the two types of models of the GCCRN network and the Taylor Lite network of this embodiment, that is, when the bone conduction signal is not introduced, the processed speech quality and speech intelligibility of the GCCRN model at low signal-to-noise ratio are inferior to those of the unprocessed air conduction signal, but when the bone conduction signal is introduced, the model score is sharply improved.

[0068] In addition, the score of the Taylor Lite network in this embodiment is slightly improved after the bone conduction signal is additionally introduced, which also shows that in the case of low signal-to-noise ratio, the bone conduction signal can be used as an additional information input, and higher quality speech quality and speech intelligibility processing results can be obtained compared to using only a single-channel air conduction signal; the processing effect of the Taylor Lite network in this embodiment for both input conditions is relatively stable, so it has a higher speech enhancement effect, and has the advantages of low parameter amount and low computational complexity.

[0069] It should be noted that, in this article, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0070] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A speech enhancement method based on air-bone dual-mode deep learning, characterized in that: The following steps are involved: Step 1: construct a data set and divide the data set into a training set, a validation set, and a test set; Step 2: cutting the speech data of the training set to obtain training set data; Step 3, constructing a Taylor Lite neural network model, which includes a high-order encoder, a zero-order term, a high-order term, and an auxiliary post-filter; Step 4: training the Taylor Lite neural network model using the training set data; Step 5: Send the speech data of the test set to the trained Taylor Lite neural network model, and output the target complex-valued spectrum.

2. The speech enhancement method based on air-bone dual-mode deep learning according to claim 1 is characterized in that: Step 1 is to synchronously record air-conducted speech in a noise-free environment. a , bone conduction speech b , give air conduction speech a Add environmental noise d to get noisy air-conducted speech x a =s a +d, construct the data set [(x a ,s b ), s a ].

3. The speech enhancement method based on air-bone conduction dual-mode deep learning according to claim 1 is characterized in that: The step 3 comprises the following steps: Step 3-1: The high-order encoder is composed of a two-dimensional gating unit-normalization-activation function; the input of the high-order encoder is the training set data (x a ,s b ), the output is converted into noisy air-conducted speech x by a high-order encoder a and bone conduction speech b The amplitude spectrum of is converted into the abstract feature R1, and the complex spectrum of noisy air-conduction speech and the complex spectrum of bone-conduction speech are converted into the abstract feature R2; Step 3-2: The zero-order term is composed of an equivalent rectangular bandwidth filter-linear layer-normalization-two-dimensional gating unit-linear layer; the abstract feature R1 is input, and the amplitude filter coefficient is output to obtain the zero-order amplitude domain term in the target complex-valued spectrum Taylor expansion; Step 3-3: The high-order term is composed of a linear layer-normalization-two-dimensional gating unit-linear layer-activation function-linear layer; the abstract feature R2 is input, and the output is the real part and imaginary part of the high-order complex-valued domain term in the target complex-valued spectrum Taylor expansion; Step 3-4: The auxiliary post-filter is composed of a normalization-gated recurrent unit-linear layer; the zero-order item output and the high-order item output are superimposed and input according to the Taylor formula, and the auxiliary post-filter suppresses the noise residue that may appear in the non-speech segment and outputs a complex-valued spectrum.

4. The speech enhancement method based on air-bone conduction dual-mode deep learning according to claim 1 is characterized in that: The dataset is divided into 80% training set, 10% validation set, and 10% test set.

5. The speech enhancement method based on air-bone conduction dual-mode deep learning according to claim 1, characterized in that: The speech data of the training set is cut into training set data according to a fixed length of 20 ms.

6. The speech enhancement method based on air-bone dual-mode deep learning according to claim 1, characterized in that: When constructing the Taylor Lite neural network model, a 4th-order Taylor expansion form is set, and the 4th-order Taylor expansion form includes a 0th-order amplitude spectrum estimation module and high-order 1st, 2nd, and 3rd-order complex residual estimation modules.

7. The speech enhancement method based on air-bone dual-mode deep learning according to claim 3 is characterized in that: For the scenario of air-bone-conducted speech noise reduction, the input channel of the high-order encoder in step 3-1 is 4; for the scenario of air-conducted speech noise reduction, the input channel of the high-order encoder in step 3-1 is 2.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute any one of claims 1 to 7.

10. A computer program product, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 7.

Images