Speech enhancement method based on distributed optical fiber acoustic wave sensing system

Abstract

The present application discloses a speech enhancement method based on a distributed optical fiber acoustic wave sensing system, and relates to the technical field of speech enhancement. The speech enhancement network constructed in the present application includes an encoder module, the output end of the encoder module is sequentially connected to the first Reshape layer, the first TCNS module, the second TCNS module, the first GRU module, the second GRU module, the second Reshape layer and the Add layer, the output end of the encoder module is also connected to the input end of the Add layer, the output end of the Add layer is connected to the decoder module, and the encoder is also connected to the decoder; the decoder in the present application can obtain a predicted enhanced complex spectrum, and the time domain waveform of the enhanced time domain speech signal obtained by the speech enhancement method described in the present application will show less noise components in terms of fit and similarity, and also has more speech features that are the same as the original speech signal.

Description

Technical Field

[0001] The present invention relates to the technical field of speech enhancement, and in particular to a speech enhancement method based on a distributed optical fiber acoustic wave sensing system. Background Art

[0002] Traditional speech enhancement methods mainly include spectral subtraction, subspace algorithm, wavelet transform and other methods. Traditional speech enhancement methods usually have poor speech enhancement effects on speech signals with more noise sources collected by DAS acquisition systems. This is mainly because traditional speech enhancement methods will have signal distortion or pseudo-sound problems while denoising, which affects the speech quality. In order to improve the speech quality, researchers usually need to manually adjust parameters based on their professional knowledge and experience. The operation is cumbersome, and the speech enhancement effect will be affected by human factors and is unstable. To this end, this application studies a speech enhancement method based on a distributed fiber optic acoustic wave sensing system. Summary of the invention

[0003] Based on the above problems, the present invention provides a speech enhancement method based on a distributed optical fiber acoustic wave sensing system.

[0004] The present invention is achieved through the following technical solutions:

[0005] A speech enhancement method based on a distributed optical fiber acoustic wave sensing system comprises the following steps:

[0006] S1, using the original speech signal and the distributed optical fiber acoustic sensor system (DAS) to collect the time domain speech signal to obtain the initial time domain speech signal; then, the initial time domain speech signal is framed to obtain the initial speech data set;

[0007] S2, dividing the time domain speech signal samples in the initial speech data set to obtain a training set, a validation set, and a test set; performing short-time Fourier transform on the time domain speech signal samples in the training set, the validation set, and the test set to obtain the complex spectra of the training set, the validation set, and the test set;

[0008] S3, constructing a speech enhancement network; the speech enhancement network includes an encoder module, the output end of the encoder module is sequentially connected to the first Reshape layer, the first TCNS module, the second TCNS module, the first GRU module, the second GRU module, the second Reshape layer and the Add layer, the output end of the encoder module is also connected to the input end of the Add layer, the output end of the Add layer is connected to the decoder module, and the encoder is also connected to the decoder; the encoder is used to obtain a high-dimensional feature map that integrates real deep features and imaginary deep features, the first Reshape layer is used to flatten the feature map output by the Add layer to obtain a one-dimensional feature sequence; the first TCNS module is used to process the one-dimensional feature sequence output by the first Reshape layer, so as to obtain a feature sequence with enhanced temporal context correlation information; the second TCNS module is used to process the first T The feature sequence output by the CNS module is processed to obtain a feature sequence with further enhanced temporal context correlation information; the first GRU module is used to process the feature sequence output by the second TCNS module to obtain a feature sequence with key speech features; the second GRU module is used to process the feature sequence output by the first GRU module to obtain a feature sequence with rich key speech features; the second Reshape layer converts the feature sequence with rich key speech features output by the second GRU module into a complex spectrum feature map; the Add layer connected to the second Reshape layer is used to add the complex spectrum feature map output by the second Reshape layer and the high-dimensional feature map output by the Add layer in the encoder module to obtain a complex spectrum feature map with rich key speech features and high-dimensional features; the decoder is used to obtain a predicted enhanced complex spectrum;

[0009] S4. Train the speech enhancement network using the training set and the SmoothL1 loss function, and verify the speech enhancement network using the verification set to obtain a speech enhancement network model;

[0010] S5. The speech signal to be enhanced is framed according to the frame processing step described in S1, and then the short-time Fourier transform step described in step S2 is used to obtain the complex spectrum corresponding to the speech signal to be enhanced, and then the complex spectrum is input into the speech enhancement network model and forward propagated once to obtain the enhanced complex spectrum; then, the enhanced complex spectrum is inversely short-time Fourier transformed to obtain the time domain speech signal after speech enhancement.

[0011] Preferably, in step S1, the initial time domain speech signal is framed, and the specific steps are as follows:

[0012] The initial time domain speech signal is cut into time domain speech signal units of 4 seconds in length to obtain time domain speech signal samples, and there is an overlap of 2 seconds between adjacent time domain speech signal samples; for time domain speech signal samples less than 4 seconds, zero padding is performed by adding a silent part at the end.

[0013] Preferably, in step S1, the distributed fiber optic acoustic wave sensing system includes an ultra-narrow linewidth laser, a coupler I, an acousto-optic modulator, an erbium-doped fiber amplifier I, and a circulator connected in sequence, the erbium-doped fiber amplifier I is connected to port 1 of the circulator, the port 2 of the circulator is connected to the single-mode optical fiber, and the port 3 of the circulator is connected to the erbium-doped fiber amplifier II, an optical filter, a coupler II, a photodetector, a data acquisition card, and a host computer in sequence.

[0014] Preferably, in step S2, the time domain speech signal samples in the initial speech data set are divided to obtain a training set, a validation set and a test set, including the following steps: 300 time domain speech signal samples are randomly selected from the initial speech data set to form a test set for speech enhancement quality testing; then, the remaining time domain speech signal samples in the initial speech data set are divided in a ratio of 8:2 to obtain a training set and a validation set, respectively.

[0015] Preferably, in step S3, the encoder module includes two branch units I with the same structure and an Add layer connected to the two branch units I, and the Add layer is connected to the first Reshape layer. In the present application, the first branch unit I is used to process the real part of the complex spectrum, and a high-dimensional feature map with deep features of the real part can be obtained; the second branch unit I is used to process the imaginary part of the complex spectrum, and a high-dimensional feature map with deep features of the imaginary part can be obtained; the Add layer is used to add the features of the high-dimensional feature map with deep features of the real part output by the first branch unit I and the high-dimensional feature map with deep features of the imaginary part output by the second branch unit I, so as to obtain a high-dimensional feature map with deep features of the real part and deep features of the imaginary part; in step S3, the decoder module includes two branch units II with the same structure and an Add layer connected to the two branch units II; the input ends of the two branch units II are connected to the output end of the Add layer connected to the output end of the second Reshape layer. The first branch unit II is used to process the real part of the complex spectrum feature map output by the Add layer connected to the second Reshape layer to obtain a predicted enhanced real part; the second branch unit II is used to process the imaginary part of the complex spectrum feature map output by the Add layer connected to the second Reshape layer to obtain a predicted enhanced imaginary part; the predicted enhanced real part output by the deconvolution unit in the first branch unit II and the predicted enhanced imaginary part output by the deconvolution unit in the second branch unit II are added through the Add layer connected to the two to obtain a predicted enhanced complex spectrum.

[0016] Preferably, in step S3, the branch unit I includes six sequentially connected convolution blocks; the branch unit II includes five sequentially connected deconvolution blocks and a deconvolution unit connected to the last deconvolution block, and an Add layer is connected between two adjacent deconvolution blocks, and an Add layer is also connected between the last deconvolution block and the deconvolution unit; the convolution block in the first branch unit I and the deconvolution block in the first branch unit II are connected in a jump manner, and the convolution block in the second branch unit I and the deconvolution block in the second branch unit II are connected in a jump manner.

[0017] Preferably, the convolution block comprises a causal convolution layer, a batch normalization layer, a PReLU activation layer, a SGE layer and an Add layer which are connected in sequence, and the output end of the PReLU activation layer is also connected to the input end of the Add layer.

[0018] Preferably, in step S3, the deconvolution block includes a causal deconvolution layer, a batch normalization layer, a PReLU activation layer, a SA-NET module and an Add layer connected in sequence, and the output end of the PReLU activation layer is also connected to the input end of the Add layer; the deconvolution unit includes a causal deconvolution layer, a batch normalization layer and a PReLU activation layer connected in sequence.

[0019] Preferably, in step S3, the TCNS module includes six sequentially connected TCNBlock layers, the TCNBlock layers include a one-dimensional dilated causal convolution layer (convolution kernel size is 3), a first batch normalization layer, a GELU activation layer, a first Dropout layer, a convolution layer (convolution kernel size is 1×1), a second batch normalization layer, a PReLU activation layer, a second Dropout layer and an Add layer, and the input end of the one-dimensional dilated causal convolution layer is also connected to the input end of the Add layer.

[0020] Preferably, in step S3, the expansion rate d of the one-dimensional dilated causal convolutional layer in the six TCNBlock layers is calculated according to formula (1):

[0021] d= (1)

[0022] In formula (1), Indicates the layer number of the TCNBlock layer; in this application, the TCNS module includes six TCNBlock layers connected in sequence, which can also be specifically expressed as: the TCNS module includes the first TCNBlock layer, the second TCNBlock layer, the third TCNBlock layer, the fourth TCNBlock layer, the fifth TCNBlock layer, and the sixth TCNBlock layer connected in sequence, and the layer number of the first TCNBlock layer is That is 1. Similarly, the number of layers from the second TCNBlock layer to the sixth TCNBlock layer is They are 2, 3, 4, 5 and 6 respectively.

[0023] Preferably, step S4 specifically includes the following steps:

[0024] S4-1, frame the original speech signal according to the frame processing steps described in S1, and use the short-time Fourier transform step described in step S2 to obtain the complex spectrum, which is the target complex spectrum; input the complex spectrum of the training set into the speech enhancement network and forward propagate it once to obtain the predicted enhanced complex spectrum; use the SmoothL1 loss function to calculate the loss value between the predicted enhanced complex spectrum and the target complex spectrum, and use the AdamW optimizer to optimize the loss gradient and back-propagate, save the parameters and update the model parameters of the speech enhancement network, and obtain the speech enhancement network with updated model parameters;

[0025] S4-2, input the complex spectrum of the verification set into the speech enhancement network whose model parameters have been updated in step S4-1, forward propagate once, and output the loss value;

[0026] S4-3. After completing step S4-1 and step S4-2 in the present application, the training of one epoch is completed; repeat the training, and when the number of training epochs reaches the preset 200 times, the parameters retained in the training epoch with the smallest loss value obtained in the process of verifying the speech enhancement network 200 times using the complex spectrum of the verification set are used as the final model parameters of the speech enhancement network. The speech enhancement network with the final model parameters is the speech enhancement network model.

[0027] Compared with the prior art, the beneficial technical effects of the present invention are:

[0028] In the present application, the first branch unit I of the encoder processes the real part of the complex spectrum to obtain a high-dimensional feature map with real deep features; the second branch unit I of the encoder processes the imaginary part of the complex spectrum to obtain a high-dimensional feature map with imaginary deep features; the Add layer in the encoder is used to add the high-dimensional feature map with real deep features output by the first branch unit I and the high-dimensional feature map with imaginary deep features output by the second branch unit I to obtain a high-dimensional feature map that combines real deep features and imaginary deep features, and the first Reshape layer flattens the feature map output by the Add layer into a one-dimensional feature sequence; after the first TCNS module processes the one-dimensional feature sequence output by the first Reshape layer, it can obtain The second TCNS module is used to process the feature sequence output by the first TCNS module to obtain a feature sequence with further enhanced temporal context correlation information; the first GRU module is used to control the storage and forgetting of information in the feature sequence output by the second TCNS module through the update gate and the reset gate, effectively filter out unnecessary information, and obtain a feature sequence with key speech features; the second GRU module is used to process the feature sequence output by the first GRU module to obtain a feature sequence with rich key speech features; the second Reshape layer converts the feature sequence with rich key speech features output by the second GRU module into a complex spectral feature map;

[0029] The Add layer connected to the second Reshape layer is used to add the complex spectrum feature map output by the second Reshape layer and the high-dimensional feature map output by the Add layer in the encoder module to obtain a complex spectrum feature map with rich key speech features and high-dimensional features.

[0030] In addition, in the present application, the convolution block in the first branch unit I and the deconvolution block in the first branch unit II adopt a jump connection method, and the convolution block in the second branch unit I and the deconvolution block in the second branch unit II adopt a jump connection method. The setting of the above-mentioned jump connection method can allow information to be transmitted between different scales, and can retain more spatial information and detail features; the real part of the complex spectrum feature map output by the Add layer connected to the second Reshape layer is processed by five sequentially connected deconvolution blocks and a deconvolution unit in the first branch unit II to obtain a predicted enhanced real part; the imaginary part of the complex spectrum feature map output by the Add layer connected to the second Reshape layer is processed by five sequentially connected deconvolution blocks and a deconvolution unit in the second branch unit II to obtain a predicted enhanced imaginary part; the predicted enhanced real part output by the deconvolution unit in the first branch unit II and the predicted enhanced imaginary part output by the deconvolution unit in the second branch unit II are added by the Add layer connected to the two to obtain a predicted enhanced complex spectrum. Through testing, it can be seen that the time domain waveform of the enhanced time domain speech signal obtained by the speech enhancement method described in the present application will show less noise components in terms of fitting and similarity, and also has more speech features that are the same as the original speech signal. Therefore, the enhanced time domain speech signal obtained by the speech enhancement method described in the present application is closer to the speech features of the original speech signal, and the amplitude fluctuation of the time domain waveform of the enhanced time domain speech signal is significantly reduced, which shows that the speech enhancement method described in the present application can effectively suppress the noise in the initial time domain speech signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a schematic diagram of the structure of the distributed optical fiber acoustic wave sensing system of this application;

[0032] Figure 2 A schematic diagram of the network structure of the speech enhancement network in the present invention;

[0033] Figure 3 for Figure 2 Schematic diagram of the network structure of the convolutional block in the encoder module;

[0034] Figure 4 for Figure 2 Schematic diagram of the network structure of the deconvolution block in the decoder module;

[0035] Figure 5 for Figure 2 Schematic diagram of the network structure of the deconvolution unit in the decoder module;

[0036] Figure 6 for Figure 2 Schematic diagram of the network structure of the TCNBlock layer in the TCNS module;

[0037] Figure 7 It is a time domain waveform comparison of an initial time domain speech signal, an original speech signal, and an enhanced time domain speech signal obtained by the speech enhancement method described in the present application based on the initial time domain speech signal. DETAILED DESCRIPTION

[0038] A speech enhancement method based on a distributed optical fiber acoustic wave sensing system comprises the following steps:

[0039] S1. Using the original speech signal and the distributed optical fiber acoustic wave sensing system (DAS) to collect the time domain speech signal, an initial time domain speech signal is obtained, and then the initial time domain speech signal is framed to obtain an initial speech data set; in the present application, the initial time domain speech signal is a noisy time domain speech signal, and the original speech signal is a noise-free speech signal. More specifically, the original speech signal is a noise-free speech signal corresponding to the initial time domain speech signal; the sampling frequencies of the initial time domain speech signal and the original speech signal are both 16kHz, and the above original speech signal comes from the CSTR VCTK corpus;

[0040] In this application, the initial time domain speech signal is framed, and the specific steps are as follows:

[0041] The initial time-domain speech signal is cut into pieces according to the length of a time-domain speech signal unit of 4s to obtain time-domain speech signal samples, and there is a 2s overlap of the time-domain speech signal between adjacent time-domain speech signal samples, for example, the last 2s of the time-domain speech signal of the previous time-domain speech signal sample overlaps with the first 2s of the time-domain speech signal of the next time-domain speech signal sample adjacent thereto; such a setting can make the two adjacent time-domain speech signal samples smooth and continuous, and reduce the influence of the edge effect on feature extraction; and the time-domain speech signal samples less than 4s are padded with zeros by adding a silent part at the end; after the framing is completed, the above-mentioned time-domain speech signal samples constitute the initial speech data set;

[0042] In addition, in this application, a distributed optical fiber acoustic sensing system (DAS), such as Figure 1 As shown, it includes an ultra-narrow linewidth laser, a coupler I, an acousto-optic modulator, an erbium-doped fiber amplifier I, and a circulator connected in sequence. The erbium-doped fiber amplifier I is connected to port 1 of the circulator, and port 2 of the circulator is connected to a single-mode optical fiber. Port 3 of the circulator is connected to an erbium-doped fiber amplifier II, an optical filter, a coupler II, a photodetector, a data acquisition card, and a host computer in sequence. In this application, the ultra-narrow line laser uses a 3kHz ultra-narrow line laser, the sampling rate of the distributed optical fiber acoustic sensor (DAS) is set to 16kHz, and the number of channels is set to 1.

[0043] The time-domain speech signal sampling principle of the distributed fiber acoustic sensor (DAS) of the present application is as follows: an ultra-narrow line laser emits laser light, coupler I couples the laser light emitted by the ultra-narrow line laser, and then the laser light is modulated into a pulse light sequence by an acousto-optic modulator, and then amplified by an erbium-doped fiber amplifier I. The amplified pulse light is input into the single-mode optical fiber through port 1 and port 2 of the circulator, and the pulse light sequence is transmitted along the single-mode optical fiber to generate Rayleigh scattered light. When a sound signal is received near the optical fiber, backward Rayleigh scattered light is generated and returns to the circulator along the single-mode optical fiber and port 2 of the circulator, and then enters the erbium-doped fiber amplifier II through port 3 of the circulator for amplification, and then filtered by an optical filter, and then coupled by coupler II, and then the photodetector converts the optical signal output by coupler II into an electrical signal, and then the data acquisition card performs acquisition, and the acquired digital signal is transmitted to the host computer for processing, and the obtained time-domain speech signal is the initial time-domain speech signal. When the initial time domain voice signal collected in the present application is a noisy time domain voice signal, when collecting the initial time domain voice signal in the present application, a working fan is placed on the single-mode optical fiber, and the noise generated by the working fan is used to simulate the noise that may occur in the real environment.

[0044] S2, dividing the time domain speech signal samples in the initial speech data set to obtain a training set, a validation set and a test set; performing short-time Fourier transform on the time domain speech signal samples in the training set, the validation set and the test set to obtain a complex spectrum of the training set, a complex spectrum of the validation set and a complex spectrum of the test set respectively;

[0045] Among them, the time domain speech signal samples in the initial speech data set are divided to obtain the training set, the validation set and the test set, including the following steps: 300 time domain speech signal samples are randomly selected from the initial speech data set to form a test set for speech enhancement quality testing; then, the remaining time domain speech signal samples in the initial speech data set are divided according to the ratio of 8:2 to obtain the training set and the validation set respectively.

[0046] In the present application, the obtained complex spectrum is a complex matrix containing frequency and time information. The complex spectrum includes real and imaginary parts, which are used to reflect the amplitude and phase information of the speech signal in the frequency domain. In the present application, the time and frequency dimensions of the real part are 401×161, and the time and frequency dimensions of the imaginary part are 401×161; moreover, the short-time Fourier transform process uses a 20ms Hamming window to continuously slide and intercept speech signals in different time periods for Fourier transform. The window length is the number of Fourier transform points. The speech signal in each window is relatively stable, and there is a 50% overlap between adjacent windows, that is, the window size is 320 points and the step size is 160 points.

[0047] S3, build a speech enhancement network;

[0048] The structure of the speech enhancement network in the present application is shown in Figure 2. The speech enhancement network includes an encoder module, the output end of the encoder module is sequentially connected to the first Reshape layer, the first TCNS module, the second TCNS module, the first GRU module, the second GRU module, the second Reshape layer and the Add layer, and the output end of the encoder module is also connected to the input end of the Add layer; the output end of the Add layer is connected to the decoder module, and the encoder is also connected to the decoder. In the present application, the encoder is used to obtain a high-dimensional feature map that integrates real deep features and imaginary deep features, the first Reshape layer is used to flatten the feature map output by the Add layer to obtain a one-dimensional feature sequence; the first TCNS module is used to process the one-dimensional feature sequence output by the first Reshape layer to obtain a feature sequence with enhanced temporal context correlation information; the second TCNS module is used to process the feature sequence output by the first TCNS module to obtain a feature sequence with further enhanced temporal context correlation information; the first GRU module is used to process the feature sequence output by the second TCNS module to obtain a feature sequence with key speech features; the second GRU module is used to process the feature sequence output by the first GRU module to obtain a feature sequence with rich key speech features; the second Reshape layer converts the feature sequence with rich key speech features output by the second GRU module into a complex spectrum feature map; the Add layer connected to the second Reshape layer is used to add the complex spectrum feature map output by the second Reshape layer and the high-dimensional feature map output by the Add layer in the encoder module to obtain a complex spectrum feature map with rich key speech features and high-dimensional features; the decoder is used to obtain a predicted enhanced complex spectrum;

[0049] The encoder module includes two branch units I with the same structure and an Add layer connected to the two branch units I, and the Add layer is connected to the first Reshape layer;

[0050] The decoder module includes two branch units II with the same structure and an Add layer connected to the two branch units II; the input ends of the two branch units II are connected to the output end of the Add layer connected to the output end of the second Reshape layer.

[0051] In the present application, the branch unit I includes six sequentially connected convolution blocks; the branch unit II includes five sequentially connected deconvolution blocks and a deconvolution unit connected to the last deconvolution block, and an Add layer is connected between two adjacent deconvolution blocks, and an Add layer is also connected between the last deconvolution block and the deconvolution unit;

[0052] The output end of the Add layer in the encoder module is connected to the first Reshape layer, the first TCNS module, the second TCNS module, the first GRU module, the second GRU module, the second Reshape layer and another Add layer in sequence, and the output end of the Add layer in the encoder module is also connected to the input end of the Add layer connected to the output end of the second Reshape layer;

[0053] In the present application, the convolution block in the first branch unit I and the deconvolution block in the first branch unit II adopt a jump connection method, specifically as follows: the output end of the first convolution block in the first branch unit I is also connected to the input end of the Add layer located between the fifth deconvolution block and the deconvolution unit in the first branch unit II; the output end of the second convolution block in the first branch unit I is also connected to the input end of the Add layer located between the fourth deconvolution block and the fifth deconvolution block in the first branch unit II; the output end of the third convolution block in the first branch unit I is also connected to the input end of the third deconvolution block in the first branch unit II The output end of the fourth convolution block in the first branch unit I is also connected to the input end of the Add layer between the second deconvolution block and the third deconvolution block in the first branch unit II; the output end of the fifth convolution block in the first branch unit I is also connected to the input end of the Add layer between the first deconvolution block and the second deconvolution block in the first branch unit II; the sixth convolution block in the first branch unit I and the sixth convolution block in the second branch unit I are connected to the Add layer connecting the first branch unit I and the second branch unit I;

[0054] In the present application, the convolution block in the second branch unit I and the deconvolution block in the second branch unit II adopt a jump connection method, specifically as follows: the output end of the first convolution block in the second branch unit I is also connected to the input end of the Add layer between the fifth deconvolution block and the deconvolution unit in the second branch unit II; the output end of the second convolution block in the second branch unit I is also connected to the input end of the Add layer between the fourth deconvolution block and the fifth deconvolution block in the second branch unit II; the output end of the third convolution block in the second branch unit I is also connected to the input end of the third deconvolution block in the second branch unit II The output end of the fourth convolution block in the second branch unit I is also connected to the input end of the Add layer between the second deconvolution block and the third deconvolution block in the second branch unit II; the output end of the fifth convolution block in the second branch unit I is also connected to the input end of the Add layer between the first deconvolution block and the second deconvolution block in the second branch unit II; the deconvolution unit in the first branch unit II and the deconvolution unit in the second branch unit II are connected to the input end of the Add layer connecting the first branch unit II and the second branch unit II;

[0055] In branch unit I of the present application, the structure of the convolution block, as shown in Figure 3, includes a causal convolution layer, a batch normalization layer, a PReLU activation layer, an SGE layer and an Add layer connected in sequence, and the output end of the PReLU activation layer is also connected to the input end of the Add layer; wherein, the convolution kernel size of the causal convolution layer is 3×5, the time dimension step size of the causal convolution layer is set to 1, and the frequency dimension step size is set to 2; in the present application, the step size of the causal convolution layer along the time dimension in branch unit I is set to 1, which can better retain the temporal characteristics of the original signal of the initial time domain speech signal in the time dimension, and effectively avoid excessive loss of time-related information. The step size of the causal convolution layer along the frequency dimension in branch unit I of the present application is 2, which is used to downsample the frequency dimension of the feature map to obtain a larger context in the frequency direction.

[0056] In branch unit II, the structure of the deconvolution block, as shown in Figure 4, includes a causal deconvolution layer, a batch normalization layer, a PReLU activation layer, a SA-NET module and an Add layer connected in sequence, and the output end of the PReLU activation layer is also connected to the input end of the Add layer; the convolution kernel size of the causal deconvolution layer is 3×5, and the time dimension step length of the causal deconvolution layer is set to 1, and the frequency dimension step length is set to 2; in the present application, the step length of the causal deconvolution layer along the time dimension in branch unit II is set to 1, which can better retain the temporal characteristics of the original signal of the initial time domain speech signal in the time dimension, and effectively avoid excessive loss of time-related information. The step length of the causal deconvolution layer along the frequency dimension in branch unit II of the present application is 2, which is used to downsample the frequency dimension of the feature map to obtain a larger context in the frequency direction.

[0057] In this application, the structure of the deconvolution unit is as follows: Figure 5 As shown, it includes a causal deconvolution layer, a batch normalization layer, and a PReLU activation layer connected in sequence; the functions of the causal deconvolution layer, the batch normalization layer, and the PReLU activation layer in the deconvolution unit are the same as those of the causal deconvolution layer, the batch normalization layer, and the PReLU activation layer in the deconvolution block, respectively.

[0058] In this application, the TCNS module includes six TCNBlock layers connected in sequence, and the structure of each TCNBlock is as follows: Figure 6 As shown, it includes a one-dimensional dilated causal convolution layer (convolution kernel size is 3), the first batch normalization layer, the GELU activation layer, the first Dropout layer, the convolution layer (convolution kernel size is 1×1), the second batch normalization layer, the PReLU activation layer, the second Dropout layer and the Add layer, and the input end of the one-dimensional dilated causal convolution layer is also connected to the input end of the Add layer; among them, the expansion rate d of the one-dimensional dilated causal convolution layer in the six TCNBlock layers is calculated according to formula (1):

[0059] d= (1)

[0060] In formula (1), Indicates the layer number of the TCNBlock layer; in this application, the TCNS module includes six TCNBlock layers connected in sequence, which can also be specifically expressed as: the TCNS module includes the first TCNBlock layer, the second TCNBlock layer, the third TCNBlock layer, the fourth TCNBlock layer, the fifth TCNBlock layer, and the sixth TCNBlock layer connected in sequence, and the layer number of the first TCNBlock layer is That is 1. Similarly, the number of layers from the second TCNBlock layer to the sixth TCNBlock layer is They are 2, 3, 4, 5, and 6 respectively;

[0061] Functions of each module in the speech enhancement module of this application:

[0062] In the present application, the first branch unit I of the encoder processes the real part of the complex spectrum to obtain a high-dimensional feature map with real deep features; the first branch unit II of the encoder processes the imaginary part of the complex spectrum to obtain a high-dimensional feature map with imaginary deep features; the Add layer in the encoder is used to add the high-dimensional feature map with real deep features output by the first branch unit I and the high-dimensional feature map with imaginary deep features output by the second branch unit I to obtain a high-dimensional feature map that combines real deep features and imaginary deep features, and the first Reshape layer flattens the feature map output by the Add layer into a one-dimensional feature sequence; after the first TCNS module processes the one-dimensional feature sequence output by the first Reshape layer, it can obtain A feature sequence with enhanced temporal context correlation information; the second TCNS module is used to process the feature sequence output by the first TCNS module, and can obtain a feature sequence with further enhanced temporal context correlation information; the first GRU module is used to control the storage and forgetting of information in the feature sequence output by the second TCNS module through the update gate and the reset gate, effectively filter out unnecessary information, and obtain a feature sequence with key speech features; the second GRU module is used to process the feature sequence output by the first GRU module, and obtain a feature sequence with rich key speech features; the second Reshape layer converts the feature sequence with rich key speech features output by the second GRU module into a complex spectral feature map;

[0063] The Add layer connected to the second Reshape layer is used to add the complex spectrum feature map output by the second Reshape layer and the high-dimensional feature map output by the Add layer in the encoder module to obtain a complex spectrum feature map with rich key speech features and high-dimensional features;

[0064] In addition, in the present application, the convolution block in the first branch unit I and the deconvolution block in the first branch unit II adopt a jump connection method, and the convolution block in the second branch unit I and the deconvolution block in the second branch unit II adopt a jump connection method. The setting of the above-mentioned jump connection method allows information to be transmitted between different scales, and can retain more spatial information and detail features; the real part of the complex spectrum feature map output by the Add layer connected to the second Reshape layer is processed by five sequentially connected deconvolution blocks and a deconvolution unit in the first branch unit II to obtain a predicted enhanced real part; the imaginary part of the complex spectrum feature map output by the Add layer connected to the second Reshape layer is processed by five sequentially connected deconvolution blocks and a deconvolution unit in the second branch unit II to obtain a predicted enhanced imaginary part; the predicted enhanced real part output by the deconvolution unit in the first branch unit II and the predicted enhanced imaginary part output by the deconvolution unit in the second branch unit II are added by the Add layer connected to the two to obtain a predicted enhanced complex spectrum;

[0065] The functions of each module in the convolution block of the present application are as follows: the causal convolution layer extracts the local detail features of the speech signal in the feature map input therein through a downsampling operation, while reducing the spatial resolution of the feature map and increasing the number of channels of the feature map to obtain a feature map with local detail features of the speech signal; the batch normalization layer is used to perform batch normalization operations on the feature map output by the causal convolution layer; the PReLU activation layer is used to activate the feature map output by the batch normalization layer; the SGE module is used to perform spatial feature enhancement and spatial noise reduction on the feature map output by the PReLU activation layer to obtain a feature map with enhanced spatial features and reduced spatial noise; the Add layer is used to add the features of the feature map output by the PReLU activation layer and the feature map output by the SGE module to obtain a fused feature map with local detail features of the speech signal, enhanced spatial position information, and reduced spatial noise. The SGE module used in the present application has the same structure and function as the SGE module disclosed in the paper "Spatial Group-wise Enhance: Enhancing Semantic Feature Learning in CNN". In the present application, the real part of the complex spectrum is input to the first branch unit I, and the first branch unit I is composed of six convolution blocks connected in sequence. The real part of the complex spectrum is processed by six convolution blocks to obtain a high-dimensional feature map with deep features of the real part; the imaginary part of the complex spectrum is input to the second branch unit I, and the second branch unit I is composed of six convolution blocks connected in sequence. The imaginary part of the complex spectrum is processed by six convolution blocks to obtain a high-dimensional feature map with deep features of the imaginary part.

[0066] The functions of each module in the deconvolution block in the present application are as follows: the causal deconvolution layer realizes the amplification of the feature map through the transposed convolution operation, which is used to restore the size and frequency resolution of the feature map; the batch normalization layer is used to perform batch normalization operation on the feature map output by the causal deconvolution layer; the PReLU activation layer is used to perform nonlinear transformation on the feature map output by the batch normalization layer, so that the speech enhancement network can learn more complex speech signal features and relationships; the SA-NET module is used to rearrange the channel features and adjust the attention of the feature map output by the PReLU activation layer to obtain a feature map with prominent key speech features; the Add layer is used to add the features of the feature map output by the PReLU activation layer and the feature map output by the SA-NET module to improve the feature diversity and obtain a comprehensive feature map with prominent key speech features; the SA-NET module used in the present application has the same structure and function as the SA-NET module disclosed in the paper "SA-Net: Shuffle Attention for Deep Convolutional Neural Networks".

[0067] The first TCNS module and the second TCNS module of the present application have the same structure and the same working principle. The present application takes the first TCNS module as an example to introduce the functions of each module, which are as follows: the one-dimensional dilated causal convolution layer in the TCNBlock layer is used to extract the temporal correlation information of the speech signal in the feature sequence input therein (i.e., the one-dimensional feature sequence output by the first Reshape layer) to obtain a feature sequence with temporal correlation information of the speech signal; the first batch normalization layer is used to normalize the features output by the one-dimensional dilated causal convolution layer, the GELU activation layer is used to perform activation operations, and the first Dropout layer is used to perform random inactivation processing on the feature sequence output by the GELU activation layer to avoid overfitting of specific features or noise in the training data, and the inactivation rate parameter of the first Dropout layer is set to 0 .3; the convolution layer (convolution kernel size is 1×1) is used to fuse the channel features in the feature map output by the first Dropout layer to obtain a feature sequence of cross-channel interaction and information fusion; the second batch normalization layer is used to perform batch normalization on the features output by the convolution layer (convolution kernel size is 1×1) to further stabilize the distribution of features, the PReLU activation layer activates the feature sequence output by the second batch normalization layer, and the second Dropout layer performs random inactivation on the feature sequence output by the PReLU activation layer. The inactivation rate parameter of the second Dropout layer is set to 0.2; the Add layer is used to add the feature sequence output by the second Dropout layer and the feature sequence input to the one-dimensional dilated causal convolution layer to obtain a feature sequence with enhanced temporal context correlation information. The TCNS module of the present application includes six TCNBlock layers connected in sequence, and the expansion rates of the one-dimensional dilated causal convolution layers in the six TCNBlock layers are different, which leads to different receptive fields of the one-dimensional dilated causal convolution layers in the six TCNBlock layers, and further leads to different scales of the temporal correlation information of the speech signals extracted by the one-dimensional dilated causal convolution layers in the six TCNBlock layers.

[0068] S4, using the training set and the SmoothL1 loss function to train the speech enhancement network, and using the verification set to verify the speech enhancement network, to obtain a speech enhancement network model, specifically including the following steps:

[0069] S4-1. First, the original speech signal is framed according to the frame processing steps described in S1, and then the short-time Fourier transform step described in step S2 is used to obtain the complex spectrum corresponding to the original speech signal, and the complex spectrum corresponding to the original speech signal is the target complex spectrum; the complex spectrum of the training set is input into the speech enhancement network for forward propagation once to obtain the predicted enhanced complex spectrum; wherein the real part of the complex spectrum is input into the first branch unit I of the speech enhancement network, and the imaginary part of the complex spectrum is input into the second branch unit I of the speech enhancement network; the loss value between the predicted enhanced complex spectrum and the target complex spectrum is calculated using the SmoothL1 loss function, and the loss gradient is optimized and back-propagated using the AdamW optimizer, the parameters are saved and the model parameters of the speech enhancement network are updated to obtain the speech enhancement network with the updated model parameters; in the present application, the training process of the speech enhancement network is developed based on the pytorch platform. During the training process, the initial learning rate is set to 0.0001, and the batch size of the complex spectrum input to the speech enhancement network is set to 16;

[0070] S4-2, input the complex spectrum of the validation set into the speech enhancement network whose model parameters have been updated in step S4-1 (the real part of the complex spectrum is input into the first branch unit I of the speech enhancement network, and the imaginary part of the complex spectrum is input into the second branch unit I of the speech enhancement network), forward propagate once, and output the loss value;

[0071] S4-3. After completing step S4-1 and step S4-2 in the present application, one epoch of training is completed; that is, after the present application trains the speech enhancement network once using the complex spectrum of the training set, it verifies it once using the complex spectrum of the verification set, and completes one epoch of training; repeat the training, and when the number of training epochs reaches the preset 200 times, the parameters retained in the training epoch with the smallest loss value obtained in the process of verifying the speech enhancement network 200 times using the complex spectrum of the verification set are used as the final model parameters of the speech enhancement network, and the speech enhancement network with the final model parameters is the speech enhancement network model.

[0072] S5. The speech signal to be enhanced is framed according to the frame processing steps described in S1, and then the short-time Fourier transform step described in step S2 is used to obtain the complex spectrum corresponding to the speech signal to be enhanced, and then the complex spectrum is input into the speech enhancement network model for forward propagation once to obtain the enhanced complex spectrum; then, the enhanced complex spectrum is inversely short-time Fourier transformed to obtain the time domain speech signal after speech enhancement. In the present application, the complex spectrum is input into the speech enhancement network model, specifically: the real part of the complex spectrum is input into the first branch unit I of the speech enhancement network model, and the imaginary part of the complex spectrum is input into the second branch unit I of the speech enhancement network model.

[0073] Comparison of ablation experiment results:

[0074] This application Figure 2 In the speech enhancement network shown, the first TCNS module, the second TCNS module, the first GRU module and the second GRU module connected in sequence are replaced with GRU modules to obtain a CED-GRU network. That is, the network structure of the CED-GRU network in the present application is an encoder module-GRU module-decoder module;

[0075] This application Figure 2 The first TCNS module, the second TCNS module, the first GRU module and the second GRU module connected in sequence in the speech enhancement network shown are replaced with TCNS modules to obtain a CED-TCNS network. That is, the network structure of the CED-TCNS network in the present application is an encoder module-TCNS module-decoder module;

[0076] The speech enhancement network, CED-GRU network and CED-TCNS network were tested using the complex spectrum of the test set in this application; the test results are shown in Table 1.

[0077] In this application, four test indicators, namely speech perception quality, peak signal-to-noise ratio, scale-invariant distortion ratio and short-time objective intelligibility, are used as evaluation indicators for evaluating the speech enhancement method described in this application and the speech enhancement effects of the above three speech enhancement methods. In Table 1, Ours method represents the speech enhancement method described in this application; PESQ is speech perception quality, and the evaluation result of the PESQ indicator ranges from -0.5 to 4.5, where -0.5 represents very poor speech quality and 4.5 represents very good speech quality; PSNR is peak signal-to-noise ratio, which represents the ratio of signal peak power to noise power; SI-SDR is scale-invariant signal-distortion ratio, which is used to evaluate speech signal separation performance; STOI is short-time objective intelligibility, which is used to evaluate the clarity of quantifiable speech signals, and the score of short-time objective intelligibility is between 0 and 1; the indicators described in the above four test indicators are all higher in value, the better the effect.

[0078] Table 1 Speech quality index test results of enhanced time domain speech signals obtained by different speech enhancement methods

[0079]

[0080] It can be found from Table 1 that the speech enhancement method proposed in the present application has achieved the highest score in all four evaluation indicators. The enhanced time-domain speech signal obtained by the speech enhancement method of the present application has reached 3.04 in the PESQ index, which is (3.04-1.98) / 1.98×100%=53.5% higher than the initial time-domain speech signal; this shows that the speech quality of the enhanced time-domain speech signal obtained by the speech enhancement method of the present application is better;

[0081] The enhanced time-domain speech signal obtained by the speech enhancement method described in the present application reaches 0.92 in the STOI index, which is (0.92-0.72) / 0.72×100%=27.8% higher than the initial time-domain speech signal. This shows that the speech enhancement method described in the present application can effectively improve the clarity and intelligibility of the original time-domain speech signal collected by the distributed optical fiber acoustic wave sensing system when enhancing the original time-domain speech signal collected by the distributed optical fiber acoustic wave sensing system.

[0082] The enhanced time domain speech signal obtained by the speech enhancement method described in the present application is improved by 19.72dB in the PSNR index compared with the initial time domain speech signal, and is improved by 22.21dB in the SI-SDR index compared with the initial time domain speech signal; this shows that the intensity of the time domain speech signal in the enhanced time domain speech signal obtained by the speech enhancement method described in the present application is significantly greater than the noise, and the signal quality of the enhanced time domain speech signal is better.

[0083] In addition, the present application also tests the time domain waveform of the initial time domain speech signal, the original speech signal, and the enhanced time domain speech signal obtained by the speech enhancement method described in the present application based on the initial time domain speech signal. The test results are as follows: Figure 7 As shown, Figure 7 In the figure, the gray waveform is a test sample, i.e., the time domain waveform of the initial time domain speech signal, the blue waveform is the time domain waveform of the corresponding original speech signal, and the red waveform is the time domain waveform of the enhanced time domain speech signal obtained by the speech enhancement method described in the present application; Figure 7 It can be seen from the figure that the time domain waveform of the enhanced time domain speech signal will show less noise components in terms of fit and similarity, and has more speech features that are the same as the original speech signal. Therefore, the enhanced time domain speech signal obtained by the present application is closer to the speech features of the original speech signal; in addition, from Figure 7 It can also be seen that the amplitude fluctuation of the time domain waveform of the enhanced time domain speech signal is significantly reduced, which shows that the speech enhancement method described in the present application can effectively suppress the noise in the initial time domain speech signal.

[0084] The above embodiments and drawings are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. It should be understood by those skilled in the art that any changes, modifications, additions or substitutions made by those skilled in the art within the essential scope of the present invention do not depart from the purpose of the present invention and should also fall within the scope of protection of the claims of the present invention.

Claims

1. A speech enhancement method based on a distributed optical fiber acoustic wave sensing system, characterized in that: The following steps are involved: S1. Using a distributed optical fiber acoustic wave sensing system to collect an initial time-domain speech signal; performing frame processing on the initial time-domain speech signal to obtain an initial speech data set; S2, dividing the time domain speech signal samples in the initial speech data set to obtain a training set, a validation set, and a test set; performing short-time Fourier transform on the time domain speech signal samples in the training set, the validation set, and the test set to obtain the complex spectra of the training set, the validation set, and the test set; S3, construct a speech enhancement network; the speech enhancement network includes an encoder module, the output end of the encoder module is sequentially connected to the first Reshape layer, the first TCNS module, the second TCNS module, the first GRU module, the second GRU module, the second Reshape layer and the Add layer, the output end of the encoder module is also connected to the input end of the Add layer, the output end of the Add layer is connected to the decoder module, and the encoder is also connected to the decoder; wherein, the encoder is used to obtain a high-dimensional feature map fused with real deep features and imaginary deep features, and the first Reshape layer is used to obtain a one-dimensional feature sequence; the first TCNS module is used to obtain a feature sequence with enhanced temporal context relevance information; the second TCNS module is used to obtain a feature sequence with further enhanced temporal context relevance information; the first GRU module is used to obtain a feature sequence with key speech features; the second GRU module is used to obtain a feature sequence with rich key speech features; the second Reshape layer is used to obtain a complex spectrum feature map; the Add layer connected to the second Reshape layer is used to obtain a complex spectrum feature map with rich key speech features and high-dimensional features; the decoder is used to obtain a predicted enhanced complex spectrum; S4. Train the speech enhancement network using the training set and the SmoothL1 loss function, and verify the speech enhancement network using the verification set to obtain a speech enhancement network model; S5. The speech signal to be enhanced is subjected to frame processing and short-time Fourier transform processing in sequence to obtain a complex spectrum, and the complex spectrum is input into the speech enhancement network model for forward propagation once to obtain an enhanced complex spectrum; the enhanced complex spectrum is subjected to inverse short-time Fourier transform to obtain a time-domain speech signal after speech enhancement.

2. The method for speech enhancement based on a distributed optical fiber acoustic wave sensing system according to claim 1, characterized in that: In step S1, the initial time domain speech signal is framed, and the specific steps are as follows: The initial time domain speech signal is cut into time domain speech signal units of 4 seconds in length to obtain time domain speech signal samples, and there is an overlap of 2 seconds between adjacent time domain speech signal samples; for time domain speech signal samples less than 4 seconds, zero padding is performed by adding a silent part at the end.

3. The method for speech enhancement based on a distributed optical fiber acoustic wave sensing system according to claim 1, characterized in that: In step S1, the distributed fiber optic acoustic wave sensing system includes an ultra-narrow linewidth laser, a coupler I, an acousto-optic modulator, an erbium-doped fiber amplifier I, and a circulator connected in sequence, the erbium-doped fiber amplifier I is connected to port 1 of the circulator, the port 2 of the circulator is connected to the single-mode optical fiber, and the port 3 of the circulator is connected to the erbium-doped fiber amplifier II, an optical filter, a coupler II, a photodetector, a data acquisition card, and a host computer in sequence.

4. The method for speech enhancement based on a distributed optical fiber acoustic wave sensing system according to claim 1, characterized in that: In step S3, the encoder module includes two branch units I with the same structure and an Add layer connected to the two branch units I, and the Add layer is connected to the first Reshape layer; in step S3, the decoder module includes two branch units II with the same structure and an Add layer connected to the two branch units II; the input ends of the two branch units II are connected to the output end of the Add layer connected to the output end of the second Reshape layer.

5. The method for speech enhancement based on a distributed optical fiber acoustic wave sensing system according to claim 1, characterized in that: In step S3, the branch unit I includes six sequentially connected convolution blocks; the branch unit II includes five sequentially connected deconvolution blocks and a deconvolution unit connected to the last deconvolution block, and an Add layer is connected between two adjacent deconvolution blocks, and an Add layer is also connected between the last deconvolution block and the deconvolution unit; the convolution block in the first branch unit I and the deconvolution block in the first branch unit II are connected in a jump manner, and the convolution block in the second branch unit I and the deconvolution block in the second branch unit II are connected in a jump manner.

6. The method for speech enhancement based on a distributed optical fiber acoustic wave sensing system according to claim 5, characterized in that: The convolution block includes a causal convolution layer, a batch normalization layer, a PReLU activation layer, a SGE layer and an Add layer which are connected in sequence, and the output end of the PReLU activation layer is also connected to the input end of the Add layer.

7. The method for speech enhancement based on a distributed optical fiber acoustic wave sensing system according to claim 5, characterized in that: In step S3, the deconvolution block includes a causal deconvolution layer, a batch normalization layer, a PReLU activation layer, a SA-NET module and an Add layer connected in sequence, and the output end of the PReLU activation layer is also connected to the input end of the Add layer; the deconvolution unit includes a causal deconvolution layer, a batch normalization layer and a PReLU activation layer connected in sequence.

8. The method for speech enhancement based on a distributed optical fiber acoustic wave sensing system according to claim 1, characterized in that: In step S3, the TCNS module includes six TCNBlock layers connected in sequence, and the TCNBlock layer includes a one-dimensional dilated causal convolution layer with a convolution kernel size of 3, a first batch normalization layer, a GELU activation layer, a first Dropout layer, a convolution layer with a convolution kernel size of 1×1, a second batch normalization layer, a PReLU activation layer, a second Dropout layer and an Add layer connected in sequence, and the input end of the one-dimensional dilated causal convolution layer is also connected to the input end of the Add layer.

9. The method for speech enhancement based on a distributed optical fiber acoustic wave sensing system according to claim 8, characterized in that: In step S3, the expansion rate d of the one-dimensional dilated causal convolutional layer in the six TCNBlock layers is calculated according to formula (1): d= 2 i-1 （1） In formula (1), i represents the layer number of the TCNBlock layer; wherein the TCNS module includes six TCNBlock layers connected in sequence, which can be specifically expressed as follows: the TCNS module includes the first TCNBlock layer, the second TCNBlock layer, the third TCNBlock layer, the fourth TCNBlock layer, the fifth TCNBlock layer, and the sixth TCNBlock layer connected in sequence, and the layer number i of the first TCNBlock layer is 1. Similarly, the layer numbers i of the second TCNBlock layer to the sixth TCNBlock layer are 2, 3, 4, 5, and 6, respectively.

10. The method for speech enhancement based on a distributed optical fiber acoustic wave sensing system according to claim 1, characterized in that: Step S4 specifically includes the following steps: S4-1, frame the original speech signal according to the frame processing steps described in S1, and use the short-time Fourier transform step described in step S2 to obtain the complex spectrum, which is the target complex spectrum; input the complex spectrum of the training set into the speech enhancement network and forward propagate it once to obtain the predicted enhanced complex spectrum; use the SmoothL1 loss function to calculate the loss value between the predicted enhanced complex spectrum and the target complex spectrum, and use the AdamW optimizer to optimize the loss gradient and back-propagate, save the parameters and update the model parameters of the speech enhancement network, and obtain the speech enhancement network with updated model parameters; S4-2, input the complex spectrum of the verification set into the speech enhancement network whose model parameters have been updated in step S4-1, forward propagate once, and output the loss value; S4-3. After completing step S4-1 and step S4-2, the training of one epoch is completed; repeat the training. When the number of training epochs reaches the preset 200 times, the parameters retained in the training epoch with the smallest loss value obtained in the process of verifying the speech enhancement network 200 times using the complex spectrum of the verification set are used as the final model parameters of the speech enhancement network. The speech enhancement network with the final model parameters is the speech enhancement network model.

Images