A voice enhancement method for a ship VHF communication audio signal

Abstract

The application discloses a speech enhancement method for a ship VHF communication audio signal, and comprises the following steps: performing short-time Fourier transform on the audio signal to extract real and imaginary parts of the audio signal; inputting the real and imaginary parts into an encoder to obtain an encoded output feature F K ; inputting the encoded output feature F K into a complex Conformer module to obtain a global feature H; reshaping the global feature H to obtain a reshaped feature H'; inputting the reshaped feature H' and the encoded output feature F K into a decoder as input information to obtain a complex scaling mask G K ; the complex scaling mask G K is a complex scaling mask of the VHF communication audio signal; and based on the complex scaling mask G K , an enhanced speech signal complex spectrum is obtained, and the enhanced speech signal complex spectrum is subjected to inverse short-time Fourier transform to obtain an enhanced speech signal. The method can fully utilize time domain, frequency domain, absolute error, relative error, signal and noise information to update weights, and thus more optimal network parameters and enhancement effects are obtained.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing technology, and in particular to a method for enhancing the voice of VHF communication audio signals on ships. Background Technology

[0002] Ship navigation data recorders (VDRs) record information such as a ship's position, operational status, equipment physical condition, commands, and maneuvers in real time in a safe and recoverable manner. Among these, the VHF communication audio information recorded by the VDR plays an irreplaceable role in ship dynamic monitoring, accident analysis, and maritime liability determination. However, real VHF communication audio signals are subject to various complex noise interferences, including environmental noise and communication noise, often masking the speech signal and resulting in a low signal-to-noise ratio and poor quality. To address these issues, this invention proposes a speech enhancement method for ship VHF communication audio signals based on complex convolutional attention networks. This method enhances the speech in ship VHF communication audio signals, improving the clarity and intelligibility of the speech.

[0003] Existing speech enhancement methods can be broadly categorized into two types: traditional methods that require prior estimation of noise and speech signals, and data-driven deep learning methods. Typical traditional methods include spectral subtraction, subspace-based methods, and Kalman filtering, which are relatively mature. These methods have achieved good results with stationary noise and non-stationary noise that satisfies prior assumptions. However, their performance significantly declines under conditions such as non-stationary noise that does not meet prior assumptions, unseen noise, and low signal-to-noise ratio. The fundamental reason is that traditional methods rely too heavily on presuppositions. If the noise assumption is underestimated, noise residue will appear, resulting in spikes in the spectrum and artificial noise. Overestimation of noise will destroy the original speech information, causing speech distortion. Both of these situations are detrimental to restoring clean speech, thus limiting the performance of traditional methods. Data-driven deep learning methods include time-domain waveform mapping, spectral mapping, and mask generation methods. Temporal waveform mapping involves inputting an audio waveform into a model, which then directly outputs a clean speech waveform. Spectral mapping transforms the temporal input to the frequency domain and uses spectral mapping for speech enhancement. Mask generation methods output a mask value, and enhanced speech is derived based on this mask value and the spectrum. Deep learning methods do not require prior assumptions and are highly robust to complex noise, thus gradually becoming the mainstream approach. However, existing deep learning methods still exhibit some deviation in their ability to approximate real speech with enhanced speech. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention discloses a method for enhancing the voice of VHF communication audio signals on ships, specifically including the following steps:

[0005] The real and imaginary components of the audio signal are extracted by performing a short-time Fourier transform.

[0006] The real and imaginary components are input into the encoder to obtain the encoded output feature F. K The encoder includes a complex encoding module, which includes a complex convolutional layer, complex batch normalization, and a PReLU activation function.

[0007] Encode the output feature F K The input is fed into the complex Conformer module to obtain global features H, where the complex Conformer module includes a feedforward layer, a multi-head self-attention layer, and a convolutional layer;

[0008] The global feature H is reshaped to obtain the reshaped feature H', and the reshaped feature H' and the encoded output feature F are obtained. K The concatenation is used as the input information for the decoder to obtain the complex scale mask G. K The decoder includes six decoding modules, each of which includes a complex convolutional layer, a complex batch normalization layer, and a PReLU activation function.

[0009] The complex ratio masking G K Complex scaling masking for VHF communication audio signals, based on complex scaling masking G K The enhanced speech signal complex spectrum is obtained, and the enhanced speech signal is obtained by performing a short-time inverse Fourier transform on the enhanced speech signal complex spectrum.

[0010] If there are 6 complex number encoding modules, the encoder output is represented as F. 6 F 0 =X, first output feature F from (k-1) complex number encoding modules. k-1 The input is fed into the complex convolutional layer of the k-th encoding module, and then subjected to two-dimensional convolution to obtain A. k A k Specifically, it is expressed as follows:

[0011]

[0012] in, and W i k Let these represent the real and imaginary parts of the complex convolution kernel, respectively. and F i k-1 They are F k-1 The real and imaginary parts of A k The input is normalized by the complex number encoding module, and the PReLU activation function is used to obtain the output feature F of the k-th encoding module. k ,Right now

[0013] F k =PReLU(Batchnorm2d(A k ))

[0014] Here, PReLU(·) and Batchnorm2d(·) represent the PReLU function and batch normalization, respectively.

[0015] The number of complex convolutional layers, complex batch normalization, and PReLU activation functions are the same in both the complex encoding module and the decoding module.

[0016] Let D k and Z k Let D be the output of the k-th convolutional layer of the decoding module and the complex convolution filter, respectively. Then the convolution output D... k for

[0017]

[0018] in and These are the real and imaginary parts of the output of the (k-1)th decoding module, respectively; and These are the real and imaginary parts of the k-th complex convolution filter, respectively.

[0019] The output of the k-th decoding module is

[0020] G k =PReLU(Batchnorm2d(D k ))

[0021] The last decoding module outputs G K Complex ratio masking for VHF communication audio signals.

[0022] Masking G according to complex proportions K Obtain the enhanced complex spectrum of the speech signal The specific method used is as follows:

[0023]

[0024] in and They are G K The real and imaginary parts of Y; r and Y i These are the real and imaginary parts of the short-time Fourier transform of the ship's VHF communication audio signal;

[0025] right The enhanced audio signal time-domain waveform is obtained by performing a short-time inverse Fourier transform.

[0026] By adopting the above technical solutions, this invention proposes a speech enhancement method for ship VHF communication audio signals based on complex convolutional attention networks. This method considers the influence of complex signal information on the construction of the enhanced speech signal, and utilizes complex convolutional networks and complex Conformer modules to extract and model local and temporal features of the signal. An attention mechanism is introduced in the complex Conformer stage, employing a non-recursive approach to model temporal information, enabling parallel computation and improving processing speed. Furthermore, the proposed time-frequency domain combined loss function fully utilizes information from the time domain, frequency domain, absolute error, relative error, signal, and noise for weight updates, resulting in superior network parameters and enhancement effects. Experimental results show that the enhancement effect of this invention on speech signals is significantly higher than other methods, fully demonstrating the effectiveness of this invention. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of the method of the present invention;

[0029] Figure 2 This is a time-domain waveform diagram of the ship's VHF communication audio signal in this invention;

[0030] Figure 3 This is a spectrogram of the ship's VHF communication audio signal in this invention;

[0031] Figure 4 This is the spectrogram of the enhanced audio signal in this invention. Detailed Implementation

[0032] To make the technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings:

[0033] Considering the poor speech quality and low signal-to-noise ratio of speech segments in ship VHF communication audio signals under complex noise interference, resulting in low clarity and intelligibility, this invention proposes a speech enhancement method for ship VHF communication audio signals based on a complex convolutional attention network. Specifically, the proposed speech enhancement method for ship VHF communication audio signals is as follows: Figure 1As shown, the real and imaginary components of the audio signal are first obtained using STFT. Then, the real and imaginary components are input into the encoder for speech feature extraction and compression to obtain abstract, compact features of the audio signal. These features are then input into a complex Conformer module to model the temporal information of the audio signal. Finally, the output of the complex Conformer module is loaded into the decoder to obtain a complex proportional mask. The real and imaginary components of the enhanced audio signal are then calculated to obtain the enhanced audio signal. The specific steps include:

[0034] S1: Downsample the audio signal to 16kHz. Set the window length and frame shift to 25ms and 6.25ms respectively, use a Hamming window as the window function, perform a short-time Fourier transform (STFT), set the number of Fourier transform points to 1024, and extract the real and imaginary components.

[0035] S2: The real and imaginary components extracted in S1 are simultaneously input into the encoder, which consists of six complex coding modules. Each coding module comprises a complex convolutional layer, complex batch normalization, and a PReLU activation function. The encoder output is represented as F. K F 0 =X. Specifically, first, the output features F of (k-1) encoding modules are... k-1 The input is fed into the complex convolutional layer of the k-th encoding module, and then subjected to two-dimensional convolution to obtain A. k A k Specifically, it is expressed as follows:

[0036]

[0037] in, and W i k Let these represent the real and imaginary parts of the complex convolution kernel, respectively. and F i k-1 They are F k-1 The real and imaginary parts of A. Then, A... k The batch normalization input to the encoding module is processed. Finally, the PReLU activation function is used to obtain the output feature F of the k-th encoding module. k ,Right now

[0038] F k =PReLU(Batchnorm2d(A k (2)

[0039] Here, PReLU(·) and Batchnorm2d(·) represent the PReLU function and batch normalization, respectively.

[0040] S3: Convert the final output F from S2 KThe input is fed into the complex Conformer module, which consists of a feedforward layer, a multi-head self-attention layer, and a convolutional layer. Let... and F i K Let H be the real and imaginary parts output by the last encoding module, respectively. Then, the global feature H output by the complex Conformer module can be obtained by the following four formulas:

[0041]

[0042] C2=C1+MHSA(C1) (4)

[0043] C3 = C2 + Conv(C2) (5)

[0044]

[0045] in _ represents concatenation; FFN(·), MHSA(·), and Conv(·) represent forward, multi-head self-attention, and convolutional layers, respectively; Layernorm(·) is the layer normalization operation.

[0046] S4: Reshape the global feature H obtained in S3 to obtain H′, and combine it with the final output F of the coding module network in S2. K The two are concatenated and used as the input to the decoder. The decoder consists of 6 decoding modules. Each decoding module includes complex convolution, complex batch normalization, and PReLU activation functions. The structure of the decoding module is the same as that of the encoding module, and the number of complex convolutional layers, complex batch normalization, and PReLU activation functions in the decoding module is the same as that in the encoding module.

[0047] Let D k and Z k These are the output of the k-th convolutional layer of the decoding module and the complex convolution filter, respectively. Then the convolution output D... k for

[0048]

[0049] in and These are the real and imaginary parts of the output of the (k-1)th decoding module, respectively; and These are the real and imaginary parts of the k-th complex convolution filter, respectively.

[0050] The output of the k-th decoding module is

[0051] G k =PReLU(Batchnorm2d(D k (8)

[0052] S5: The final output G of the decoder K This involves complex scaling masking of the audio signal, resulting in the enhanced complex spectrum of the speech signal:

[0053]

[0054] in and They are G K The real and imaginary parts of Y; r and Y i These are the real and imaginary parts of the short-time Fourier transform of the ship's VHF communication audio signal.

[0055] Complex spectrum of enhanced speech signal The enhanced audio signal time-domain waveform is obtained by performing a short-time inverse Fourier transform.

[0056] The loss function used in training this model is a novel approach proposed in this invention, L. CD It also incorporates the absolute loss function L log-PCM and relative loss function L SI-SNR The advantages include not only the absolute loss of frequency domain short-time Fourier transform amplitude of speech and noise, but also the relative loss in the time domain. Therefore, L CD This allows for better optimization of the mask mapping target and the acquisition of superior weight parameters. (The last part, "L," appears to be an unrelated fragment and is left untranslated.) log-PCM and L SI-SNR The two loss functions are weighted to obtain L. CD ,Right now

[0057] L CD =αL SI-SNR +(1-α)L log-PCM (10)

[0058] Here, α is a hyperparameter, which is set to 0.2.

[0059] L in (10) SI-SNR Defined as

[0060]

[0061] Where s, s mix These are, respectively, the actual speech signal, the estimated speech signal, and the mixed audio signal.

[0062]

[0063]

[0064] In formula (10) L log-PCM Defined as

[0065]

[0066] Where s, n, These are, respectively, the real speech signal, speech signal estimation, real noise, and noise estimation.

[0067]

[0068] Among them, S r (t,f), S i (t,f) These are the real part of the STFT of the real speech signal, the imaginary part of the STFT of the real speech signal, the real part of the STFT of the estimated speech signal, and the imaginary part of the STFT of the estimated speech signal, respectively. Similarly, we define...

[0069]

[0070] Where, N r (t,f), N i (t,f) These are the real part of the STFT of the real noise signal, the imaginary part of the STFT of the real noise signal, the real part of the STFT of the noise signal estimation, and the imaginary part of the STFT of the noise signal estimation, respectively.

[0071] The learning rate was set to 0.001 during the experiment, and the Adam optimizer was used. The number of iterations was set to 30, and the batch size was 1.

[0072] This invention proposes a novel cross-domain loss function that takes into account both time-domain and frequency-domain errors when training network parameters. This loss function also incorporates absolute and relative errors, as well as speech and noise errors, to achieve higher performance in speech enhancement.

[0073] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for enhancing the voice of a ship's VHF communication audio signal, characterized in that... include: The real and imaginary components of the audio signal are extracted by performing a short-time Fourier transform. inputting the real and imaginary parts into an encoder to obtain an encoded output feature F K , the encoder comprising a complex encoding module, the complex encoding module comprising a complex convolution layer, a complex batch normalization, and a PReLU activation function; Encode the output feature F K The input is fed into the complex Conformer module to obtain global features H, where the complex Conformer module includes a feedforward layer, a multi-head self-attention layer, and a convolutional layer; The global feature H is reshaped to obtain the reshaped feature H', and the reshaped feature H' and the encoded output feature F are obtained. K The concatenation is used as the input information for the decoder to obtain the complex scale mask G. K The decoder includes six decoding modules, each of which includes a complex convolutional layer, a complex batch normalization layer, and a PReLU activation function. The complex ratio masking G K Complex scaling masking for VHF communication audio signals, based on complex scaling masking G K The enhanced speech signal complex spectrum is obtained, and the enhanced speech signal is obtained by performing a short-time inverse Fourier transform on the enhanced speech signal complex spectrum.

2. The voice enhancement method for ship VHF communication audio signals according to claim 1, characterized in that: If there are 6 complex number encoding modules, the encoder output is represented as F. 6 F 0 =X, first output feature F from (k-1) complex number encoding modules. k-1 The input is fed into the complex convolutional layer of the k-th encoding module, and then subjected to two-dimensional convolution to obtain A. k A k Specifically, it is expressed as follows: in, and W i k Let these represent the real and imaginary parts of the complex convolution kernel, respectively. and F i k-1 They are F k-1 The real and imaginary parts of A k The input is normalized by the complex number encoding module, and the PReLU activation function is used to obtain the output feature F of the k-th encoding module. k ,Right now F k =PReLU(Batchnorm2d(A k )) Here, PReLU(·) and Batchnorm2d(·) represent the PReLU function and batch normalization, respectively.

3. The voice enhancement method for ship VHF communication audio signals according to claim 1, characterized in that: The number of complex convolutional layers, complex batch normalization, and PReLU activation functions are the same in both the complex encoding module and the decoding module. Let D k and Z k Let D be the output of the k-th convolutional layer of the decoding module and the complex convolution filter, respectively. Then the convolution output D... k for in and These are the real and imaginary parts of the output of the (k-1)th decoding module, respectively; and These are the real and imaginary parts of the k-th complex convolution filter, respectively. The output of the k-th decoding module is G k =PReLU(Batchnorm2d(D k )) The last decoding module outputs G K Complex ratio masking for VHF communication audio signals.

4. The voice enhancement method for ship VHF communication audio signals according to claim 1, characterized in that: Masking G according to complex proportions K Obtain the enhanced complex spectrum of the speech signal The specific method used is as follows: in and They are G K The real and imaginary parts of Y; r and Y i These are the real and imaginary parts of the short-time Fourier transform of the ship's VHF communication audio signal; right The enhanced audio signal time-domain waveform is obtained by performing a short-time inverse Fourier transform.

Images