Unmanned aerial vehicle speech enhancement method and system based on harmonic characteristics and deep learning

Abstract

The application discloses a UAV speech enhancement method and system based on harmonic characteristics and deep learning, and the method comprises the following steps: collecting rotor self-noise signals of a UAV, i.e., original sampling frequencies; processing the original sampling frequencies to obtain feature vectors; processing the feature vectors to generate noise dynamic change labels; combining the feature vectors and the noise dynamic change labels into a data set; constructing a deep learning model; inputting the data set into the deep learning model for processing; outputting a prediction result of the noise dynamic change labels; extracting the collected data to obtain a noisy speech amplitude spectrum; performing smoothing processing on the noisy speech amplitude spectrum to obtain an enhanced signal; and outputting the enhanced signal. The application can more accurately separate noise and speech by extracting the rotor self-noise signals of the UAV and combining the deep learning model to predict the noise change trend, so that the speech signal-to-noise ratio is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of speech signal processing, and particularly relates to a UAV speech enhancement method and system based on harmonic characteristics and deep learning. BACKGROUND

[0002] In recent years, UAVs are equipped with cameras to provide users with a wide range of applications in visual applications, but in some special scenarios, visual processing cannot meet the needs of users, such as rescue activities in bad weather. Due to weather reasons, the camera carried by the UAV cannot clearly capture the rescue situation, so a microphone array can be carried on the UAV, so that the UAV has hearing in addition to vision to solve some situations that cannot be met by visual processing.

[0003] At present, the methods used have limitations, such as DJI (DJI), Parrot, which reduce noise from the source by improving the design of the rotor (such as shape, material) and the structure of the motor, and the noise elimination effect is limited, iZotope, Adobe, and other audio software processing companies provide software-based noise elimination methods, which have limited support for real-time processing, MIT, Stanford, and other noise processing based on traditional signal processing techniques have limited effect on non-linear noise, and with the development of deep learning, there are many deep learning-based speech enhancement systems, which basically process static noise, but since the self-noise of the UAV is dynamic during flight, there is no deep learning speech enhancement system for the characteristics of the rotor UAV self-noise.

[0004] The present application processes the harmonic characteristics of the self-noise of the UAV to guide deep learning and construct a deep learning model to dynamically analyze the periodic harmonic characteristics of the self-noise of the UAV, dynamically adjust the parameters according to the analysis results, and realize deep learning speech enhancement based on the characteristics of the rotor UAV self-noise. SUMMARY

[0005] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present application provide a UAV speech enhancement method and system based on harmonic characteristics and deep learning, which solves the problems mentioned in the background art.

[0006] To achieve the above purpose, the present application provides the following technical scheme: a UAV speech enhancement method based on periodic harmonic characteristics and deep learning prediction, comprising the following steps:

[0007] Step S1: Collecting the rotor self-noise signal of the UAV, i.e. the original sampling frequency;

[0008] Step S2: processing the original sampling frequency to obtain a new sampling frequency, filtering the new sampling frequency to obtain a normalized signal, extracting the normalized signal to obtain a normalized signal spectrum, using a peak detection algorithm to find the peak value in the normalized signal spectrum, taking the peak value as the fundamental frequency, extracting the normalized signal spectrum to obtain the harmonic component, arranging the fundamental frequency and the harmonic component in order to form a feature vector;

[0009] Step S3: processing the feature vector to generate a noise dynamic change label, combining the feature vector and the noise dynamic change label into a data set, constructing a deep learning model, inputting the data set into the deep learning model for processing, and outputting a prediction result of the noise dynamic change label;

[0010] Step S4: extracting the rotor self-noise signal of the unmanned aerial vehicle to obtain a noisy speech amplitude spectrum, using the minimum statistical method to calculate the energy distribution of the rotor self-noise signal of the unmanned aerial vehicle according to the prediction result of the noise dynamic change label, smoothing the noisy speech amplitude spectrum to obtain a smoothed spectrum, processing the smoothed spectrum to obtain the spectral characteristics of the noise, subtracting the spectral characteristics of the noise from the noisy speech amplitude spectrum to obtain an enhanced amplitude spectrum, processing the noisy speech amplitude spectrum to obtain phase information, combining the enhanced amplitude spectrum and the phase information to construct an enhanced spectrum, processing the enhanced spectrum to obtain an enhanced signal;

[0011] Step S5: outputting the enhanced signal.

[0012] Further, in step S1, the rotor self-noise signal of the unmanned aerial vehicle is collected, and the specific process is as follows:

[0013] The high-sensitivity microphone array is mounted on the unmanned aerial vehicle, and the high-sensitivity microphone array is connected to a high-performance digital-to-analog converter. The rotor self-noise signal of the unmanned aerial vehicle is collected through the high-sensitivity microphone array. The rotor self-noise signal of the unmanned aerial vehicle is an analog signal. The analog signal is converted into a digital signal, which is the original sampling frequency.

[0014] Further, in step S2, the feature vector is processed, and the specific process is as follows:

[0015] The original sampling frequency is down-sampled to obtain a new sampling frequency. Before down-sampling, a low-pass filter is used to remove frequency components higher than half of the new sampling frequency. The minimum integer of the original sampling frequency and the new sampling frequency is taken. The down-sampling factor M is set to 3, which means:

[0016] M = foriginal / fnew = 3 (1);

[0017] In the formula, f original represents the original sampling frequency; f new represents the new sampling frequency; the high-frequency noise and low-frequency interference in the new sampling frequency are filtered using a Butterworth filter to obtain a filtered signal, the order n of the Butterworth filter is 4, and the formula is:

[0018]

[0019] In the formula, f represents frequency; H(f) represents the frequency response function of the Butterworth filter; fc represents the cutoff frequency of the Butterworth filter; the maximum absolute value |Xmax| of the filtered signal is calculated, and all filtered signals are divided by the maximum absolute value |Xmax| of the filtered signal to achieve signal amplitude standardization in a fixed range [-1, 1], and the formula is:

[0020] Xnormalized=X / |Xmax|(3);

[0021] In the formula, Xnormalized represents a normalized signal, X represents the rotor self-noise signal of the unmanned aerial vehicle; and Xmax represents the maximum absolute value of the signal.

[0022] The short-time Fourier transform is performed on each frame of the normalized signal to obtain a normalized signal spectrum, and the formula is:

[0023]

[0024] In the formula, X(f, t) represents the transform result of the normalized signal at time t and frequency f; x(n') represents the nth sample of the rotor self-noise signal; w(n'-t) represents the value of the window function at time index n' and window start position t; e -j2Πfn / 320 represents a complex exponential function; j represents an imaginary unit; and e represents the base number of a natural logarithm.

[0025] The peak value detection algorithm is used to find the peak value in the normalized signal spectrum, and the peak value is taken as the fundamental frequency ffundamental.

[0026] The integer multiple frequency position of the fundamental frequency is found in the normalized signal spectrum, and the harmonic component Eharmonic,i of the fundamental frequency is extracted, and the formula is:

[0027] Eharmonic,i=|S(i*ffundamental,t)|i=2,3,…,10(5);

[0028] In the formula, Eharmonic,i represents the ith harmonic component; S represents the spectrum function; and ffundamental,t represents the fundamental frequency of the current frame t.

[0029] The fundamental frequency component and the harmonic components are arranged in order to form a feature vector, and the feature vector is represented as F = [ffundamental,t, Eharmonic,2, Eharmonic,3, …, Eharmonic,10]; Eharmonic,2 represents the second harmonic component; Eharmonic,3 represents the third harmonic component; and Eharmonic,10 represents the tenth harmonic component.

[0030] Further, the prediction result of the noise dynamic change label in step S3 is as follows:

[0031] The feature vector F is taken as input data, and the input data is processed to calculate the change value of the fundamental frequency and the harmonic components of the previous frame in the input data, and a noise dynamic change label Y = [Δffundamental, ΔEharmonic,2, ΔEharmonic,3, …, ΔEharmonic,10] is generated. ΔEharmonic,10 represents the dynamic change value of the tenth harmonic component, and is represented as:

[0032] Δffundamental = ffundamental,t - ffundamental,t-1(6);

[0033] ΔEharmonic,i = Eharmonic,i,t - Eharmonic,i,t-1(7);

[0034] In the formula, Δffundamental represents the difference between the current frame fundamental frequency and the previous frame fundamental frequency; ffundamental,t-1 represents the fundamental frequency of the previous frame t-1; ΔEharmonic,i represents the energy of the current frame i-th harmonic component minus the energy of the previous frame i-th harmonic component; Eharmonic,i,t represents the i-th harmonic component in the previous frame t; and Eharmonic,i,t-1 represents the i-th harmonic component in the previous frame t-1.

[0035] The feature vector F and the noise dynamic change label Y are combined into a data set;

[0036] A deep learning model is constructed, the data set is input into the deep learning model for processing, and a prediction result Ypred = [Δffundamental, ΔEharmonic,2, ΔEharmonic,3, …, ΔEharmonic,10] of the noise dynamic change label is output.

[0037] The mean square error is used as a loss function, and the error between the prediction result Ypred of the noise dynamic change label and the noise dynamic change label Y is calculated, and is represented as:

[0038]

[0039] In the formula, MSE represents mean square error; N represents total number of noise dynamic change labels; Ypred,i represents the prediction result of the ith noise dynamic change label; Yi represents the ith noise dynamic change label; and n' represents sample serial number.

[0040] Further, the enhanced signal in step S4 has the following specific process:

[0041] Periodic harmonic characteristic extraction is performed on the rotor self-noise signal of the unmanned aerial vehicle to obtain a noisy speech amplitude spectrum |Xm(f)|.

[0042] According to the prediction result Ypred of the noise dynamic change label, the energy distribution of the rotor self-noise signal of the unmanned aerial vehicle is calculated using the minimum statistical method, the noisy speech amplitude spectrum |Xm(f)| is subjected to time smoothing processing to obtain a smoothed frequency spectrum |X~m(f)|, and the noise spectrum estimation is updated by using the recursive average method, which is represented as:

[0043] |Nm(f)| = α·|Nm-1(f)| + (1-α)·|X~m(f)| (9);

[0044] In the formula, |Nm(f)| represents the frequency spectrum characteristic of the noise; α represents an over-reduction factor; and |Nm-1(f)| represents the amplitude of the smoothed frequency spectrum of the previous frame.

[0045] The noisy speech amplitude spectrum |Xm(f)| is subtracted from the frequency spectrum characteristic |Nm(f)| of the noise using a speech enhancement algorithm to obtain an enhanced amplitude spectrum, which is represented as:

[0046]

[0047] In the formula, |Sm(f)| represents the enhanced amplitude spectrum; and max represents the maximum value.

[0048] The over-reduction factor α is used to control the strength of noise suppression of the noisy speech amplitude spectrum |Xm(f)|, the max(…, 0) function is used to determine that the noisy speech amplitude spectrum |Xm(f)| is a non-negative value, and the noisy speech amplitude spectrum |Xm(f)| is then processed to retain the phase information φm(f) of the noise, which is represented as:

[0049] φm(f) = ∠Xm(f) (11);

[0050] The enhanced amplitude spectrum |Sm(f)| and the phase information φm(f) are combined to obtain an enhanced frequency spectrum Gm(f), which is represented as:

[0051] Gm(f) = |Sm(f)|·φm(f)e jφm(f) (12);

[0052] where e jφm(f) denotes a phase factor of complex number;

[0053] Based on the Δffundamental and the ΔEharmonic,i, the over-reduction factor α of the speech enhancement algorithm is adjusted dynamically. When the values of the Δffundamental and the ΔEharmonic,i increase, the over-reduction factor α is increased. The enhanced parameter after the dynamic parameter adjustment is output. The enhanced spectrum Gm(f) is reconstructed in time domain by using the enhanced parameter. The time domain signal s m (n) is obtained by performing inverse Fourier transform on the enhanced spectrum Gm(f).

[0054] sm(n) = IFFT(Gm(f)) (13);

[0055] where IFFT denotes inverse Fourier transform;

[0056] The overlap-add method is used again. The time domain signal s m (n) of each frame is added to the overlapping part of the time domain signal s m (n) of the previous frame, and the enhanced signal s(n) is finally output.

[0057] Further, an unmanned aerial vehicle speech enhancement system based on periodic harmonic characteristics and deep learning prediction is used for the unmanned aerial vehicle speech enhancement method based on periodic harmonic characteristics and deep learning prediction, and includes:

[0058] A self-noise acquisition module is configured to acquire a rotor self-noise signal of an unmanned aerial vehicle, i.e., an original sampling frequency.

[0059] A periodic harmonic characteristic extraction module is configured to process the original sampling frequency to obtain a new sampling frequency, filter the new sampling frequency to obtain a normalized signal, extract the normalized signal to obtain a normalized signal spectrum, use a peak detection algorithm to find peaks in the normalized signal spectrum, take the peaks as fundamental frequencies, extract the normalized signal spectrum to obtain harmonic components, arrange the fundamental frequencies and the harmonic components in order to form a feature vector.

[0060] A deep learning model prediction noise dynamic change module is configured to process the feature vector to generate a noise dynamic change label, combine the feature vector and the noise dynamic change label into a data set, construct a deep learning model, input the data set into the deep learning model for processing, and output a prediction result of the noise dynamic change label.

[0061] The voice enhancement module is used for extracting the rotor self-noise signal of the unmanned aerial vehicle, obtaining a noisy voice amplitude spectrum, calculating the energy distribution of the rotor self-noise signal of the unmanned aerial vehicle by using a minimum statistical method according to the prediction result of the noise dynamic change label, performing smoothing processing on the noisy voice amplitude spectrum to obtain a smoothed spectrum, processing the smoothed spectrum to obtain the spectral characteristics of the noise, obtaining an enhanced amplitude spectrum by subtracting the spectral characteristics of the noise from the noisy voice amplitude spectrum, processing the noisy voice amplitude spectrum to obtain phase information, combining the enhanced amplitude spectrum and the phase information to construct an enhanced spectrum, processing the enhanced spectrum to obtain an enhanced signal;

[0062] The output module is used for outputting the enhanced signal.

[0063] Compared with the prior art, the present application has the following advantages:

[0064] (1) The present application can more accurately separate noise and voice by extracting the periodic harmonic characteristics of the rotor self-noise signal of the unmanned aerial vehicle and combining the deep learning model to predict the noise change trend, thereby greatly improving the voice signal-to-noise ratio;

[0065] Compared with the traditional static noise reduction method (such as spectral subtraction or fixed filter), the dynamic parameter adjustment of the present application can adapt to the complex noise environment such as rotor speed change and air flow disturbance during the flight of the unmanned aerial vehicle, so that the enhanced voice is clearer and more intelligible.

[0066] (2) The present application can reduce voice distortion by dynamically adjusting the change of the self-noise of the unmanned aerial vehicle and adjusting the voice enhancement parameters in real time;

[0067] The traditional noise reduction method (such as Wiener filter with fixed parameters or noise threshold) is prone to cause excessive suppression of voice or residual noise when the noise of the unmanned aerial vehicle dynamically changes;

[0068] The present application can avoid voice distortion by using a CNN-RNN hybrid model to predict noise changes (such as fundamental frequency and harmonic energy fluctuation) in real time and dynamically adjusting the reduction factor and filter parameters;

[0069] Experiments show that when the rotor speed suddenly changes (such as acceleration or deceleration), the voice distortion degree (PESQ score) of the present method is about 20% higher than that of the traditional method. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 The present application is a method flowchart. DETAILED DESCRIPTION

[0071] As Figure 1As shown, the present application provides technical solutions: a UAV speech enhancement method based on periodic harmonic feature and deep learning prediction, comprising the following steps:

[0072] Step S1: collect the rotor self-noise signal of the UAV, that is, the original sampling frequency;

[0073] Step S2: process the original sampling frequency to obtain a new sampling frequency, filter the new sampling frequency to obtain a normalized signal, extract the normalized signal to obtain a normalized signal spectrum, use a peak detection algorithm to find the peak value in the normalized signal spectrum, take the peak value as the fundamental frequency, extract the normalized signal spectrum to obtain harmonic components, arrange the fundamental frequency and the harmonic components in order to form a feature vector;

[0074] Step S3: process the feature vector to generate a noise dynamic change label, combine the feature vector and the noise dynamic change label into a data set, construct a deep learning model, input the data set into the deep learning model for processing, and output the prediction result of the noise dynamic change label;

[0075] Step S4: extract the rotor self-noise signal of the UAV to obtain a noisy speech amplitude spectrum, use the minimum statistical method to calculate the energy distribution of the rotor self-noise signal of the UAV according to the prediction result of the noise dynamic change label, smooth the noisy speech amplitude spectrum to obtain a smoothed spectrum, process the smoothed spectrum to obtain the spectral characteristics of the noise, subtract the spectral characteristics of the noise from the noisy speech amplitude spectrum to obtain an enhanced amplitude spectrum, process the noisy speech amplitude spectrum to obtain phase information, combine the enhanced amplitude spectrum and the phase information to construct an enhanced spectrum, process the enhanced spectrum to obtain an enhanced signal;

[0076] Step S5: output the enhanced signal.

[0077] In step S1, the rotor self-noise signal of the UAV is collected, and the specific process is as follows:

[0078] The high-sensitivity microphone array is mounted on the unmanned aerial vehicle, and the high-sensitivity microphone array is connected to a 24-bit high-performance digital-to-analog converter (ADC). The rotor self-noise signal of the unmanned aerial vehicle is collected by the high-sensitivity microphone array. The rotor self-noise signal of the unmanned aerial vehicle is an analog signal. The analog signal is converted into a digital signal, that is, the original sampling frequency. The main frequency range of the rotor self-noise signal of the unmanned aerial vehicle is usually several tens of Hz to several thousand Hz. According to the Nyquist sampling theorem, the sampling frequency of the high-sensitivity microphone array should be at least 2 times the highest frequency of the original sampling frequency, fsample≥2×fmax. The sampling rate of the high-performance digital-to-analog converter is set to 48 kHz to ensure the fidelity of the original sampling frequency; fsample represents the sampling frequency; and fmax represents the highest frequency.

[0079] In step S2, the feature vector is specifically processed as follows:

[0080] In order to reduce the data amount and improve the calculation efficiency, the original sampling frequency is reduced from 48 kHz to 16 kHz, and the new sampling frequency is 16 kHz, which meets the basic demand of voice communication. Before reducing the sampling frequency, a low-pass filter is used to remove frequency components higher than half of the new sampling frequency, so as to prevent aliasing and remove frequency components higher than 8 kHz. The minimum integer of the original sampling frequency and the new sampling frequency is taken, and the down-sampling factor M is set to 3, that is:

[0081] M=foriginal / fnew=3(1);

[0082] In the formula, foriginal represents the original sampling frequency (48 kHz); and fnew represents the new sampling frequency (16 kHz).

[0083] A Butterworth filter is used to filter high-frequency noise and low-frequency interference in the new sampling frequency, to obtain a filtered signal, and to retain the main frequency range of the rotor self-noise signal of the unmanned aerial vehicle: the main frequency range is 50 Hz-5 kHz, the order n of the Butterworth filter is 4, that is:

[0084]

[0085] In the formula, f represents the frequency; H(f) represents the frequency response function of the Butterworth filter; fc represents the cut-off frequency 5000 Hz of the Butterworth filter; the maximum absolute value |Xmax| of the filtered signal is calculated, all filtered signals are divided by the maximum absolute value |Xmax| of the filtered signal to achieve signal amplitude standardization in a fixed range [-1, 1], that is:

[0086] Xnormalized=X / |Xmax|(3);

[0087] In the formula, Xnormalized represents a normalized signal, X represents a rotor self-noise signal of the unmanned aerial vehicle; and Xmax represents a maximum absolute value of the signal.

[0088] The normalized signal is divided into short-time frames, facilitating subsequent spectral analysis. Each frame has a length of 20 ms, and when the sampling rate is 16 kHz, the frame length of 20 ms is 0.02*16000 = 320 samples.

[0089] The normalized signal is subjected to periodic harmonic characteristic extraction. First, short-time Fourier transform is performed on each frame of the normalized signal to obtain a normalized signal spectrum. The time-domain signal of the normalized signal spectrum is converted into a frequency-domain signal, facilitating analysis of the frequency-domain signal components. According to the pre-processing, each frame of the normalized signal frequency-domain signal has 320 samples, and the window function is a Hamming window, represented as:

[0090]

[0091] In the formula, X(f, t) represents the transform result of the normalized signal at time t and frequency f; x(n') represents the nth sample of the rotor self-noise signal of the unmanned aerial vehicle; w(n'-t) represents the value of the window function at time index n' and window start position t; e -j2Πfn / 320 represents a complex exponential function; j represents an imaginary unit; and e represents the base number of a natural logarithm.

[0092] A peak detection algorithm is used to find the peak value in the normalized signal spectrum, and the peak value is taken as the fundamental frequency ffundamental.

[0093] The integer multiple frequency position of the fundamental frequency is found in the normalized signal spectrum, and the harmonic component Eharmonic,i of the fundamental frequency is extracted, represented as:

[0094] Eharmonic,i = |S(i*ffundamental,t)| i = 2, 3,..., 10 (5).

[0095] In the formula, Eharmonic,i represents the ith harmonic component; S represents a spectrum function; and ffundamental,t represents the fundamental frequency of the current frame t.

[0096] The fundamental frequency and the harmonic components are arranged in order to form a feature vector, represented as F = [ffundamental,t, Eharmonic,2, Eharmonic,3,..., Eharmonic,10]. Eharmonic,2 represents the second harmonic component; Eharmonic,3 represents the third harmonic component; and Eharmonic,10 represents the tenth harmonic component.

[0097] In step S3, the prediction result of the noise dynamic change label is obtained, and the specific process is as follows:

[0098] The feature vector F is taken as input data, the input data is processed, the change value of the fundamental frequency and the harmonic component in the previous frame of the input data is calculated, and the noise dynamic change label Y is generated, Y = [Δffundamental, ΔEharmonic, 2, ΔEharmonic, 3, …, ΔEharmonic, 10], ΔEharmonic, 10 represents the dynamic change value of the tenth harmonic component, and represents:

[0099] Δffundamental = ffundamental, t - ffundamental, t-1 (6);

[0100] ΔEharmonic, i = Eharmonic, i, t - Eharmonic, i, t-1 (7);

[0101] In the formula, Δffundamental represents the difference between the current frame fundamental frequency and the previous frame fundamental frequency value; ffundamental, t-1 represents the fundamental frequency of the previous frame t-1; ΔEharmonic, i represents the energy of the i-th harmonic component in the current frame minus the energy of the i-th harmonic component in the previous frame; Eharmonic, i, t represents the i-th harmonic component in the previous frame t; Eharmonic, i, t-1 represents the i-th harmonic component in the previous frame t-1;

[0102] The feature vector F and the noise dynamic change label Y are combined into a data set;

[0103] A deep learning model is constructed: a deep learning model based on CNN and RNN is constructed by combining the local feature extraction capability of CNN and the time sequence modeling capability of RNN;

[0104] The deep learning model includes an input layer, a CNN layer, a pooling layer, an RNN layer, a full connection layer, and an output layer;

[0105] The input layer is used to receive the time sequence feature vector F, and the input dimension of the input layer is (T, 10), T is the time step, and 10 is the feature vector dimension (fundamental frequency + 9 harmonic components);

[0106] The CNN layer uses a 1D convolution layer (Conv1D) to extract local features of the feature vector F, the convolution kernel size of the CNN layer is set to 3, the step is 1, the filter number is 64, the activation function of the CNN layer uses ReLU, and the output dimension of the CNN layer is (T, 64); Where 64 represents the channel dimension (i.e. the number of feature maps);

[0107] The pooling layer uses a max pooling layer (MaxPooling1D) to down-sample the output of the CNN layer, a pooling window size of 2, and an output dimension of the pooling layer of (T / 2, 64); T / 2 represents a time dimension halved;

[0108] The RNN layer uses an LSTM (Long Short-Term Memory network) to model the time-dependent relationship of the output of the pooling layer, an LSTM unit number of 128, a complete time sequence output returned, and an output dimension of the LSTM of (T / 2, 128); the LSTM processes time dependence and outputs 128 LSTM units;

[0109] The output of the LSTM is mapped to an output space using a fully connected layer (Dense) to obtain a prediction result of the noise dynamic change label, a number of neurons of the fully connected layer is 64, and a ReLU is used as an activation function;

[0110] The output layer is used to output a prediction result Ypred of the noise dynamic change label [Deltaffundamental, DeltaEharmonic, 2, DeltaEharmonic, 3, …, DeltaEharmonic, 10], and an output dimension of the output layer is (T / 2, 10);

[0111] The mean square error (MSE) is used as a loss function to calculate the error between the prediction result Ypred of the noise dynamic change label and the noise dynamic change label Y, and the smaller the MSE value, the better the fitting, which is represented as:

[0112]

[0113] In the formula, MSE represents the mean square error; N represents the total number of noise dynamic change labels; Y pred,i represents the prediction result of the i-th noise dynamic change label; Y i represents the i-th noise dynamic change label; and n' represents a sample serial number.

[0114] In step S4, the enhanced signal is obtained by the following specific process:

[0115] The periodic harmonic characteristic of the rotor self-noise signal of the unmanned aerial vehicle is extracted to obtain a noisy speech amplitude spectrum |Xm(f)|;

[0116] The energy distribution of the rotor self-noise signal of the unmanned aerial vehicle is calculated using the minimum statistical method according to the prediction result Ypred of the noise dynamic change label, the noisy speech amplitude spectrum |Xm(f)| is time-smoothed to obtain a smoothed frequency spectrum |X~m(f)|, and the noise spectrum estimation is updated by the recursive average method, which is represented as:

[0117] | Nm(f) | = a · | Nm-1(f) | + (1 - a) · | X~m(f) | (9)

[0118] In the formula, | Nm(f) | represents the spectral characteristics of the noise; a represents an over-reduction factor, usually set between 0.9-0.99; | Nm-1(f) | represents the amplitude of the smoothed spectrum of the previous frame;

[0119] The noisy speech amplitude spectrum | Xm(f) | is subtracted from the spectral characteristics | Nm(f) | of the noise using the speech enhancement algorithm to obtain the enhanced amplitude spectrum, denoted as:

[0120]

[0121] In the formula, | Sm(f) | represents the enhanced amplitude spectrum; max represents the maximum value;

[0122] The over-reduction factor a is used to control the strength of noise suppression of the noisy speech amplitude spectrum | Xm(f) |, and the max(…, 0) function is used to determine that the noisy speech amplitude spectrum | Xm(f) | is a non-negative value, and then the noisy speech amplitude spectrum | Xm(f) | is processed to retain the phase information φm(f), denoted as:

[0123] φm(f) = ∠Xm(f) (11) ;

[0124] The enhanced amplitude spectrum | Sm(f) | and the phase information φm(f) are combined to obtain the enhanced spectrum Gm(f), denoted as:

[0125] Gm(f) = | Sm(f) | · φm(f) e jφm(f) (12) ;

[0126] In the formula, e jφm(f) represents a phase factor in complex form;

[0127] Based on Δffundamental and ΔEharmonic,i, the over-reduction factor a of the speech enhancement algorithm is adjusted by dynamic parameters. When the values of Δffundamental and ΔEharmonic,i increase, the over-reduction factor a is increased, and the enhanced parameter after dynamic parameter adjustment is output. The enhanced parameter is used to perform time domain reconstruction on the enhanced spectrum Gm(f), and the inverse Fourier transform is performed on the enhanced spectrum Gm(f) to obtain the time domain signal s m (n), denoted as:

[0128] sm(n) = IFFT(Gm(f)) (13) ;

[0129] In the formula, IFFT represents the inverse Fourier transform;

[0130] Reusing overlap-add method, for each frame of time-domain signal s m (n) the previous frame of time-domain signal s m (n) overlap and add, finally output the enhanced signal s(n).

[0131] Among them, a kind of unmanned aerial vehicle speech enhancement system based on periodic harmonic feature and deep learning prediction, for the one based on periodic harmonic feature and deep learning prediction of unmanned aerial vehicle speech enhancement method, it includes:

[0132] Self-noise acquisition module, for collecting the rotor self-noise signal of unmanned aerial vehicle, i.e. original sampling frequency;

[0133] Periodic harmonic feature extraction module, for processing the original sampling frequency, obtaining new sampling frequency, filtering the new sampling frequency, obtaining normalized signal, extracting the normalized signal, obtaining normalized signal spectrum, using peak detection algorithm to find the peak value in the normalized signal spectrum, taking the peak value as the fundamental frequency, extracting the normalized signal spectrum, obtaining harmonic component, arranging the fundamental frequency and harmonic component in order to form feature vector;

[0134] Deep learning model prediction noise dynamic change module, for processing the feature vector, generating noise dynamic change label, combining the feature vector and noise dynamic change label into data set, constructing deep learning model, inputting data set into deep learning model for processing, outputting the prediction result of noise dynamic change label;

[0135] Speech enhancement module, for extracting the rotor self-noise signal of unmanned aerial vehicle, obtaining noisy speech amplitude spectrum, calculating the energy distribution of the rotor self-noise signal of unmanned aerial vehicle using minimum statistical method according to the prediction result of noise dynamic change label, smoothing the noisy speech amplitude spectrum to obtain smoothed spectrum, processing the smoothed spectrum to obtain the spectral characteristics of noise, subtracting the spectral characteristics of noise from the noisy speech amplitude spectrum to obtain enhanced amplitude spectrum, processing the noisy speech amplitude spectrum to obtain phase information, combining the enhanced amplitude spectrum and phase information to construct enhanced spectrum, processing the enhanced spectrum to obtain enhanced signal;

[0136] Output module, for outputting the enhanced signal.

[0137] Although the embodiments of the present application have been shown and described, it can be understood by those skilled in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for drone speech enhancement based on periodic harmonic features and deep learning prediction, characterized in that, Comprising the following steps: Step S1: Collecting the rotor self-noise signal of the unmanned aerial vehicle, that is, the original sampling frequency; Step S2: Processing the original sampling frequency to obtain a new sampling frequency, filtering the new sampling frequency to obtain a normalized signal, extracting the normalized signal to obtain a normalized signal spectrum, using a peak detection algorithm to find the peak value in the normalized signal spectrum, taking the peak value as the fundamental frequency, extracting the normalized signal spectrum to obtain harmonic components, arranging the fundamental frequency and the harmonic components in order to form a feature vector; Step S3: Processing the feature vector to generate a noise dynamic change label, combining the feature vector and the noise dynamic change label into a data set, constructing a deep learning model, inputting the data set into the deep learning model for processing, and outputting the prediction result of the noise dynamic change label; Taking the feature vector F as input data, processing the input data, calculating the change value of the fundamental frequency and the harmonic components in the previous frame in the input data, generating a noise dynamic change label Y=[Deltaffundamental, DeltaEharmonic, 2, DeltaEharmonic, 3, …, DeltaEharmonic, 10], DeltaEharmonic, 10 represents the dynamic change value of the tenth harmonic component, which represents: Deltaffundamental=ffundamental, t-ffundamental, t-1; DeltaEharmonic, i=Eharmonic, i, t-Eharmonic, i, t-1; Where, Deltaffundamental represents the difference between the current frame fundamental frequency and the previous frame fundamental frequency; ffundamental, t-1 represents the fundamental frequency of the previous frame t-1; DeltaEharmonic, i represents the energy of the current frame i-th harmonic component minus the energy of the previous frame i-th harmonic component; Eharmonic, i, t represents the i-th harmonic component in the previous frame t; Eharmonic, i, t-1 represents the i-th harmonic component in the previous frame t-1; Step S4: Extracting the rotor self-noise signal of the unmanned aerial vehicle to obtain a noisy speech amplitude spectrum, using the minimum statistical method to calculate the energy distribution of the rotor self-noise signal of the unmanned aerial vehicle according to the prediction result of the noise dynamic change label, smoothing the noisy speech amplitude spectrum to obtain a smoothed spectrum, processing the smoothed spectrum to obtain the spectral characteristics of the noise, subtracting the spectral characteristics of the noise from the noisy speech amplitude spectrum to obtain an enhanced amplitude spectrum, processing the noisy speech amplitude spectrum to obtain phase information, combining the enhanced amplitude spectrum and the phase information to construct an enhanced spectrum, processing the enhanced spectrum to obtain an enhanced signal; Step S5: Outputting the enhanced signal.

2. The drone speech enhancement method based on periodic harmonic features and deep learning prediction according to claim 1, characterized in that: In step S1, the rotor self-noise signal of the unmanned aerial vehicle is collected, and the specific process is as follows: The high-sensitivity microphone array is arranged on the unmanned aerial vehicle, and the high-sensitivity microphone array is connected with a high-performance digital-analog converter; the high-sensitivity microphone array is used to collect the rotor self-noise signal of the unmanned aerial vehicle; the rotor self-noise signal of the unmanned aerial vehicle is an analog signal; the analog signal is converted into a digital signal, which is an original sampling frequency.

3. The drone speech enhancement method based on periodic harmonic features and deep learning prediction according to claim 1, characterized in that: In step S2, the feature vector is obtained through the following steps: The original sampling frequency is down-sampled to obtain a new sampling frequency; before down-sampling, a low-pass filter is used to remove frequency components higher than half of the new sampling frequency; the minimum integer of the original sampling frequency and the new sampling frequency is taken; the down-sampling factor M is set to 3, which means: M=foriginal / fnew=3; In the formula, foriginal represents the original sampling frequency; fnew represents the new sampling frequency; A Butterworth filter is used to filter high-frequency noise and low-frequency interference in the new sampling frequency to obtain a filtered signal; the order n of the Butterworth filter is 4, which means: ； In the formula, denotes the frequency; denotes the frequency response function of a Butterworth filter; denotes the cut-off frequency of a Butterworth filter; The maximum absolute value |Xmax| of the filtered signal is calculated; all filtered signals are divided by the maximum absolute value |Xmax| of the filtered signal to achieve signal amplitude normalization within a fixed range [-1, 1], which means: Xnormalized=X / |Xmax|; In the formula, Xnormalized represents the normalized signal, X represents the rotor self-noise signal of the unmanned aerial vehicle, and Xmax represents the maximum absolute value of the signal. Short-time Fourier transform is performed on each frame of the normalized signal to obtain the normalized signal spectrum, which means: ； wherein represents the result of a transform of the normalized signal at time t and frequency f; represents the i-th sample of the rotor self-noise signal; represents the i-th sample of the rotor self-noise signal; represents the value of the window function at time index i and window start position t; represents the value of the window function at time index i and window start position t; represents the complex exponential function; j represents the imaginary unit; represents the base of the natural logarithm; A peak detection algorithm is used to find the peak value in the normalized signal spectrum, and the peak value is taken as the fundamental frequency ffundamental. The integer multiple frequency position of the fundamental frequency in the normalized signal spectrum is found, and the harmonic component Eharmonic,i of the fundamental frequency is extracted, which means: Eharmonic,i=|S(i*ffundamental,t)|i=2,3,…,10; In the formula, Eharmonic,i represents the i-th harmonic component; S represents the spectrum function; ffundamental,t represents the fundamental frequency of the current frame t. The fundamental frequency and the harmonic components are arranged in order to form a feature vector, which is represented as F=[ffundamental,t,Eharmonic,2,Eharmonic,3,…,Eharmonic,10]; Eharmonic,2 represents the second harmonic component; Eharmonic,3 represents the third harmonic component; Eharmonic,10 represents the tenth harmonic component.

4. The drone speech enhancement method based on periodic harmonic features and deep learning prediction according to claim 3, characterized in that: In step S3, the prediction result of the noise dynamic change label is obtained through the following steps: The feature vector F and the noise dynamic change label Y are combined into a data set; A deep learning model is constructed, and the data set is input into the deep learning model for processing to output the prediction result Ypred of the noise dynamic change label, which is represented as Ypred=[Δffundamental,ΔEharmonic,2,ΔEharmonic,3,…,ΔEharmonic,10]; The error between the prediction result Ypred of the noise dynamic change label and the noise dynamic change label Y is calculated using mean square error as a loss function, and is represented by: 2 ； In the formula, represents the mean square error; represents the total number of noise dynamic change labels; represents the prediction result of the i-th noise dynamic change label; represents the i-th noise dynamic change label; represents the sample serial number.

5. The drone speech enhancement method based on periodic harmonic features and deep learning prediction according to claim 4, characterized in that: The enhanced signal in step S4 is specifically processed as follows: The periodic harmonic characteristic of the rotor self-noise signal of the unmanned aerial vehicle is extracted to obtain a noisy speech amplitude spectrum |Xm(f)|; The energy distribution of the rotor self-noise signal of the unmanned aerial vehicle is calculated using the minimum statistical method according to the prediction result Ypred of the noise dynamic change label, the noisy speech amplitude spectrum |Xm(f)| is time-smoothed to obtain a smoothed frequency spectrum |X~m(f)|, and the noise spectrum estimation is updated through the recursive average method, and is represented by: |Nm(f)| = a * |Nm-1(f)| + (1-a) * |X~m(f)|; In the formula, |Nm(f)| represents the spectral characteristics of the noise, a represents a reduction factor, and |Nm-1(f)| represents the amplitude of the smoothed spectrum of the previous frame. The noisy speech amplitude spectrum |Xm(f)| is subtracted from the spectral characteristics |Nm(f)| of the noise using a speech enhancement algorithm to obtain an enhanced amplitude spectrum, and is represented by: ∣Sm(f)∣= ; In the formula, |Sm(f)| represents the enhanced amplitude spectrum; represents the maximum value; over-reduction factor To control the strength of noise suppression of the noisy speech amplitude spectrum |Xm(f), the max(...,0) function is used to determine the noisy speech amplitude spectrum |Xm(f) to be non-negative, and then the noisy speech amplitude spectrum |Xm(f) is processed, and the phase information of the noise φm(f) is retained, which is represented as: phi m (f) = angle (Xm (f)) ; The enhanced amplitude spectrum |Sm(f)| and the phase information phi m (f) are combined to obtain an enhanced spectrum Gm(f), and are represented by: Gm(f) = |Sm(f) | · φm(f)e jϕm(f) ; where e jϕm(f) denotes the complex phase factor; Based on the Δffundamental and the ΔEharmonic, i, the over-reduction factor a of the speech enhancement algorithm is adjusted dynamically. When the values of the Δffundamental and the ΔEharmonic, i increase, the over-reduction factor a is increased. The enhanced parameters after the dynamic parameter adjustment are output. The enhanced spectrum Gm(f) is reconstructed in the time domain by using the enhanced parameters. The time domain signal s is obtained by inverse Fourier transform of the enhanced spectrum Gm(f) m (n) represents: sm (n) = IFFT (Gm (f)) ; In the formula, IFFT represents an inverse Fourier transform. Using the overlap-add method, each frame of time-domain signal s m (n) is added to the previous frame of time-domain signal s m (n) and the overlap portion is added to the final output of the enhanced signal s(n).

6. A UAV speech enhancement system based on periodic harmonic features and deep learning prediction, used for a UAV speech enhancement method based on periodic harmonic features and deep learning prediction according to any one of claims 1 to 5, characterized in that, It comprises: A self-noise acquisition module is configured to acquire the rotor self-noise signal of the unmanned aerial vehicle, i.e., an original sampling frequency; A periodic harmonic feature extraction module is configured to process the original sampling frequency to obtain a new sampling frequency, filter the new sampling frequency to obtain a normalized signal, extract the normalized signal to obtain a normalized signal spectrum, use a peak detection algorithm to find the peak value in the normalized signal spectrum, take the peak value as a fundamental frequency, extract the normalized signal spectrum to obtain harmonic components, arrange the fundamental frequency and the harmonic components in order to form a feature vector; A deep learning model prediction noise dynamic change module is configured to process the feature vector to generate a noise dynamic change label, combine the feature vector and the noise dynamic change label into a data set, construct a deep learning model, input the data set into the deep learning model for processing, and output a prediction result of the noise dynamic change label; A speech enhancement module is configured to extract the rotor self-noise signal of the unmanned aerial vehicle to obtain a noisy speech amplitude spectrum, calculate the energy distribution of the rotor self-noise signal of the unmanned aerial vehicle using the minimum statistical method according to the prediction result of the noise dynamic change label, smooth the noisy speech amplitude spectrum to obtain a smoothed frequency spectrum, process the smoothed frequency spectrum to obtain the spectral characteristics of the noise, subtract the spectral characteristics of the noise from the noisy speech amplitude spectrum to obtain an enhanced amplitude spectrum, process the noisy speech amplitude spectrum to obtain phase information, combine the enhanced amplitude spectrum and the phase information to construct an enhanced spectrum, and process the enhanced spectrum to obtain an enhanced signal; An output module is configured to output the enhanced signal.

Images