A speech enhancement method and system integrating ultrasonic signal features

Abstract

The present invention discloses a speech enhancement method and system integrating ultrasonic signal features. The ultrasonic signal is first predefined, and then the speaker and microphone provided by the device are used to actively transmit and receive the ultrasonic signal. The channel is estimated to obtain the channel impulse response as a reflection of the movement characteristics of the facial vocal organs when the user speaks, and the channel impulse response is input into the neural network as a supplementary modal information of the speech to realize speech enhancement. The present invention makes full use of the user's vocal action characteristics to assist the speech enhancement task, improves the speech enhancement effect, and has a wide range of application prospects.

Description

Technical Field

[0001] The present invention belongs to the technical field of sound processing, and in particular relates to a speech enhancement method and system integrating ultrasonic signal features. Background Art

[0002] During the transmission process, speech will inevitably be interfered by the surrounding environment. These interferences will seriously affect the quality of speech reception, causing the received signal to no longer be the original pure speech signal, but a speech signal with various interference noises. This will reduce the quality and intelligibility of the speech, affect the listening experience of the speech listener and the accuracy of speech recognition. Therefore, it is necessary to use speech enhancement technology to reduce the impact of the environment and improve speech quality and intelligibility.

[0003] Among various speech enhancement methods, speech enhancement based on deep neural networks has attracted widespread attention from researchers due to its outstanding results. Based on deep neural networks, the introduction of multimodal information has effectively improved the performance of speech enhancement. However, multimodal speech enhancement mainly uses two modalities: sound and vision. Visual information is affected by lighting conditions and requires the support of visual sensors, which also brings privacy issues. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a speech enhancement method and system that integrates ultrasonic signal features in order to address the deficiencies in the above-mentioned prior art, so as to make full use of the vocal movements of the user when speaking and do not require the support of visual information, thereby effectively improving the effect of speech enhancement.

[0005] The present invention adopts the following technical solutions:

[0006] A speech enhancement method integrating ultrasonic signal features, characterized in that it comprises the following steps:

[0007] S1. The mobile device transmits and receives a predefined ultrasonic signal at the same time, and sends the user voice while transmitting the ultrasonic signal; the mobile device receives the ultrasonic signal and the user voice that depict the vocalization action; the ultrasonic signal is used for channel estimation to extract the channel impulse response matrix, the channel impulse response matrix is ​​first-order differentiated bit by bit along the time axis to obtain the first-order differential channel impulse response, the user voice is downsampled, and then the time-frequency spectrum of the voice signal is obtained by short-time Fourier transform;

[0008] S2. The first-order differential channel impulse response obtained in step S1 and the time-frequency spectrum of the speech signal are respectively input into a deep complex neural network, the deep complex neural network predicts a complex-valued ratio mask, the time-frequency spectrum of the speech signal is multiplied bit by bit by the predicted complex-valued ratio mask to obtain the time-frequency spectrum of the enhanced speech, and then the enhanced time series speech signal is obtained by inverse short-time Fourier transform to achieve speech enhancement.

[0009] Specifically, step S1 is as follows:

[0010] S101, the ultrasonic signal at the transmitting end selects a 26-bit GSM sequence as the basic sequence, and the number of sampling points of each ultrasonic signal is 480. When the ultrasonic sampling rate is 48KHz, the system calculates the channel impulse response at a frequency of 100Hz, upsamples the GSM training sequence by 12 times, and then adds 168 bits of zero value at the end as a protection bit to form a training sequence, and finally multiplies the transmitted signal by The above frequency is converted to the ultrasonic frequency band, and band-pass filtering is performed to keep the ultrasonic signal at 18-22KHz. The data of the training sequence frame generated by sampling and zero-filling interpolation according to the GSM sequence is obtained to obtain the cyclic training sequence matrix M;

[0011] S102, at the receiving end, the received ultrasonic signal is divided into two parts by a filter: the noisy user voice is obtained by a low-pass filter with a cutoff frequency of 8 kHz, the user voice part is downsampled from 48 kHz to 16 kHz, and then a short-time Fourier transform is performed with a 20 millisecond Hamming window and a 10 millisecond jump length to obtain a time-frequency spectrum of the voice signal as the input of the neural network speech branch;

[0012] S103, taking the first-order difference of the channel impulse response matrix bit by bit along the time axis to obtain dCIR, and then using dCIR as the input of the ultrasound branch of the neural network.

[0013] Furthermore, in step S101, the cyclic training sequence matrix M is composed of the data part D of the training sequence = {m1, m2, ..., m P}, the cyclic training sequence matrix M is:

[0014]

[0015] Where P is the length of the data portion in the training sequence.

[0016] Further, in step S102, the received signal is filtered through a high-pass filter with a cutoff frequency of 18KHz to obtain the ultrasonic part of the signal. First, frame detection is performed to align the transmitted signal and the received signal. Then, the ultrasonic part of the received signal obtained after high-pass filtering is multiplied by and As the real and imaginary parts of the received baseband signal r(t), the out-of-band noise is then eliminated through a low-pass filter with a cutoff frequency of 2KHz.

[0017] Furthermore, the signal impulse response is calculated by the least squares channel estimation algorithm to obtain a 70×100 complex CIR matrix h as follows:

[0018] h=argmin‖R-Mh‖ 2

[0019] Where R is the received signal and h is the signal impulse response.

[0020] Specifically, in step S2, the time-frequency spectrum of the enhanced speech is specifically:

[0021]

[0022] Where N is the time-frequency spectrum of noisy speech,

[0023] Predicted complex value ratio mask M CRM for:

[0024]

[0025] Among them, N r , N s is the real and imaginary part of the spectrogram of noisy speech; S r , S i are the real and imaginary parts of pure speech, is the real part of CRM, j is the imaginary unit, is the imaginary part of CRM, r is the real part of the complex number, and i is the imaginary part of the complex number.

[0026] Specifically, in step S2, the neural network adopts a complex codec structure, and a complex LSTM is added between the codec; the encoder is used to extract high-dimensional features from the input noisy speech spectrogram and the first-order difference channel impulse response, each encoding block contains branches of the noisy speech spectrogram and the first-order difference channel impulse response, and each branch contains a complex two-dimensional convolution, complex batch normalization and leaky complex rectifier unit activation; the encoded high-dimensional audio and first-order difference channel impulse response features are input into an interactive module for converting and sharing information; then, the decoder uses complex deconvolution to reconstruct the low-resolution features to the scale of the original input, and a jump connection is used between the codec. At the end of the ultrasonic branch of the encoder, a user authentication branch is added, and user prediction is performed after full connection.

[0027] Furthermore, the interaction module is specifically:

[0028] The two input branch features F1 and F2 are connected, and then fed into complex convolution, complex batch normalization, and complex Sigmoid activation to predict the valid features that need to be saved in F2. The new feature of F1 is expressed as: The new feature of F2 is expressed as H(·) represents the combined operation of complex convolution, complex batch normalization, and complex sigmoid activation. Furthermore, the loss function L of the neural network is:

[0029] L=αL SISDR +βL s

[0030] Among them, L SISDR is the signal scale invariant signal-to-noise ratio loss function, L s is the cross entropy loss function predicted by the user, and α and β are hyperparameters.

[0031] Another technical solution of the present invention is a speech enhancement system integrating ultrasonic signal features, comprising:

[0032] Input module, the mobile device simultaneously transmits and receives a predefined ultrasonic signal, and sends the user voice while transmitting the predefined ultrasonic signal; the mobile device receives the ultrasonic signal and the user voice that depict the vocalization action; the ultrasonic part is used for channel estimation to extract the channel impulse response matrix, the channel impulse response matrix is ​​first-order differentiated bit by bit along the time axis to obtain the first-order differential channel impulse response, the user voice part that has been low-pass filtered is downsampled, and the time-frequency spectrum of the voice signal is obtained by short-time Fourier transform;

[0033] The enhancement module inputs the first-order differential channel impulse response obtained by the input module and the time-frequency spectrum of the speech signal into the deep complex neural network respectively. The deep complex neural network predicts the complex-valued ratio mask, and multiplies the time-frequency spectrum of the speech signal with the predicted complex-valued ratio mask bit by bit to obtain the time-frequency spectrum of the enhanced speech. Then, the enhanced time series speech signal is obtained by inverse short-time Fourier transform to achieve speech enhancement.

[0034] Compared with the prior art, the present invention has at least the following beneficial effects:

[0035] The present invention discloses a speech enhancement method integrating ultrasonic signal characteristics. The ultrasonic signal is first predefined, and then the speaker and microphone provided by the device are used to actively transmit and receive the ultrasonic signal. The channel estimation is performed to obtain the channel impulse response (CIR) as a motion feature reflecting the facial vocal organs (such as lips, chin, and tongue) of the user when speaking, and the CIR is input into the neural network as a supplementary modal information of the speech to realize speech enhancement. In order to simultaneously process the amplitude and phase of the complex speech signal in the time-frequency domain to utilize all the information of the speech signal, the present invention adopts a complex neural network, whose internal operations all comply with the complex number operation rules, and the network output is a complex ratio mask (CRM). The CIR is multiplied with the time-frequency spectrum of the input speech to obtain the predicted speech time-frequency spectrum, and the speech time-frequency spectrum is further subjected to an inverse short-time Fourier transform to obtain the enhanced speech. The first-order difference channel impulse response can effectively reflect the motion features of the facial organs when the user speaks, and as the available redundant information, the speech enhancement task effect is improved.

[0036] Furthermore, in order to estimate the channel impulse response, the transmitted signal and the received signal need to be known. At the same time, in order to ensure the synchronization of the received signal and the transmitted signal, a 26-bit GSM sequence with good autocorrelation is used as the basic sequence. In order to make it have sufficient length, it is first upsampled by 12 times, and then zeros are added at the end to prevent the echo of the current frame from mixing with the next frame. The signal is modulated to a frequency band of 18 to 22 kHz to ensure that the human ear cannot perceive the sound signal at the transmitter. At the receiving end, the received signal is divided into two parts, one part is the ultrasonic part above 18 kHz, which is used for channel estimation and calculation of the channel impulse response; the other part is the noisy speech part as input. In order to reduce data complexity, the sampling rate is reduced from 48 kHz to 16 kHz.

[0037] Furthermore, the cyclic training matrix M represents data of a frame of training signal sequence, which is used as a transmission signal during channel estimation.

[0038] Furthermore, the low-pass filter with a cut-off frequency of 2 kHz is used to eliminate out-of-band noise and prevent sound signals outside the ultrasonic frequency band from affecting channel estimation.

[0039] Furthermore, the least squares channel estimation algorithm can effectively calculate the time-varying characteristics of the channel, and has a simple structure and low computational complexity.

[0040] Furthermore, compared to predicting speech signals directly in the time domain, the time-frequency domain method considers both the time domain and frequency information of speech, and the prediction result is more accurate. The method of predicting complex ratio mask can simultaneously predict the amplitude and phase of speech signals in the time-frequency domain, and can achieve almost lossless reconstruction of pure speech.

[0041] Furthermore, since the input of the network is dCIR and time-frequency domain speech signals, both of which are complex matrices, the complex encoding and decoding structure can effectively model the real and imaginary parts of the input complex signal. The output result will consider both the real and imaginary parts of the signal, making the prediction result more reliable.

[0042] Furthermore, the user's movements and speech are naturally related, so the features learned in one modality can be used to supplement the features of another modality. The interactive module can realize information exchange between the speech branch and the ultrasound branch, and can use the feature information of one branch to restore some lost features in the other branch or delete some unnecessary features.

[0043] Furthermore, L SISDR The loss function can reflect the difference between the predicted signal and the clear signal. S The loss function is used for user prediction and can leverage prior knowledge of the target speaker to improve speech enhancement.

[0044] In summary, the method of the present invention is applicable to commercial mobile devices equipped with a speaker and a microphone, and can effectively utilize the correlation between the real part and the imaginary part of the complex signal, thereby improving the speech enhancement effect.

[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a system diagram of the present invention;

[0047] Figure 2 This is a diagram of ultrasonic signal transmission and reception of the present invention;

[0048] Figure 3 This is a network structure diagram of the present invention;

[0049] Figure 4 : is a network structure parameter diagram of the present invention, wherein (a) is the structure parameter of the ultrasound branch encoder, and (b) is the structure parameter of the speech branch encoder (upper) and decoder (lower);

[0050] Figure 5 This is a structural diagram of the interaction module of the present invention;

[0051] Figure 6 Schematic diagram of complex convolution operation of the present invention;

[0052] Figure 7 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0055] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0056] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0057] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0058] The present invention provides a speech enhancement method integrating ultrasonic signal features, which is applicable to commercial mobile devices equipped with speakers and microphones; firstly, an ultrasonic signal is predefined, and then the speaker and microphone of the device are used to actively transmit and receive the ultrasonic signal, and the channel is estimated to obtain the channel impulse response (CIR) as a motion feature reflecting the facial vocal organs (such as lips, chin, and tongue) of the user when speaking, and the CIR is input into a neural network as a supplementary modal information of the speech to achieve speech enhancement. In order to simultaneously process the amplitude and phase of the complex speech signal in the time-frequency domain to utilize all the information of the speech signal, the present invention adopts a complex neural network, whose internal operations all comply with the complex number operation rules, and the network output is a complex ratio mask (Complex Ratio Mask, CRM), which is multiplied with the time-frequency spectrum of the input speech to obtain the predicted speech time-frequency spectrum, and the speech time-frequency spectrum is further inversely short-time Fourier transformed to obtain the enhanced speech. The present invention makes full use of the user's vocal action features to assist the speech enhancement task, improves the speech enhancement effect, and has a wide range of application prospects.

[0059] See also Figure 1 and Figure 7 The present invention provides a speech enhancement method integrating ultrasonic signal features, including an ultrasonic signal transceiver module and a deep network prediction module, and the specific steps are as follows:

[0060] S1. A commercial mobile device (such as a smartphone) with a speaker and a microphone is used to simultaneously transmit and receive a predefined ultrasonic signal. While transmitting the ultrasonic signal, the user speaks. At this time, the signal received by the microphone is divided into two parts. One part is an ultrasonic signal that depicts the vocal action, which is obtained by a high-pass filter with a cutoff frequency of 18KHz; the other part is the user's voice, which is obtained by a low-pass filter with a cutoff frequency of 8KHz. The ultrasonic part is used for channel estimation to extract the channel impulse response (CIR), which is used as auxiliary information reflecting the vocal action to achieve noise reduction for the voice part.

[0061] See also Figure 2 , the ultrasonic signal transceiver module is specifically:

[0062] S101, the ultrasonic signal at the transmitting end selects a 26-bit GSM sequence as the basic sequence, and the number of sampling points of each ultrasonic signal is 480. When the ultrasonic sampling rate is 48KHz, the system can calculate the CIR at a frequency of 100Hz, upsample the GSM sequence by 12 times, and then add 168 bits of zero value as a protection bit at the end to form a training sequence, and finally multiply the transmitted signal by The above frequency is converted to the ultrasonic frequency band (f c=20KHz), and band-pass filtering is performed to keep the ultrasonic signal at 18-22KHz;

[0063] S102. At the receiving end, the received ultrasonic signal is first divided into two parts by a filter: the noisy user voice is obtained by a low-pass filter with a cutoff frequency of 8 kHz, the user voice part is downsampled from 48 kHz to 16 kHz to reduce the complexity of the model, and then a short-time Fourier transform (STFT) is performed with a 20-ms Hamming window and a 10-ms jump length to obtain a time-frequency spectrum of the speech signal as the input of the neural network speech branch.

[0064] The received signal is filtered through a high-pass filter with a cutoff frequency of 18KHz to obtain the ultrasonic part of the signal. At the receiving end, the GSM training sequence has the characteristic of good autocorrelation. First, frame detection is performed to align the transmitted signal and the received signal. Then, the ultrasonic part of the received signal obtained after high-pass filtering is multiplied by and As the real and imaginary parts of the received baseband signal r(t), the out-of-band noise is then eliminated through a low-pass filter with a cutoff frequency of 2KHz.

[0065] The cyclic training sequence matrix M consists of the data part of the training sequence D = {m1, m2, ..., m P}Transformation yields:

[0066]

[0067] Wherein, P is the length of the data part in the training sequence, P=D+N=312, and N is 70.

[0068] The channel impulse response h is obtained by the least squares channel estimation algorithm:

[0069] h=argmin‖R-Mh‖ 2

[0070] Where R is the received signal and h is the signal impulse response.

[0071] By h LS =(M H M) -1 M H R obtains the signal impulse response h. According to the design, a 70×100 complex CIR matrix h is obtained every second.

[0072] S103, taking the first-order difference of the CIR matrix bit by bit along the time axis to obtain the first-order differential channel impulse response dCIR, which can effectively eliminate the reflection of static objects in the environment and highlight the ultrasonic characteristics of mouth movement. Then, dCIR is used as the input of the ultrasonic branch of the neural network.

[0073] S2. In the deep network prediction module, dCIR and speech time-spectrogram are respectively input into the deep complex neural network. The network predicts the complex ratio mask (CRM). The speech time-spectrogram and the complex ratio mask are multiplied bit by bit to obtain the time-spectrogram of the enhanced speech. Then, the enhanced time series speech signal can be obtained by inverse short-time Fourier transform.

[0074] The neural network performs speech enhancement by predicting a complex ratio mask (CRM), which is the ratio between the spectrogram of clean and noisy speech and is defined as:

[0075]

[0076] Among them, N r , N s is the real and imaginary part of the spectrogram of noisy speech; S r , S i are the real and imaginary parts of the pure speech. In addition, in order to narrow the search space to optimize the model, the amplitude of CRM is limited to [0,1) using the hyperbolic tangent function, and the phase of CRM is also changed accordingly:

[0077]

[0078]

[0079]

[0080] Where D is the output of the network.

[0081] The final estimated pure speech spectrogram can be calculated by the following formula:

[0082]

[0083] Wherein, N is the spectrogram of noisy speech.

[0084] See also Figure 3 The neural network adopts a complex codec structure and adds a complex LSTM between the codec. In the codec, the complex LSTM is used to model the temporal correlation. The encoder is used to extract high-dimensional features from the input noisy speech spectrogram and dcir. Each encoding block contains branches for the noisy speech spectrogram and dcir, each of which contains a complex two-dimensional convolution, complex batch normalization and leaky CReLU activation. The network structure parameters are as follows: Figure 4 shown.

[0085] Then, the encoded high-dimensional audio and DCIR features are input into an interaction module for converting and sharing information; the interaction module structure is as follows Figure 5As shown, the two input branch features F1 and F2 are first connected, and then fed into complex convolution, complex batch normalization, and complex Sigmoid activation to predict the valid features that need to be saved in F2. The new features of F1 can be expressed as: Similarly, the new feature of F2 can be expressed as Where H(·) represents the combined operation of complex convolution, complex batch normalization, and complex sigmoid activation. The new features output by the interaction module can effectively retain the part of another feature that needs to be retained on the basis of the original feature, thus realizing the fusion of the two branch information.

[0086] Subsequently, the decoder uses complex deconvolution to reconstruct the low-resolution features to the scale of the original input. Skip connections are used between the encoder and decoder to effectively promote the transfer of gradients. At the end of the ultrasonic branch of the encoder, a user authentication branch is added, and the user is predicted after full connection, which can further use the user's identity information to improve the speech enhancement effect.

[0087] The operations of complex neural networks all conform to the complex number operation rules. Taking complex convolution as an example, let the convolution kernel be W = A + iB, and the input be I = x + iy, then the output F Conv for:

[0088] F Conv =W*I=(A*xB*y)+i(B*x+A*y)

[0089] Among them, the neural network loss function is:

[0090] L=αL SISDR +βL s

[0091] Among them, L SISDR is the signal scale invariant signal-to-noise ratio loss function, L s is the cross entropy loss function predicted by the user, α and β are hyperparameters, which are 1 and 0.1 respectively.

[0092] Scale-invariant signal-to-noise ratio loss function L SISDR for:

[0093]

[0094] S E =S′-S T

[0095]

[0096] Among them, S′ is the predicted speech sequence, S is the original pure speech sequence, and S TIt is the projection of S′ on S, and SISNR reflects the similarity between S′ and S.

[0097] Cross entropy loss function L s for:

[0098]

[0099] Where n is the number of speakers, here it is 10, is the true label of the i-th speaker identity, is the prediction of the identity label of the i-th speaker.

[0100] In yet another embodiment of the present invention, a speech enhancement system integrating ultrasonic signal features is provided. The system can be used to implement the above-mentioned speech enhancement method integrating ultrasonic signal features. Specifically, the speech enhancement system integrating ultrasonic signal features includes an input module and an enhancement module.

[0101] Among them, the input module, the mobile device simultaneously transmits and receives a predefined ultrasonic signal, and sends the user voice while transmitting the predefined ultrasonic signal; the mobile device receives the ultrasonic signal and the user voice that depict the vocalization action; the ultrasonic part is used for channel estimation to extract the channel impulse response matrix, and the channel impulse response matrix is ​​first-order differentiated bit by bit along the time axis to obtain the first-order differential channel impulse response, and the user voice part that has been low-pass filtered is downsampled, and the time-frequency spectrum of the voice signal is obtained by short-time Fourier transform;

[0102] The enhancement module inputs the first-order differential channel impulse response obtained by the input module and the time-frequency spectrum of the speech signal into the deep complex neural network respectively. The deep complex neural network predicts the complex-valued ratio mask, and multiplies the time-frequency spectrum of the speech signal with the predicted complex-valued ratio mask bit by bit to obtain the time-frequency spectrum of the enhanced speech. Then, the enhanced time series speech signal is obtained by inverse short-time Fourier transform to achieve speech enhancement.

[0103] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the speech enhancement method integrating ultrasonic signal features, including:

[0104] The mobile device transmits and receives a predefined ultrasonic signal simultaneously, and sends the user voice while transmitting the ultrasonic signal; the mobile device receives the ultrasonic signal and the user voice that depicts the vocalization action; the ultrasonic signal is used for channel estimation to extract the channel impulse response matrix, and the channel impulse response matrix is ​​first-order differentiated bit by bit along the time axis to obtain the first-order differential channel impulse response, the user voice is downsampled, and then the time-frequency spectrum of the voice signal is obtained by short-time Fourier transform; the first-order differential channel impulse response and the time-frequency spectrum of the voice signal are respectively input into a deep complex neural network, the deep complex neural network predicts a complex-valued ratio mask, the time-frequency spectrum of the voice signal is multiplied bit by bit with the predicted complex-valued ratio mask to obtain the time-frequency spectrum of the enhanced voice, and then the enhanced time series voice signal is obtained by inverse short-time Fourier transform to achieve voice enhancement.

[0105] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0106] The processor may load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the speech enhancement method for integrating ultrasonic signal features in the above embodiment; the processor may load and execute the following steps:

[0107] The mobile device transmits and receives a predefined ultrasonic signal simultaneously, and sends the user voice while transmitting the ultrasonic signal; the mobile device receives the ultrasonic signal and the user voice that depicts the vocalization action; the ultrasonic signal is used for channel estimation to extract the channel impulse response matrix, and the channel impulse response matrix is ​​first-order differentiated bit by bit along the time axis to obtain the first-order differential channel impulse response, the user voice is downsampled, and then the time-frequency spectrum of the voice signal is obtained by short-time Fourier transform; the first-order differential channel impulse response and the time-frequency spectrum of the voice signal are respectively input into a deep complex neural network, the deep complex neural network predicts a complex-valued ratio mask, the time-frequency spectrum of the voice signal is multiplied bit by bit with the predicted complex-valued ratio mask to obtain the time-frequency spectrum of the enhanced voice, and then the enhanced time series voice signal is obtained by inverse short-time Fourier transform to achieve voice enhancement.

[0108] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0109] See also Figure 6 , is a schematic diagram of complex convolution operation. Assume that the input matrix I = x + jy, the complex convolution kernel W = A + jB, then the output F after the convolution operation can be expressed as F = W*I = (A*x-B*y) + j(B*x + A*y). The use of complex convolution can further utilize the intrinsic connection between amplitude and phase for prediction, which is beneficial to speech enhancement tasks.

[0110] In order to demonstrate the improvement of speech enhancement achieved by this invention, it was compared with the related work PHASEN system. The average signal-to-distortion ratio (SDR) of noisy data was 4.99dB, the average perceptual evaluation of speech quality (PESQ) was 2.18, and the average short-time objective intelligibility measure (STOI) was 0.77.

[0111] After the noisy speech is enhanced, the SNR is 16.91dB, the PESQ is 3.27, and the STOI is 0.90.

[0112] The three values ​​after the PHASEN system enhancement are 13.34dB, 3.15, and 0.88 respectively.

[0113] Compared with PHASEN, the present invention has a significantly improved enhancement effect on noisy speech.

[0114] In summary, the present invention provides a speech enhancement method and system integrating ultrasonic signal features, which is applicable to commercial mobile devices equipped with speakers and microphones. The speaker and microphone provided by the device are used to actively transmit and receive ultrasonic signals, and channel estimation is performed to obtain the channel impulse response (CIR) as a reflection of the movement characteristics of the facial vocal organs (such as lips, chin, and tongue) when the user speaks, which is input into the neural network as a supplementary modal information to interact with the noisy speech, thereby improving the enhancement effect. In order to simultaneously process the amplitude and phase of the complex speech signal in the time-frequency domain to utilize all the information of the speech signal, the network adopts a complex neural network, whose internal operations all comply with the complex operation rules, and can utilize the correlation between the real part and the imaginary part of the complex signal, effectively improving the speech quality and intelligibility.

[0115] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0116] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0119] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A speech enhancement method integrating ultrasonic signal features, characterized in that: The following steps are involved: S1. The mobile device transmits and receives a predefined ultrasonic signal at the same time, and sends the user voice while transmitting the ultrasonic signal; the mobile device receives the ultrasonic signal and the user voice that depict the vocalization action; the ultrasonic signal is used for channel estimation to extract the channel impulse response matrix, and the channel impulse response matrix is ​​first-order differentiated bit by bit along the time axis to obtain the first-order differential channel impulse response, the user voice is downsampled, and then the time-frequency spectrum of the voice signal is obtained by short-time Fourier transform, which is specifically: S101, the ultrasonic signal at the transmitting end selects the GSM sequence as the basic sequence. When the sampling rate of the ultrasonic wave is 48KHz, the channel impulse response is calculated at a frequency of 100Hz, the GSM training sequence is upsampled by 12 times, and then a 168-bit zero value is added at the end as a protection bit to form a training sequence. Finally, the transmitting signal is multiplied by The frequency is converted to the ultrasonic frequency band, and bandpass filtering is performed to keep the ultrasonic signal at 18~22KHz. The data of the training sequence frame generated by sampling and zero-filling interpolation based on the GSM sequence is obtained to obtain the cyclic training sequence matrix ; S102, at the receiving end, the received ultrasonic signal is divided into two parts by a filter: the noisy user voice is obtained by a low-pass filter with a cutoff frequency of 8 kHz, the user voice part is downsampled from 48 kHz to 16 kHz, and then a short-time Fourier transform is performed with a 20 millisecond Hamming window and a 10 millisecond jump length to obtain a time-frequency spectrum of the voice signal as the input of the neural network speech branch; S103, taking the first-order difference of the channel impulse response matrix bit by bit along the time axis to obtain a first-order differential channel impulse response, and then using the first-order differential channel impulse response as an input of the ultrasound branch of the neural network; S2. The first-order differential channel impulse response obtained in step S1 and the time-frequency spectrum of the speech signal are respectively input into a deep complex neural network, the deep complex neural network predicts a complex-valued ratio mask, the time-frequency spectrum of the speech signal is multiplied bit by bit by the predicted complex-valued ratio mask to obtain the time-frequency spectrum of the enhanced speech, and then the enhanced time series speech signal is obtained by inverse short-time Fourier transform to achieve speech enhancement.

2. The method for speech enhancement by integrating ultrasonic signal features according to claim 1, characterized in that: In step S101, the training sequence matrix is ​​looped The data part of the training sequence D = { }, loop training sequence matrix for: Where P is the length of the data portion in the training sequence.

3. The method for speech enhancement by integrating ultrasonic signal features according to claim 1, characterized in that: In step S102, the received signal is filtered through a high-pass filter with a cutoff frequency of 18KHz to obtain the ultrasonic part of the signal. First, frame detection is performed to align the transmitted signal and the received signal. Then, the ultrasonic part of the received signal obtained after high-pass filtering is multiplied by and As the real and imaginary parts of the received baseband signal r(t), the out-of-band noise is then eliminated through a low-pass filter with a cutoff frequency of 2KHz.

4. The method for speech enhancement by integrating ultrasonic signal features according to claim 3, characterized in that: The signal impulse response is calculated by the least squares channel estimation algorithm to obtain a 70×100 complex CIR matrix as follows: in, To receive the signal, is the signal impulse response.

5. The method for speech enhancement by integrating ultrasonic signal features according to claim 1, characterized in that: In step S2, the time-spectrogram of the enhanced speech is specifically: in, Predict the complex-valued ratio mask for the spectrogram of noisy speech for: in, , are the real and imaginary parts of the spectrogram of noisy speech; , are the real and imaginary parts of pure speech, is the real part of CRM, is an imaginary unit, is the imaginary part of CRM, is the real part of the complex number, is the imaginary part of the complex number.

6. The method for speech enhancement by integrating ultrasonic signal features according to claim 1, characterized in that: In step S2, the neural network adopts a complex codec structure, and a complex LSTM is added between the codec; the encoder is used to extract high-dimensional features from the input noisy speech spectrogram and the first-order difference channel impulse response, each encoding block contains branches of the noisy speech spectrogram and the first-order difference channel impulse response, and each branch contains complex two-dimensional convolution, complex batch normalization and leaky complex rectifier unit activation; the encoded high-dimensional audio and first-order difference channel impulse response features are input into an interactive module for converting and sharing information; Subsequently, the decoder uses complex deconvolution to reconstruct the low-resolution features to the scale of the original input, and uses skip connections between the encoder and decoder. At the end of the encoder ultrasound branch, a user authentication branch is added, and user prediction is performed after full connection.

7. The method for speech enhancement by integrating ultrasonic signal features according to claim 6, characterized in that: The interaction modules are as follows: The two input branch features F1 and F2 are connected, and then fed into complex convolution, complex batch normalization, and complex Sigmoid activation to predict the valid features that need to be saved in F2. The new feature of F1 is expressed as: , the new feature of F2 is expressed as , H(·) represents the combined operation of complex convolution, complex batch normalization, and complex sigmoid activation.

8. The method for speech enhancement by integrating ultrasonic signal features according to claim 6, characterized in that: Loss Function of Neural Network for: in, is the signal scale invariant signal-to-noise ratio loss function, is the cross entropy loss function predicted by the user, and α and β are hyperparameters.

9. A speech enhancement system integrating ultrasonic signal features, characterized in that: The method for speech enhancement by integrating ultrasonic signal features according to claim 1 comprises: Input module, the mobile device simultaneously transmits and receives a predefined ultrasonic signal, and sends the user voice while transmitting the predefined ultrasonic signal; the mobile device receives the ultrasonic signal and the user voice that depict the vocalization action; the ultrasonic part is used for channel estimation to extract the channel impulse response matrix, the channel impulse response matrix is ​​first-order differentiated bit by bit along the time axis to obtain the first-order differential channel impulse response, the user voice part that has been low-pass filtered is downsampled, and the time-frequency spectrum of the voice signal is obtained by short-time Fourier transform; The enhancement module inputs the first-order differential channel impulse response obtained by the input module and the time-frequency spectrum of the speech signal into the deep complex neural network respectively. The deep complex neural network predicts the complex-valued ratio mask, and multiplies the time-frequency spectrum of the speech signal with the predicted complex-valued ratio mask bit by bit to obtain the time-frequency spectrum of the enhanced speech. Then, the enhanced time series speech signal is obtained by inverse short-time Fourier transform to achieve speech enhancement.

Images