Sound enhancement method and earphone

Abstract

The invention provides a sound enhancement method and an earphone, and the method comprises the steps: collecting a sound signal, and judging whether the sound signal is a sound signal sent by a user or not; if the sound signal is the sound signal sent by the user, performing short-time Fourier transform on the sound signal to obtain a sound spectrogram; performing different convolution processing on the sound spectrogram to obtain an amplitude characteristic and a phase characteristic respectively, and performing information exchange on the amplitude characteristic and the phase characteristic by using an exchange process; performing prediction processing on the amplitude characteristic after information exchange to obtain a predicted amplitude, and performing prediction processing on the phase characteristic after information exchange to obtain a predicted phase; and combining the predicted amplitude with the predicted phase by using a combination formula to obtain a predicted spectrogram, and carrying out noise reduction processing on the predicted spectrogram by using a noise reduction method to obtain enhanced sound, thereby improving the user experience.

Description

Technical Field

[0001] The invention belongs to the technical field of audio processing, and in particular relates to a sound enhancement method and earphone. Background Art

[0002] With the rapid development of artificial intelligence and voice recognition technology, smart headphones, as a portable electronic device, have gradually become an indispensable part of users' daily life and work.

[0003] When existing earphones collect the voice of a speaker, they are often disturbed by background noise, which causes the sound signal to be polluted and seriously affects the accuracy of voice recognition. Therefore, it is very necessary to denoise and enhance the speaker's voice.

[0004] However, current sound enhancement technologies usually have some problems, and the enhancement effect is not ideal, which directly affects the user experience. In order to improve this situation, it is necessary to develop more effective sound enhancement methods to improve the auditory experience of the speaker's voice. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a sound enhancement method and earphones, which are used to solve the technical problems in the prior art.

[0006] On the one hand, the invention provides the following technical solution, a sound enhancement method, the method comprising:

[0007] Collecting a sound signal, and determining whether the sound signal is a sound signal emitted by a user;

[0008] If the sound signal is a sound signal emitted by a user, performing a short-time Fourier transform on the sound signal to obtain a sound spectrogram;

[0009] Performing different convolution processing on the sound spectrogram to obtain amplitude features and phase features respectively, and using an exchange process to exchange information between the amplitude features and the phase features;

[0010] Performing prediction processing on the amplitude feature after the information exchange to obtain a predicted amplitude, and performing prediction processing on the phase feature after the information exchange to obtain a predicted phase;

[0011] The predicted amplitude and the predicted phase are combined by using a combination formula to obtain a predicted spectrogram, and the predicted spectrogram is subjected to noise reduction processing by using a noise reduction method to obtain an enhanced sound.

[0012] Compared with the prior art, the beneficial effects of the present application are as follows: by collecting sound signals and determining whether the sound signals are sound signals emitted by the user, it is helpful to distinguish between speech and noise, and improve the accuracy of speech recognition or speech enhancement applications. By combining the predicted amplitude with the predicted phase using a combination formula to obtain a predicted spectrogram, the wearer's voice is effectively enhanced, and then the predicted spectrogram is subjected to noise reduction processing using a noise reduction method to obtain an enhanced sound step, which removes the residual noise in the enhanced sound and further enhances the wearer's voice, thereby improving the user experience.

[0013] Furthermore, the step of determining whether the sound signal is a sound signal emitted by a user includes:

[0014] Acquire a bone conduction signal in the sound signal, and perform time-frequency transformation on the bone conduction signal to obtain a frequency domain signal;

[0015] Calculating the time domain information of the sound signal based on the bone conduction signal, and determining whether the time domain information and the frequency domain signal are greater than their corresponding thresholds respectively;

[0016] If so, the sound signal is a sound signal emitted by the user.

[0017] Further, the exchange process includes an extraction process and an exchange process;

[0018] The extraction process includes: inputting the amplitude feature into a first extraction network to obtain a first extraction feature;

[0019] The first extracted feature is processed by three convolution layers to obtain an amplitude convolution feature;

[0020] The convolutional features are input into a first extraction network to obtain second extraction features.

[0021] Further, the exchange process includes: performing secondary convolution layer processing on the phase feature to obtain a phase convolution feature, and exchanging information between the phase convolution feature and the second extracted feature;

[0022] The second extracted features after information exchange are subjected to the extraction process at least once more, and the phase convolution features after information exchange are subjected to the exchange process at least once more.

[0023] Furthermore, the step of performing prediction processing on the amplitude feature after information exchange to obtain the predicted amplitude includes:

[0024] The amplitude features after information exchange are reshaped into a one-dimensional feature map, and the one-dimensional feature map is sequentially input into a bidirectional long short-term memory network and three fully connected layers to obtain a predicted amplitude.

[0025] Furthermore, the step of performing prediction processing on the phase feature after information exchange to obtain a predicted phase includes:

[0026] The phase feature after information exchange is reshaped into a complex-valued feature map, and the amplitude of the complex-valued feature map is normalized to obtain a predicted phase.

[0027] Furthermore, the combination formula includes:

[0028]

[0029] Among them, A′ represents the predicted spectrogram, abs(A) represents the absolute value of the sound spectrogram, F represents the predicted amplitude, and D represents the predicted phase.

[0030] Furthermore, the noise reduction method includes spectral subtraction.

[0031] In a second aspect, the invention provides the following technical solution: an earphone comprising a memory, a processor, and a sound enhancement program stored in the memory and executable on the processor, wherein the sound enhancement program implements the sound enhancement method as claimed in claim 1 when executed by the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0033] Figure 1 The present invention provides a flow chart of a sound enhancement method.

[0034] The embodiments of the present invention will be further described below with reference to the accompanying drawings. DETAILED DESCRIPTION

[0035] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout are the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the embodiments of the present invention, and should not be construed as limiting the present invention.

[0036] In the description of the embodiments of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0037] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0038] In the embodiments of the present invention, please refer to Figure 1 As shown, a sound enhancement method includes the following steps S01 to S05:

[0039] S01, collecting a sound signal, and determining whether the sound signal is a sound signal emitted by a user;

[0040] Specifically, the step of determining whether the sound signal is a sound signal emitted by a user includes:

[0041] S11, obtaining a bone conduction signal in the sound signal, and performing time-frequency transformation processing on the bone conduction signal to obtain a frequency domain signal;

[0042] S12, calculating the time domain information of the sound signal based on the bone conduction signal, and determining whether the time domain information and the frequency domain signal are greater than their corresponding thresholds respectively;

[0043] S13, if yes, then the sound signal is a sound signal emitted by the user

[0044] It is understandable that when the wearer is making a voice call, the headset will continue to use its built-in sound collection device (including vibration sensor) to collect sound signals. These sound signals include the voice uttered by the wearer and the noise of the surrounding environment.

[0045] Once the headset collects these sound signals, it will immediately use the bone conduction signal captured by the vibration sensor to determine whether the sound signal is the sound signal emitted by the wearer. Specifically, the headset determines whether the captured sound is the voice of the wearer or the noise of the surrounding environment by analyzing the bone conduction signal. This recognition method helps the headset to distinguish and process voice and noise more accurately, thereby improving the quality of voice calls.

[0046] In this embodiment, the frequency domain signal includes the zero-crossing rate and the pitch period; the frequency domain information includes the spectrum energy and the spectrum centroid, and the zero rate, the pitch period, the spectrum energy and the spectrum centroid are compared with the corresponding thresholds to determine whether the sound signal is a sound signal emitted by the user. The specific comparison is as follows;

[0047] Spectrum energy comparison: compare the obtained spectrum energy with the first threshold. If the spectrum energy is greater than or equal to the first threshold, the condition is met.

[0048] Zero crossing rate comparison: Compare the obtained zero crossing rate with the second threshold. If the zero crossing rate is less than or equal to the second threshold, the condition is met.

[0049] Spectral centroid comparison: Compare the obtained spectral centroid (brightness) with the third threshold. If the spectral centroid is less than or equal to the third threshold, the condition is met.

[0050] Pitch period comparison: the obtained pitch period is compared with the fourth threshold and the fifth threshold respectively. If the pitch period is between the fourth threshold and the fifth threshold, the condition is satisfied.

[0051] Only when all the above conditions are met at the same time, the sound signal is whether it is a sound signal emitted by the user. If any one of the conditions is not met, the sound signal is not a sound signal emitted by the user. Such a design helps to distinguish between speech and noise and improve the accuracy of speech recognition or speech enhancement applications.

[0052] S02, if the sound signal is a sound signal emitted by a user, performing a short-time Fourier transform on the sound signal to obtain a sound spectrogram;

[0053] Specifically, first, the sound signal is subjected to a short-time Fourier transform (STFT) to convert it from the time domain to the frequency domain to form an STFT spectrogram (sound spectrogram). The purpose of this step is to analyze the distribution of the speech signal at different frequencies and times, and provide a frequency domain representation for subsequent signal processing.

[0054] S03, performing different convolution processing on the sound spectrogram to obtain amplitude features and phase features respectively, and performing information exchange on the amplitude features and the phase features using an exchange process;

[0055] The step of performing different convolution processing on the sound spectrogram includes inputting the sound spectrogram into two branches respectively, and performing feature calculation separately to obtain amplitude features and phase features respectively. Specifically, the two branches are formed by different two-dimensional convolutions, wherein the branch for generating amplitude features is amplitude masking prediction, and the branch for generating amplitude features includes a 1×7 and a 7×1 convolution. The branch for generating phase features is phase prediction, and the branch for generating phase features includes a 5×3 and a 25×1 convolution.

[0056] Specifically, the exchange process includes an extraction process and an exchange process;

[0057] The extraction process includes: inputting the amplitude feature into a first extraction network to obtain a first extraction feature;

[0058] The first extracted feature is processed by three convolution layers to obtain an amplitude convolution feature;

[0059] The convolutional features are input into a first extraction network to obtain second extraction features.

[0060] More specifically, the exchange process includes: performing a secondary convolution layer process on the phase feature to obtain a phase convolution feature, and exchanging information between the phase convolution feature and the second extracted feature;

[0061] The second extracted features after information exchange are subjected to the extraction process at least once more, and the phase convolution features after information exchange are subjected to the exchange process at least once more.

[0062] In this embodiment, the first extraction network and the second extraction network are both harmonic extraction networks. Performing the extraction process at least once is performing the extraction process twice, and performing the exchange process at least once is performing the exchange process twice. It can be understood that a total of three information exchanges are performed.

[0063] S04, performing prediction processing on the amplitude feature after the information exchange to obtain a predicted amplitude, and performing prediction processing on the phase feature after the information exchange to obtain a predicted phase;

[0064] Specifically, the step of performing prediction processing on the amplitude feature after information exchange to obtain the predicted amplitude includes:

[0065] The amplitude features after information exchange are reshaped into a one-dimensional feature map, and the one-dimensional feature map is sequentially input into a bidirectional long short-term memory network and three fully connected layers to obtain a predicted amplitude.

[0066] In this embodiment, the amplitude feature after information exchange is first further processed through a 1×1 convolution layer, the purpose of which is to reduce the number of feature channels of the amplitude feature after information exchange to Cr=8, where Cr is the number of feature channels after 1×1 convolution.

[0067] Next, these features (amplitude features after 1×1 convolutional layer processing) are reshaped into a one-dimensional feature map. This reshaping operation is to convert the feature map into a format suitable for subsequent Bi-LSTM processing.

[0068] These one-dimensional feature maps are then fed into the Bi-LSTM layer. Bi-LSTM, or bidirectional long short-term memory network, is able to capture the dependencies between the past and the future in sequence data, which is very useful for processing time series data such as speech signals. Bi-LSTM can consider both past and future information at the same time, thus providing more comprehensive sequence features.

[0069] After Bi-LSTM processing, the features are fed into three fully connected layers (FC) for further nonlinear transformation. The fully connected layers can be seen as the "decision-making" part of the network, which maps the features extracted by the previous layers to the final output space. Among these fully connected layers, except for the last layer which uses the Sigmoid activation function, all other layers use the ReLU activation function. The Sigmoid function is used to limit the output between 0 and 1, which makes sense in amplitude prediction because the amplitude values ​​are usually in this range. The ReLU function is used to introduce nonlinearity to help the network learn more complex feature representations.

[0070] Finally, the outputs of these fully connected layers are used to predict the amplitude. This predicted amplitude will be combined with the phase information to reconstruct the spectrogram of the speech signal. In this way, the PHASEN-SS network can extract clear speech features from noisy speech signals, providing a basis for subsequent speech enhancement and noise reduction.

[0071] Specifically, the step of performing prediction processing on the phase feature after information exchange to obtain a predicted phase includes:

[0072] The phase feature after information exchange is reshaped into a complex-valued feature map, and the amplitude of the complex-valued feature map is normalized to obtain a predicted phase.

[0073] In this embodiment, the phase feature after information exchange is reduced to 2 channels through a 1×1 convolution layer to form a complex-valued feature map. The two channels in the complex-valued feature map correspond to the real part and the imaginary part of the phase, respectively, so that the properties of complex numbers can be used to represent the phase information.

[0074] Next, the amplitude of the complex-valued feature map is normalized to 1. This normalization step ensures that the feature map only contains phase information and is not affected by amplitude changes. The normalized phase prediction result is recorded as the predicted phase.

[0075] S05, combining the predicted amplitude and the predicted phase by using a combination formula to obtain a predicted spectrogram, and performing noise reduction processing on the predicted spectrogram by using a noise reduction method to obtain an enhanced sound.

[0076] Specifically, the combination formula includes:

[0077]

[0078] Among them, A′ represents the predicted spectrogram, abs(A) represents the absolute value of the sound spectrogram, F represents the predicted amplitude, and D represents the predicted phase.

[0079] Specifically, the noise reduction method includes spectral subtraction. In order to obtain clearer speech, the spectral subtraction method will be applied to the predicted spectrogram for further enhancement. The spectral subtraction technique aims to reduce the residual noise in the prediction result to improve the quality of the speech signal and ultimately obtain a purer sound output.

[0080] In summary, a sound enhancement method has the following effects:

[0081] The steps of collecting sound signals and determining whether the sound signals are sound signals emitted by the user help to distinguish between speech and noise, and improve the accuracy of speech recognition or speech enhancement applications. The step of combining the predicted amplitude with the predicted phase using a combination formula to obtain a predicted spectrogram effectively enhances the wearer's voice, and then performing noise reduction processing on the predicted spectrogram using a noise reduction method to obtain an enhanced sound step, removes the residual noise in the enhanced sound, and further enhances the wearer's voice, thereby improving the user experience.

[0082] The embodiment of the present invention further provides an earphone, which includes a structural housing, a communication module, a main control module (such as a microcontroller unit MCU), a speaker, a microphone, an inner ear microphone, a vibration sensor, a memory, etc. It also includes the sound enhancement method mentioned in the above embodiment. Since the sound enhancement method has the advantage of enhancing the wearer's sound and thus improving the user's experience, the earphone in this embodiment also has the advantage of enhancing the wearer's sound and thus improving the user's experience.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A sound enhancement method, characterized in that: The method comprises: Collecting a sound signal, and determining whether the sound signal is a sound signal emitted by a user; If the sound signal is a sound signal emitted by a user, performing a short-time Fourier transform on the sound signal to obtain a sound spectrogram; Performing different convolution processing on the sound spectrogram to obtain amplitude features and phase features respectively, and using an exchange process to exchange information between the amplitude features and the phase features; Performing prediction processing on the amplitude feature after the information exchange to obtain a predicted amplitude, and performing prediction processing on the phase feature after the information exchange to obtain a predicted phase; The predicted amplitude and the predicted phase are combined by using a combination formula to obtain a predicted spectrogram, and the predicted spectrogram is subjected to noise reduction processing by using a noise reduction method to obtain an enhanced sound.

2. The sound enhancement method according to claim 1, characterized in that: The step of determining whether the sound signal is a sound signal issued by a user comprises: Acquire a bone conduction signal in the sound signal, and perform time-frequency transformation on the bone conduction signal to obtain a frequency domain signal; Calculating the time domain information of the sound signal based on the bone conduction signal, and determining whether the time domain information and the frequency domain signal are greater than their corresponding thresholds respectively; If so, the sound signal is a sound signal emitted by the user.

3. The sound enhancement method according to claim 1, characterized in that: The exchange process includes an extraction process and an exchange process; The extraction process includes: inputting the amplitude feature into a first extraction network to obtain a first extraction feature; The first extracted feature is processed by three convolution layers to obtain an amplitude convolution feature; The convolutional features are input into a first extraction network to obtain second extraction features.

4. The sound enhancement method according to claim 3, characterized in that: The exchange process includes: performing secondary convolution layer processing on the phase feature to obtain a phase convolution feature, and exchanging information between the phase convolution feature and the second extracted feature; The second extracted features after information exchange are subjected to the extraction process at least once more, and the phase convolution features after information exchange are subjected to the exchange process at least once more.

5. The sound enhancement method according to claim 1, characterized in that: The step of performing prediction processing on the amplitude feature after information exchange to obtain the predicted amplitude includes: The amplitude features after information exchange are reshaped into a one-dimensional feature map, and the one-dimensional feature map is sequentially input into a bidirectional long short-term memory network and three fully connected layers to obtain a predicted amplitude.

6. The sound enhancement method according to claim 1, characterized in that: The step of performing prediction processing on the phase feature after information exchange to obtain a predicted phase comprises: The phase feature after information exchange is reshaped into a complex-valued feature map, and the amplitude of the complex-valued feature map is normalized to obtain a predicted phase.

7. The sound enhancement method according to claim 1, characterized in that: The combination formula includes: Among them, A′ represents the predicted spectrogram, abs(A) represents the absolute value of the sound spectrogram, F represents the predicted amplitude, and D represents the predicted phase.

8. The sound enhancement method according to claim 1, characterized in that: The noise reduction method includes spectral subtraction.

9. A headset, characterized in that: The headset includes a memory, a processor, and a sound enhancement program stored in the memory and executable on the processor, wherein the sound enhancement program implements the sound enhancement method according to any one of claims 1 to 8 when executed by the processor.

Images