A sound processing method and device, earphone and storage medium

Abstract

The present disclosure relates to a sound processing method, device, earphone and storage medium. The sound processing method comprises: acquiring a first sound signal collected by a first microphone, the first microphone being arranged inside a sound playing device, the first sound signal comprising bone conduction sound and air conduction sound; acquiring a second sound signal collected by a second microphone, the second microphone being arranged on the surface of the sound playing device, the second sound signal comprising air conduction sound; performing timbre correction on the first sound signal to obtain a third sound signal, the timbral difference degree between the third sound signal and the second sound signal being less than a difference degree threshold; and processing based on the second sound signal and the third sound signal to obtain a target sound signal. Through the present disclosure, the quality of the external microphone signal can be improved, the unnatural listening experience can be improved, and the intelligibility of the voice in the call can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of audio signal processing, and more particularly to a sound processing method, apparatus, headphones, and storage medium. Background Technology

[0002] In recent years, true wireless stereo (TWS) earbuds have rapidly gained popularity due to their greater portability compared to traditional earbuds. One of the most common application scenarios for TWS earbuds is voice calls. Voice calls are used in various scenarios, including commuting on public transportation, at the market, and in public places. Therefore, call noise cancellation, as an important technology to assist voice calls, is widely used in TWS earbuds. The goal of call noise cancellation is to effectively reduce the impact of ambient noise during calls while ensuring the intelligibility of human voices, thereby improving the call experience.

[0003] With the increasing demands for sound quality and call quality in recent years, current call noise reduction algorithms not only require ensuring the intelligibility of speech, but also aim to guarantee a natural and comfortable subjective listening experience while reducing noise interference.

[0004] In related technologies, directly replacing the mid-low frequencies of the external microphone signal with the mid-low frequencies of the internal microphone can lead to a muffled or unnatural sound under loud noise, resulting in a poor user experience. Summary of the Invention

[0005] To overcome the problems existing in the related technologies, this disclosure provides a sound processing method, apparatus, headphones, and storage medium.

[0006] According to a first aspect of the present disclosure, a sound processing method is provided, the method comprising:

[0007] A first sound signal is acquired by a first microphone, which is located inside the sound playback device, and the first sound signal includes bone conduction sound and air conduction sound; a second sound signal is acquired by a second microphone, which is located on the surface of the sound playback device, and the second sound signal includes air conduction sound; the first sound signal is timbre corrected to obtain a third sound signal, wherein the timbre difference between the third sound signal and the second sound signal is less than a difference threshold.

[0008] The target sound signal is obtained by processing the second sound signal and the third sound signal.

[0009] In one embodiment, the step of timbre correction of the first sound signal to obtain a third sound signal includes: adaptively tracking the spectral amplitude of the second sound signal with the spectral amplitude of the first sound signal to obtain a fourth sound signal, wherein the difference in spectral amplitude between the second sound signal and the fourth sound signal is less than a threshold; and obtaining the third sound signal based on the fourth sound signal.

[0010] In one embodiment, the step of adaptively tracking the spectral amplitude of the first sound signal to obtain the spectral amplitude of the second sound signal to obtain the fourth sound signal includes: dividing the frequency domain signals corresponding to the first sound signal and the second sound signal into sub-bands to obtain the amplitude of the first sub-band and the amplitude of the second sub-band; using an adaptive filtering algorithm to adaptively track the amplitude of the first sub-band and the amplitude of the second sub-band to obtain the gain signal of the first sound signal; and multiplying the gain signal with the frequency domain signal of the first sound signal to obtain the fourth sound signal.

[0011] In one embodiment, obtaining the third sound signal based on the fourth sound signal includes: extracting the spectral envelope of the second sound signal and the spectral envelope of the fourth sound signal; mapping the spectral envelope of the fourth sound signal to the spectral envelope of the second sound signal, so as to combine the spectral information of the second sound signal in the spectral envelope of the fourth sound signal to obtain the third sound signal.

[0012] In one embodiment, mapping the spectral envelope of the fourth audio signal to the spectral envelope of the second audio signal includes: mapping the spectral envelope of the fourth audio signal to the spectral envelope of the second audio signal based on a deep neural network; or replacing the spectral envelope of the fourth audio signal with the spectral envelope of the second audio signal.

[0013] In one embodiment, before performing timbre correction on the first sound signal to obtain the third sound signal, the method further includes: determining that the noise intensity included in the second sound signal is greater than a threshold.

[0014] In one embodiment, obtaining a denoised audio signal based on the second audio signal and the third audio signal includes: replacing the corresponding frequency components in the second audio signal with the frequency components of the third audio signal to obtain the denoised audio signal.

[0015] According to a second aspect of the present disclosure, a sound processing apparatus is provided, comprising: an acquisition unit for acquiring a first sound signal collected by a first microphone, the first microphone being disposed inside a sound playback device, the first sound signal including bone conduction sound and air conduction sound; acquiring a second sound signal collected by a second microphone, the second microphone being disposed on the surface of the sound playback device, the second sound signal including air conduction sound; a correction unit for performing timbre correction on the first sound signal to obtain a third sound signal, the timbre difference between the third sound signal and the second sound signal being less than a difference threshold; and a determination unit for processing based on the second sound signal and the third sound signal to obtain a target sound signal. In one embodiment, the correction unit performs timbre correction on the first sound signal to obtain the third sound signal in the following manner: adaptively tracking the spectral amplitude of the first sound signal to the spectral amplitude of the second sound signal to obtain a fourth sound signal, the spectral amplitude difference between the second sound signal and the fourth sound signal being less than a threshold; and obtaining the third sound signal based on the fourth sound signal.

[0016] In one embodiment, the acquisition unit adaptively tracks the spectral amplitude of the first sound signal to the spectral amplitude of the second sound signal to obtain a fourth sound signal in the following manner: the frequency domain signals corresponding to the first sound signal and the second sound signal are divided into sub-bands to obtain a first sub-band amplitude and a second sub-band amplitude; an adaptive filtering algorithm is used to adaptively track the first sub-band amplitude to the second sub-band amplitude to obtain a gain signal of the first sound signal; the gain signal is multiplied by the frequency domain signal of the first sound signal to obtain the fourth sound signal.

[0017] In one embodiment, the acquisition unit obtains the third sound signal based on the fourth sound signal in the following manner: extracting the spectral envelope of the second sound signal and the spectral envelope of the fourth sound signal; mapping the spectral envelope of the fourth sound signal to the spectral envelope of the second sound signal, so as to combine the spectral information of the second sound signal in the spectral envelope of the fourth sound signal to obtain the third sound signal.

[0018] In one embodiment, the acquisition unit maps the spectral envelope of the fourth sound signal to the spectral envelope of the second sound signal in the following manner: based on a deep neural network, it maps the spectral envelope of the fourth sound signal to the spectral envelope of the second sound signal; or it replaces the spectral envelope of the fourth sound signal with the spectral envelope of the second sound signal.

[0019] In one embodiment, before performing timbre correction on the first sound signal to obtain the third sound signal, the correction unit is further configured to: determine that the noise intensity included in the second sound signal is greater than a threshold.

[0020] In one embodiment, the determining unit obtains the denoised audio signal based on the second audio signal and the third audio signal in the following manner: replacing the corresponding frequency components in the second audio signal with the frequency components of the third audio signal to obtain the denoised audio signal.

[0021] According to a third aspect of the present disclosure, an apparatus is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the sound processing method described in the first aspect or any embodiment of the first aspect.

[0022] According to a fourth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions that, when executed by a processor of a second device, enable the second device to perform the sound processing method described in the second aspect or any embodiment of the second aspect.

[0023] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: acquiring a first sound signal collected by the built-in microphone of a sound playback device and a second sound signal collected by an external microphone; obtaining a third sound signal based on timbre correction of the first sound signal; and replacing the corresponding frequency portion of the second sound signal with the corrected third sound signal to obtain a timbre-converted sound signal. This disclosure can effectively improve the signal-to-noise ratio of the second sound signal, making the timbre similar to the original, improving the clarity and intelligibility of the timbre, and alleviating the problem of muffled and unnatural sound during calls in noisy environments using sound devices.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0026] Figure 1 This is a flowchart illustrating a commonly used call noise reduction method according to an exemplary embodiment.

[0027] Figure 2 This is a flowchart illustrating a sound processing method according to an exemplary embodiment.

[0028] Figure 3This is a flowchart illustrating a method for determining a third sound signal according to an exemplary embodiment.

[0029] Figure 4 This is a flowchart illustrating a method for determining a fourth sound signal according to an exemplary embodiment.

[0030] Figure 5 This is a flowchart illustrating an adaptive amplitude mapping according to an exemplary embodiment.

[0031] Figure 6 This is a flowchart illustrating a method for acquiring a third sound signal according to an exemplary embodiment.

[0032] Figure 7 This is a flowchart illustrating an envelope mapping according to an exemplary embodiment.

[0033] Figure 8 This is a block diagram of a sound processing apparatus according to an exemplary embodiment.

[0034] Figure 9 This is a block diagram (general structure of a mobile terminal) of an earphone according to an exemplary embodiment.

[0035] Figure 10 This is a block diagram illustrating an earphone according to an exemplary embodiment. (General structure of a server). Detailed Implementation

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.

[0037] Call noise reduction algorithms can be traced back to the 1970s. The earliest call noise reduction algorithms mainly used filters and spectral subtraction to reduce the impact of noise on speech signals. With the rapid development of machine learning technology, deep learning methods have begun to be used to obtain better noise reduction effects.

[0038] In related technologies, common dual-microphone call noise reduction solutions for headphones include... Figure 1 As shown, Figure 1This is a flowchart illustrating a common call noise reduction method according to an exemplary embodiment. The internal microphone has a slightly higher signal-to-noise ratio (SNR) in noisy environments, while the external microphone typically collects air conduction signals, resulting in a lower SNR. The specific scheme is as follows: First, the external microphone signal is fed forward (FF) to determine if it is in a high-noise environment. If it is not in a high-noise environment, the external microphone signal FF is directly used for speech enhancement, thus obtaining the noise-reduced output signal. If it is in a high-noise environment, the internal microphone signal (Feedback, FB) is used to improve the signal quality of the external microphone. Because the high-frequency components of the internal microphone cannot be effectively collected, only the mid-to-low frequencies of the internal microphone are used to directly replace the mid-to-low frequency components of the external microphone signal, improving the quality of the external microphone signal, resulting in signal FF1. Then, speech enhancement is performed on FF1, effectively improving speech intelligibility in noisy environments.

[0039] TWS earbuds typically feature an external microphone (FF) that captures air conduction signals and a built-in microphone (FB) inside the earbud. The external microphone captures the sound emitted from the mouth, traveling through the air to reach it. In noisy environments, ambient noise is captured along with the target signal, resulting in low signal-to-noise ratios and poor speech intelligibility in noisy scenarios like subways and food streets. The built-in microphone captures signals from both bone conduction and air conduction pathways, resulting in significantly higher signal-to-noise ratios and better speech intelligibility compared to the external microphone. However, because it includes bone conduction, the internal microphone can sound muffled and unnatural. In high-noise environments, the internal microphone needs to be activated to improve speech intelligibility, but its muffled signal necessitates timbre conversion to make the internal microphone's timbre similar to the external microphone, thus improving the listening experience.

[0040] The above methods can improve speech intelligibility, but the method of improving the external microphone signal is relatively simple. Using headphones, different wearers and different headphone modes will cause the correspondence between the internal and external microphone signals to be different. Directly replacing the corresponding components will result in the sound sounding muffled or unnatural under loud noise, resulting in a poor user experience.

[0041] In view of this, this disclosure proposes a timbre conversion method. This disclosure improves the timbre of the internal microphone, enhances the signal quality of the external microphone, solves the problem of muffled voice, and makes TWS earphones provide a more natural call experience in noisy environments.

[0042] The following embodiments of this disclosure describe a scenario where noise reduction is performed during a phone call using a sound playback device in a noisy environment.

[0043] Figure 2 This is a flowchart illustrating a sound processing method according to an exemplary embodiment, see below. Figure 2 As shown, the sound processing method includes the following steps:

[0044] In step S11, a first sound signal is acquired by a first microphone. The first microphone is located inside the sound playback device. The first sound signal includes bone conduction sound and air conduction sound.

[0045] In this embodiment, the sound playback device can be a TWS earphone, and the first sound signal collected by the first microphone of the TWS earphone can be understood as an internal microphone signal. The first microphone is located inside the TWS earphone. The first sound signal is characterized by including bone conduction path transmission signals and air conduction path transmission signals; that is, the first sound signal includes bone conduction sound and air conduction sound.

[0046] In step S12, a second sound signal collected by a second microphone is acquired. The second microphone is disposed on the surface of the sound playback device, and the second sound signal includes air-conducted sound.

[0047] In this embodiment of the disclosure, the second sound signal acquired by the second microphone of the TWS earphone can be understood as an external microphone signal. The second microphone is disposed on the surface of the TWS earphone. The characteristic of the second sound signal is that it acquires the signal of sound emitted from the mouth that travels through the air to reach the external microphone; that is, the second sound signal includes air-conducted sound.

[0048] In step S13, the first sound signal is timbre corrected to obtain a third sound signal. The timbre difference between the third sound signal and the second sound signal is less than the difference threshold.

[0049] In this embodiment, the built-in microphone collects signals that include both bone conduction and air conduction signals. Therefore, the signal-to-noise ratio and speech intelligibility of the built-in signal are significantly higher than those of the external microphone. However, because the built-in microphone signal includes a bone conduction path, it can suffer from muffled speech and an unnatural sound. Therefore, in high-noise environments, the built-in microphone needs to be activated to improve speech intelligibility. Speech intelligibility is related to timbre, meaning timbre correction is necessary.

[0050] The signal-to-noise ratio (SNR) can be understood as the ratio of the intensity of the sound signal to the intensity of the noise signal. When the SNR is low, the noise is severe and seriously affects the sound quality.

[0051] In step S14, the target sound signal is obtained by processing the second and third sound signals.

[0052] In this embodiment of the disclosure, a processed audio signal can be obtained based on a second audio signal (i.e., an external microphone signal) and a third audio signal obtained by timbre correction of the first audio signal. This can be understood as obtaining an audio signal with timbre adjusted under high-noise conditions, i.e., a target audio signal. The target audio signal is used for subsequent noise reduction processing.

[0053] In this embodiment of the present disclosure, in a high-noise scenario, the first sound signal is corrected to obtain the third sound signal, and the processed signal is obtained based on the second sound signal and the third sound signal, which can improve the signal matching degree and effectively improve the quality of the second sound signal collected by the external microphone.

[0054] The following describes the determination of the third sound signal according to the embodiments disclosed herein.

[0055] Figure 3 This is a flowchart illustrating a method for determining a third sound signal according to an exemplary embodiment, see also... Figure 3 As shown, the method for determining the third sound signal includes the following steps:

[0056] In step S21, the spectral amplitude of the first sound signal is adaptively tracked to track the spectral amplitude of the second sound signal to obtain the fourth sound signal, and the difference in spectral amplitude between the second sound signal and the fourth sound signal is less than a threshold.

[0057] In this embodiment, adaptive tracking can be understood as adaptive spectral amplitude mapping. Its main purpose is to ensure that the amplitude of the first sound signal collected by the built-in microphone corresponds to the amplitude of the second sound signal collected by the external microphone. This ensures that the frequency response is similar to that of the external microphone, thereby achieving similar timbre for the audio segment between the built-in and external microphones. The sound signal obtained based on adaptive spectral tracking can be understood as the fourth sound signal. The difference in spectral amplitude between the second sound signal collected by the external microphone and the fourth sound signal obtained based on adaptive spectral tracking is less than a threshold value, where the threshold value can be understood as maintaining similarity between the second and fourth sound signals within a certain amplitude range.

[0058] In step S22, a third sound signal is obtained based on the fourth sound signal.

[0059] In this embodiment of the disclosure, the third sound signal is obtained by using the fourth sound signal obtained by adaptive spectrum tracking, which can make the timbre similar to the second sound signal, making the second sound signal sound more natural in high noise scenarios.

[0060] The following describes the specific implementation process for determining the fourth sound signal in this embodiment.

[0061] Figure 4 This is a flowchart illustrating a method for determining a fourth sound signal according to an exemplary embodiment, see also... Figure 4 As shown, the method for determining the fourth sound signal includes the following steps:

[0062] In step S31, the frequency domain signals corresponding to the first sound signal and the second sound signal are divided into sub-bands to obtain the amplitude of the first sub-band and the amplitude of the second sub-band.

[0063] In this embodiment of the disclosure, see Figure 5 As shown, Figure 5 This is a flowchart illustrating an adaptive amplitude mapping according to an exemplary embodiment. First, a time-frequency conversion is performed on the first and second audio signals, which can be achieved using Fourier transform to convert the time-domain signal to the frequency domain. This facilitates subsequent analysis.

[0064] In this embodiment of the disclosure, after conversion to the frequency domain signal, the amplitude is obtained by sub-band division according to the frequency that conforms to auditory perception. Among them, bark sub-band division can be used. A bark sub-band can be understood as a unit of sound perception by the human ear. By dividing the bark sub-band, the human ear's perception of sound can be combined with the process of speech denoising, thereby improving the denoising effect.

[0065] The Bark subband partitioning method can be implemented using a partitioning formula, which is as follows:

[0066]

[0067] Where z is the corresponding bark subband value, f is the frequency, and acrtan is the arctangent function, this formula converts the frequency into a bark value, thereby dividing the signal into subbands and obtaining the corresponding subband amplitude. For example, taking a 16kHz sampling rate signal as an example, the maximum frequency of the signal is 8kHz, which will be divided into 18 bark subbands. The above formula uses an auditory perception frequency division method for subband division. It is understood that the above method in the embodiments of this disclosure is not a limitation of the embodiments of this disclosure, and other subband division methods can also be used in the embodiments of this disclosure.

[0068] In step S32, an adaptive filtering algorithm is used to adaptively track the amplitude of the second sub-band to obtain the gain signal of the first sound signal.

[0069] In this embodiment, after obtaining the amplitude values ​​of the first and second sub-bands based on the Bark sub-band division formula, an adaptive filtering algorithm is used to adaptively track the amplitude value of the second sub-band from the first sub-band amplitude. The calculation formula for the adaptive filtering algorithm is as follows:

[0070]

[0071] Where n is the frame index, W(n) is the gain value of the nth frame, W(n+1) is the gain value of the (n+1)th frame, x(n) is the amplitude of the built-in microphone signal, e(n) is the error estimate, i.e., the difference between the amplitude of the external microphone signal and x(n)*W(n), mu is the convergence factor, eps is a constant, and |x(n)| 2 Given a real number, the amplitude of the first sub-band is adaptively tracked by the amplitude of the second sub-band using the above formula to obtain the gain signal of the first sound signal.

[0072] In step S33, the gain signal is multiplied by the frequency domain signal of the first sound signal to obtain the fourth sound signal.

[0073] In this embodiment of the disclosure, in order to obtain the fourth sound signal, the gain value W(n) with built-in tracking can be inversely transformed back to a linear frequency domain signal through an interpolation algorithm. Finally, the frequency domain signal of the first sound signal is multiplied by the gain signal that has been inversely transformed back to a linear frequency domain signal by the gain signal to obtain the fourth sound signal.

[0074] In this embodiment of the disclosure, the frequency response of the first sound signal is adjusted to obtain the fourth sound signal, which effectively improves the timbre of the built-in microphone, making the overall listening experience of the fourth sound signal similar to that of the second sound signal, and the listening experience more natural.

[0075] The following describes the specific implementation process of the method for obtaining a third sound signal according to the embodiments of this disclosure.

[0076] Figure 6 This is a flowchart illustrating a method for acquiring a third audio signal according to an exemplary embodiment, see below. Figure 6 As shown, the method for obtaining a third sound signal includes the following steps:

[0077] In step S41, the spectral envelope of the second sound signal and the spectral envelope of the fourth sound signal are extracted.

[0078] In this embodiment of the disclosure, the spectral envelope describes the energy distribution of the audio signal at different frequencies and can be used to represent the timbre characteristics of the audio signal. See also... Figure 7 As shown, Figure 7 This is a flowchart illustrating an envelope mapping according to an exemplary embodiment. First, the spectral envelopes of the second and fourth audio signals are extracted. This can be done by first calculating the modulus of the frequency domain signals and then taking the logarithm, followed by a Discrete Cosine Transform (DCT). The DCT transformation formula is as follows:

[0079]

[0080] Where x[n] represents the nth sample value of the original signal, N (which can be set to N = 257) represents the number of sampling points of the original signal, k is the frequency, and C k It is a constant. cos represents the cosine function, π is pi, and X[k] is the amplitude after transformation. The amplitude after DCT transformation can be calculated using the above formula. After DCT transformation, low frequencies contain envelope information, while high frequencies represent details. For example, we select the first M dimensions (M=15) for inverse DCT transformation to obtain the spectral envelope information. The formula for inverse DCT transformation is:

[0081]

[0082] The meanings of the symbols in the formula are the same as in the DCT transform formula. The first M represents the features, and the frequency domain signal's spectral envelope can be obtained through the inverse transform described above. That is, the spectral envelopes of the second and fourth sound signals in this embodiment of the present disclosure.

[0083] In step S42, the spectral envelope of the fourth sound signal is mapped to the spectral envelope of the second sound signal, so as to combine the spectral information of the second sound signal in the spectral envelope of the fourth sound signal to obtain the third sound signal.

[0084] In this embodiment of the disclosure, the spectral envelope of the fourth sound signal is mapped to the spectral envelope of the second sound signal, and the spectral information corresponding to the fourth sound signal and the second sound signal is combined to obtain a third sound signal that matches the signal better.

[0085] In this embodiment of the disclosure, the first sound signal of the built-in microphone, namely the third sound signal, is further adjusted by spectral envelope mapping to make the third sound signal more similar to the second sound signal. Since the external microphone sounds more natural, the similarity of timbre is improved, which can further improve the subjective listening experience when the built-in microphone is enabled in high-noise scenarios, making the listening experience more natural.

[0086] The following describes the specific implementation process of mapping the spectral envelope of the fourth audio signal to the spectral envelope of the second audio signal in the embodiments of this disclosure.

[0087] Based on a deep neural network, the spectral envelope of the fourth sound signal is mapped to the spectral envelope of the second sound signal; or, the spectral envelope of the fourth sound signal is replaced with the spectral envelope of the second sound signal.

[0088] In this embodiment, after extracting the spectral envelopes of the second and fourth audio signals, the spectral envelope of the fourth audio signal is mapped to the spectral envelope of the second audio signal. A deep neural network method can be used; based on a deep neural network, the timbre of the built-in microphone can be adjusted to be similar to that of the external microphone. Alternatively, an envelope substitution method can be used to replace the spectral envelope of the fourth audio signal with the spectral envelope of the second audio signal. Then, the spectral envelope of the second audio signal is combined with the spectral details of the third audio signal, and an exponentiation operation is performed to obtain the timbre-adjusted audio signal output.

[0089] In this embodiment of the disclosure, envelope mapping or envelope replacement can adjust the timbre of the built-in microphone to be similar to that of the external microphone, increasing the signal matching degree, enabling adaptive switching of the built-in microphone signal and noise reduction in high-noise scenarios, ensuring the intelligibility of speech, and improving the problem of unnatural sound.

[0090] The following describes the method for correcting the timbre of the first audio signal to obtain the third audio signal, according to embodiments of this disclosure.

[0091] It is determined that the noise intensity included in the second sound signal is greater than the threshold.

[0092] In this embodiment, when the noise intensity is less than a threshold, the signal-to-noise ratio of the external microphone is relatively high, and the second audio signal is used directly; when the noise intensity is greater than the threshold, the low-frequency component of the built-in microphone with adjusted timbre is determined to be the third audio signal. The threshold can be predefined.

[0093] In this embodiment of the disclosure, when the noise intensity is greater than a threshold, the timbre of the first sound signal is corrected, which can avoid unnecessary repetitive operations.

[0094] The embodiments disclosed herein will be described below in which the frequency components of the third audio signal replace the corresponding frequency components in the second audio signal.

[0095] Based on the second and third audio signals, the noise-reduced audio signal is obtained, including:

[0096] The noise-reduced audio signal is obtained by replacing the corresponding frequency components in the second audio signal with the frequency components of the third audio signal.

[0097] In this embodiment of the disclosure, after determining that the noise intensity is greater than a threshold, the first sound signal is timbre corrected to obtain a third sound signal. Since the second sound signal is closer to the human voice, the frequency components of the third sound signal can be replaced with the corresponding frequency components in the second sound signal to obtain a noise-reduced sound signal, which has a higher similarity to the original.

[0098] In this embodiment of the disclosure, replacing the frequency components of the third audio signal with the corresponding frequency components in the second audio signal can effectively improve the signal-to-noise ratio of the second audio signal. Furthermore, after the spectrum replacement, speech enhancement can be performed to further improve the clarity and intelligibility of the speech signal by reducing the noise components in the speech signal.

[0099] Based on the same concept, embodiments of this disclosure also provide a sound processing apparatus.

[0100] It is understood that the sound processing apparatus provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0101] Figure 8 This is a block diagram illustrating a sound processing apparatus according to an exemplary embodiment. (Refer to...) Figure 8 The device 100 includes an acquisition unit 101, a correction unit 102, and a determination unit 103.

[0102] The acquisition unit 101 is used to acquire a first sound signal collected by a first microphone, the first microphone being disposed inside the sound playback device, the first sound signal including bone conduction sound and air conduction sound; and to acquire a second sound signal collected by a second microphone, the second microphone being disposed on the surface of the sound playback device, the second sound signal including air conduction sound.

[0103] The correction unit 102 is used to correct the timbre of the first sound signal to obtain a third sound signal, wherein the timbre difference between the third sound signal and the second sound signal is less than the difference threshold.

[0104] The determining unit 103 is used to process the second sound signal and the third sound signal to obtain the target sound signal.

[0105] In one embodiment, the correction unit 102 corrects the timbre of the first sound signal to obtain the third sound signal by adaptively tracking the spectral amplitude of the second sound signal to obtain the fourth sound signal, wherein the difference in spectral amplitude between the second sound signal and the fourth sound signal is less than a threshold; and the third sound signal is obtained based on the fourth sound signal.

[0106] In one embodiment, the acquisition unit 101 adaptively tracks the spectral amplitude of the first sound signal to the spectral amplitude of the second sound signal to obtain the fourth sound signal in the following manner: the frequency domain signals corresponding to the first sound signal and the second sound signal are divided into sub-bands to obtain the amplitude of the first sub-band and the amplitude of the second sub-band; an adaptive filtering algorithm is used to adaptively track the amplitude of the first sub-band to obtain the gain signal of the first sound signal; the gain signal is multiplied by the frequency domain signal of the first sound signal to obtain the fourth sound signal.

[0107] In one embodiment, the acquisition unit 101 obtains a third sound signal based on a fourth sound signal in the following manner: extracting the spectral envelope of a second sound signal and the spectral envelope of a fourth sound signal; mapping the spectral envelope of the fourth sound signal to the spectral envelope of the second sound signal, so as to combine the spectral information of the second sound signal in the spectral envelope of the fourth sound signal to obtain the third sound signal.

[0108] In one embodiment, the acquisition unit 101 maps the spectral envelope of the fourth sound signal to the spectral envelope of the second sound signal in the following manner: based on a deep neural network, the spectral envelope of the fourth sound signal is mapped to the spectral envelope of the second sound signal; or the spectral envelope of the fourth sound signal is replaced with the spectral envelope of the second sound signal.

[0109] In one embodiment, before performing timbre correction on the first sound signal to obtain the third sound signal, the correction unit 102 is further configured to: determine that the noise intensity included in the second sound signal is greater than a threshold.

[0110] In one embodiment, the determining unit 103 obtains a denoised audio signal based on a second audio signal and a third audio signal in the following manner: replacing the corresponding frequency components in the second audio signal with the frequency components of the third audio signal to obtain the denoised audio signal.

[0111] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0112] Figure 9 This is a block diagram illustrating an earphone 200 according to an exemplary embodiment. For example, the earphone 200 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0113] Reference Figure 9The headset 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0114] Processing component 202 typically controls the overall operation of headset 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0115] Memory 204 is configured to store various types of data to support operation of headset 200. Examples of this data include instructions for any application or method operating on headset 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0116] The power component 206 provides power to the various components of the headset 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the headset 200.

[0117] The multimedia component 208 includes a screen that provides an output interface between the headset 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the headset 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0118] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when headset 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0119] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0120] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of the headphones 200. For example, sensor assembly 214 may detect the on / off state of the headphones 200, the relative positioning of components such as the display and keypad of the headphones 200, changes in the position of the headphones 200 or one of its components, the presence or absence of user contact with the headphones 200, the orientation or acceleration / deceleration of the headphones 200, and temperature changes of the headphones 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0121] Communication component 216 is configured to facilitate wired or wireless communication between headset 200 and other devices. Headset 200 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0122] In an exemplary embodiment, the earphone 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0123] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by the processor 220 of the headset 200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0124] Figure 10 This is a block diagram illustrating a headset 300 according to an exemplary embodiment. For example, the headset 300 may be provided as a server. (Refer to...) Figure 10 The headset 300 includes a processing component 322, which further includes one or more processors, and memory resources represented by memory 332 for storing instructions, such as application programs, that can be executed by the processing component 322. The application programs stored in memory 332 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 322 is configured to execute instructions to perform the methods described above.

[0125] The headset 300 may also include a power supply component 326 configured to perform power management of the headset 300, a wired or wireless network interface 350 configured to connect the headset 300 to a network, and an input / output (I / O) interface 358. The headset 300 can operate on an operating system stored in memory 332, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0126] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0127] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0128] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0129] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0130] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0131] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A sound processing method, characterized in that, include: Acquire a first sound signal collected by a first microphone, wherein the first microphone is disposed inside a sound playback device, and the first sound signal includes bone conduction sound and air conduction sound; Acquire a second sound signal collected by a second microphone, the second microphone being disposed on the surface of the sound playback device, the second sound signal including air-conducted sound; The first sound signal is timbre corrected to obtain a third sound signal, wherein the timbre difference between the third sound signal and the second sound signal is less than the difference threshold. The target sound signal is obtained by processing the second sound signal and the third sound signal. The step of correcting the timbre of the first sound signal to obtain the third sound signal includes: The frequency domain signals corresponding to the first sound signal and the second sound signal are divided into sub-bands to obtain the amplitude of the first sub-band and the amplitude of the second sub-band. An adaptive filtering algorithm is used to adaptively track the amplitude of the second sub-band to obtain the gain signal of the first sound signal; The gain signal is multiplied by the frequency domain signal of the first sound signal to obtain the fourth sound signal, and the difference in spectral amplitude between the second sound signal and the fourth sound signal is less than the first threshold. The third sound signal is obtained based on the fourth sound signal.

2. The method according to claim 1, characterized in that, The process of obtaining the third sound signal based on the fourth sound signal includes: Extract the spectral envelope of the second sound signal and the spectral envelope of the fourth sound signal; The spectral envelope of the fourth sound signal is mapped to the spectral envelope of the second sound signal, and the spectral information of the second sound signal is combined in the spectral envelope of the fourth sound signal to obtain the third sound signal.

3. The method according to claim 2, characterized in that, Mapping the spectral envelope of the fourth audio signal to the spectral envelope of the second audio signal includes: Based on a deep neural network, the spectral envelope of the fourth sound signal is mapped to the spectral envelope of the second sound signal; or Replace the spectral envelope of the fourth sound signal with the spectral envelope of the second sound signal.

4. The method according to claim 1, characterized in that, Before performing timbre correction on the first sound signal to obtain the third sound signal, the method further includes: It is determined that the noise intensity included in the second sound signal is greater than the second threshold.

5. The method according to claim 1 or 4, characterized in that, The process of processing the second and third sound signals to obtain the target sound signal includes: The target sound signal is obtained by replacing the corresponding frequency components in the second sound signal with the frequency components of the third sound signal.

6. A sound processing device, characterized in that, include: An acquisition unit is configured to acquire a first sound signal collected by a first microphone, the first microphone being disposed inside a sound playback device, the first sound signal including bone conduction sound and air conduction sound, and to acquire a second sound signal collected by a second microphone, the second microphone being disposed on the surface of the sound playback device, the second sound signal including air conduction sound; The correction unit is used to correct the timbre of the first sound signal to obtain a third sound signal, wherein the timbre difference between the third sound signal and the second sound signal is less than the difference threshold. The determining unit processes the second and third sound signals to obtain the target sound signal; The correction unit corrects the timbre of the first sound signal to obtain the third sound signal in the following manner: The frequency domain signals corresponding to the first sound signal and the second sound signal are divided into sub-bands to obtain the amplitude of the first sub-band and the amplitude of the second sub-band. An adaptive filtering algorithm is used to adaptively track the amplitude of the second sub-band to obtain the gain signal of the first sound signal; The gain signal is multiplied by the frequency domain signal of the first sound signal to obtain the fourth sound signal, and the difference in spectral amplitude between the second sound signal and the fourth sound signal is less than the first threshold. The third sound signal is obtained based on the fourth sound signal.

7. The apparatus according to claim 6, characterized in that, The correction unit obtains the third sound signal based on the fourth sound signal in the following manner: Extract the spectral envelope of the second sound signal and the spectral envelope of the fourth sound signal; The spectral envelope of the fourth sound signal is mapped to the spectral envelope of the second sound signal, and the spectral information of the second sound signal is combined in the spectral envelope of the fourth sound signal to obtain the third sound signal.

8. The apparatus according to claim 7, characterized in that, The correction unit maps the spectral envelope of the fourth audio signal to the spectral envelope of the second audio signal in the following manner: Based on a deep neural network, the spectral envelope of the fourth sound signal is mapped to the spectral envelope of the second sound signal; or Replace the spectral envelope of the fourth sound signal with the spectral envelope of the second sound signal.

9. The apparatus according to claim 6, characterized in that, The correction unit is also used for: Before performing timbre correction on the first sound signal to obtain the third sound signal, it is determined that the noise intensity included in the second sound signal is greater than a second threshold.

10. The apparatus according to claim 6 or 9, characterized in that, The determining unit processes the second and third sound signals in the following manner to obtain the target sound signal: The target sound signal is obtained by replacing the corresponding frequency components in the second sound signal with the frequency components of the third sound signal.

11. An earphone, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to execute the sound processing method according to any one of claims 1-5.

12. A storage medium, characterized in that, The storage medium stores instructions that, when executed by the terminal's processor, enable the terminal to perform the sound processing method according to any one of claims 1-5.

Images