Audio processing method, electronic device, chip system, storage medium and program product

Abstract

The embodiment of the invention provides an audio processing method, electronic equipment, a chip system, a storage medium and a program product, and relates to the technical field of terminals. The method comprises the steps that based on a first operation, a first audio is acquired, the first audio comprises a first tooth sound signal and a second tooth sound signal, the amplitude of the first tooth sound signal is consistent with the amplitude of the second tooth sound signal, and the first sharpness of the first tooth sound signal is smaller than the second sharpness of the second tooth sound signal; and playing a second audio, the second audio being the first audio after tooth sound suppression, the second audio comprising a third tooth sound signal and a fourth tooth sound signal, the third tooth sound signal being the processed first tooth sound signal, and the fourth tooth sound signal being the processed second tooth sound signal, a first difference value between the amplitude of the first tooth sound signal and the amplitude of the third tooth sound signal is smaller than a second difference value between the amplitude of the second tooth sound signal and the amplitude of the fourth tooth sound signal. In this way, the damage of tooth sound suppression to the timbre of the audio data is small, and the user experience can be improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an audio processing method, electronic device, chip system, storage medium and program product. Background Art

[0002] Sibilance is the sound produced when airflow and teeth rub against each other when people speak. Its energy is mainly concentrated in 4kHz-12kHz, which is in the frequency band that human ears are more sensitive to, and it is easy to cause a bad subjective hearing experience for users. Therefore, before electronic devices play audio data, they will perform sibilance recognition and sibilance suppression on audio data. Exemplarily, after the electronic device recognizes one or more frames of sibilance in the audio data, it can suppress sibilance through an equalizer (EQ) or a multiband dynamic range compressor (MBDRC).

[0003] However, such a method may cause timbre damage, making the audio timbre distorted after sibilance suppression. Summary of the invention

[0004] The embodiments of the present application provide an audio processing method, an electronic device, a chip system, a storage medium, and a program product, which are applied to the field of terminal technology. The electronic device can suppress based on the sharpness distribution and the sound pressure level distribution corresponding to the frequency band, and the timbre of the audio data is less damaged during the sibilance suppression process, thereby improving the user experience.

[0005] In a first aspect, an embodiment of the present application proposes an audio processing method, the method comprising: displaying a first interface of a first application, the first interface comprising a first control; based on a first operation of the user on the first control, obtaining a first audio, the first audio comprising a first sibilant sound signal and a second sibilant sound signal, the amplitude of the first sibilant sound signal is consistent with the amplitude of the second sibilant sound signal, and the first sharpness of the first sibilant sound signal is less than the second sharpness of the second sibilant sound signal; playing a second audio, the second audio is an audio obtained after sibilant sound suppression of the first audio, the second audio comprising a third sibilant sound signal and a fourth sibilant sound signal, the third sibilant sound signal is the processed first sibilant sound signal, the fourth sibilant sound signal is the processed second sibilant sound signal, and the first difference between the amplitude of the first sibilant sound signal and the amplitude of the third sibilant sound signal is less than the second difference between the amplitude of the second sibilant sound signal and the amplitude of the fourth sibilant sound signal.

[0006] Among them, the first application can be an application that can play audio, and the first control can be a control for triggering the playing of audio. The first application can be, for example, a music application, and the first control can be a control for playing music; the first application can also be a video application, and the first control can be a control for triggering the playing of video; the first application can also be a call application, and the first control can be a control for making or answering a call; or, the first application can also be any other application that can play audio, and the first control can be any other control that can trigger the playing of audio. The first operation can be, but is not limited to, a click operation, a sliding operation, etc. The first audio is any one or more frames of audio data, for example, the xth frame of audio data described below. The first sibilant signal can be a frame of sibilant sound, or it can be a partial audio signal in a frame of sibilant sound, such as the kth frequency band described below, that is, the first sibilant sound signal is an audio signal corresponding to a frequency band in the xth frame of audio data (sibilant sound). The second sibilant sound signal can be a frame of sibilant sound, or it can be a partial audio signal in a frame of sibilant sound, such as the wth frequency band described below, that is, the second sibilant sound signal is an audio signal corresponding to a frequency band in a frame of sibilant sound (the xth frame of audio data or other frame of sibilant sound).

[0007] The first sharpness may reflect the sharpness of the first sibilant signal heard by the user, and the second sharpness may reflect the sharpness of the second sibilant signal heard by the user. The amplitude of the first sibilant signal and the amplitude of the second sibilant signal being consistent in magnitude may be that the amplitude of the first sibilant signal is completely equal to the amplitude of the second sibilant signal, or the difference between the amplitude of the first sibilant signal and the amplitude of the second sibilant signal is less than a threshold.

[0008] The third sibilant signal is the first sibilant signal after sibilant suppression. The terminal device can perform sibilant on the first sibilant signal based on the gain corresponding to the first sibilant signal, and obtain the third sibilant signal. The greater the gain corresponding to the first sibilant signal, the greater the degree to which the first sibilant signal is suppressed, and the greater the difference between the amplitude of the first sibilant signal and the amplitude of the third sibilant signal. The fourth sibilant signal is the second sibilant signal after sibilant suppression. The terminal device can perform sibilant on the second sibilant signal based on the gain corresponding to the second sibilant signal, and obtain the fourth sibilant signal. The greater the gain corresponding to the second sibilant signal, the greater the degree to which the second sibilant signal is suppressed, and the greater the difference between the amplitude of the second sibilant signal and the amplitude of the fourth sibilant signal.

[0009] It can be understood that there is a positive correlation between the amplitude and the frequency band sound pressure level. The amplitude of the first sibilance signal is consistent with the amplitude of the second sibilance signal, and it can also be understood that the sum of the frequency band sound pressure level distribution corresponding to the first sibilance signal is consistent with the sum of the frequency band sound pressure level distribution corresponding to the second sibilance signal.

[0010] In the audio processing method of the present application, the terminal device suppresses the sibilant signal based on the sharpness and amplitude of the sibilant signal. For sibilant signals with similar amplitudes but different sharpness, the greater the sharpness of the sibilant signal, the greater the degree of suppression of the sibilant signal, so that the sharpness of the suppressed sibilant signal is significantly reduced; the smaller the sharpness of the sibilant signal, the smaller the degree of suppression of the sibilant signal. In this way, suppressing the sibilant signal based on the sharpness and amplitude can not only significantly reduce the sharpness of the sibilant signal with higher sharpness, but also reduce the degree of suppression of the sibilant signal with lower sharpness, so that the suppressed sibilant signal does not become dull, and the sibilant suppression has less damage to the timbre.

[0011] In combination with the first aspect, in certain implementations of the first aspect, the first sharpness is the sum of the sharpness distributions corresponding to the first sibilant sound signals, and the second sharpness is the sum of the sharpness distributions corresponding to the second sibilant sound signals.

[0012] Among them, the sharpness distribution corresponding to the first sibilant signal can be understood as the sharpness distribution of the Brak frequency band corresponding to the first sibilant signal, such as the sharpness of one or more Bark frequency bands corresponding to the kth frequency band below, and the first sharpness is, for example, S1 below; the sharpness distribution corresponding to the second sibilant signal can be understood as the sharpness distribution of the Brak frequency band corresponding to the second sibilant signal, such as the sharpness of one or more Bark frequency bands corresponding to the wth frequency band below, and the second sharpness is, for example, S2 below. The sum of the sharpness distributions corresponding to the first sibilant signal can be calculated, for example, by the part of formula 13 below that calculates the sum of the sharpness distributions; the sum of the sharpness distributions corresponding to the second sibilant signal can be calculated, for example, by the part of formula 13 below that calculates the sum of the sharpness distributions.

[0013] In this way, the Brak frequency band is based on the frequency scale of human auditory perception, so that the first sharpness of the first sibilant signal and the second sharpness of the second sibilant signal can more truly reflect the human hearing sense, thereby suppressing the sibilant signal that sounds sharper to the user.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the third sibilant sound signal is a signal obtained by processing the first sibilant sound signal based on the gain of the first sibilant sound signal, and the fourth sibilant sound signal is a signal obtained by processing the second sibilant sound signal based on the gain of the second sibilant sound signal, and the gain of the first sibilant sound signal is less than the gain of the second sibilant sound signal.

[0015] In this way, since the first sharpness of the first sibilant signal is smaller than the second sharpness of the second sibilant signal, the gain of the first sibilant signal is smaller than the gain of the second sibilant signal. That is, when the amplitudes are similar, the smaller the sharpness, the smaller the degree of suppression of the sibilant signal, so that the sibilant signal with a smaller sharpness is suppressed to a smaller extent or is not suppressed, and the timbre damage caused in the sibilant suppression process is smaller.

[0016] In combination with the first aspect, in certain implementations of the first aspect, the gain of the first sibilant sound signal is determined based on the first sharpness and the sum of the frequency band sound pressure level distribution corresponding to the first sibilant sound signal, and the gain of the second sibilant sound signal is determined based on the second sharpness and the sum of the frequency band sound pressure level distribution corresponding to the second sibilant sound signal, the sum of the frequency band sound pressure level distribution corresponding to the first sibilant sound signal is positively correlated with the amplitude of the first sibilant sound signal, and the sum of the frequency band sound pressure level distribution corresponding to the second sibilant sound signal is positively correlated with the amplitude of the second sibilant sound signal.

[0017] Among them, since the frequency band sound pressure level is a logarithmic measure of the amplitude, the sum of the frequency band sound pressure level distribution corresponding to the first sibilant signal is positively correlated with the amplitude of the first sibilant signal. It can also be understood that the larger the amplitude of the first sibilant signal, the larger the sum of the frequency band sound pressure level distribution corresponding to the first sibilant signal; and the sum of the frequency band sound pressure level distribution corresponding to the second sibilant signal is positively correlated with the amplitude of the second sibilant signal. It can also be understood that the larger the amplitude of the second sibilant signal, the larger the sum of the frequency band sound pressure level distribution corresponding to the second sibilant signal. The first sharpness of the first sibilant signal and the second sharpness of the second sibilant signal can be calculated according to the sharpness distribution summation function part included in formula 13 below. The sum of the frequency band sound pressure level distribution corresponding to the first sibilant signal and the sum of the frequency band sound pressure level distribution corresponding to the second sibilant signal can be calculated according to the frequency band sound pressure level distribution summation function part included in Formula 14 below: Perform calculations.

[0018] In this way, the sum of the frequency band sound pressure level distribution can reflect the amplitude or energy of the sibilant signal, and the frequency band sound pressure level can be understood as a logarithmic measure of the amplitude. This makes the value of the suppression sharpness calculated based on the sum of the frequency band sound pressure level distribution relatively small, thereby facilitating the setting of the first threshold for determining the gain. The first threshold and the suppression sharpness can be described below.

[0019] In combination with the first aspect, in certain implementations of the first aspect, the first sibilant sound signal belongs to the c1th frame audio data, the second sibilant sound signal belongs to the c2th frame audio data, the c1th frame audio data and the c2th frame audio data are sibilant sounds, the c1th frame audio data is identified by the terminal device based on the sharpness distribution corresponding to the c1th frame audio data and the spectral centroid of the c1th frame audio data, and the c2th frame audio data is identified by the terminal device based on the sharpness distribution corresponding to the c2th frame audio data and the spectral centroid of the c2th frame audio data.

[0020] The c1-th frame audio data and the c2-th frame audio data may be the same frame of sibilance (c1 and c2 are the same), or different frames of sibilance (c1 and c2 are different). For example, if the c1-th frame audio data and the c2-th frame audio data are different frames of sibilance, the first sibilance signal may be the k-th frequency band in the c1-th frame audio data, and the second sibilance signal may be the k-th frequency band in the c2-th frame audio data, that is, the frequency range of the first sibilance signal and the second sibilance signal is the same.

[0021] The sharpness distribution corresponding to the c1-th frame audio data and the sharpness distribution corresponding to the c2-th frame audio data may both be a set of sharpnesses including one or more Bark frequency bands.

[0022] In this way, the sharpness distribution can reflect the sharpness of the audio data, and the spectrum centroid can reflect the center position of the energy distribution in the spectrum, so that the terminal device can identify the sibilant sound with a higher energy distribution at a high frequency and a higher sharpness. It is not easy to identify the audio data with a higher energy distribution at a low frequency as sibilant sound, which makes the accuracy of sibilant sound recognition higher.

[0023] In combination with the first aspect, in certain implementations of the first aspect, the product of the sum of the sharpness distribution corresponding to the audio data of the c1th frame and the normalized spectral centroid of the audio data of the c1th frame is greater than a fourth threshold, and the product of the sum of the sharpness distribution corresponding to the audio data of the c2th frame and the normalized spectral centroid of the audio data of the c2th frame is greater than the fourth threshold.

[0024] Among them, the product of the sum of the sharpness distribution corresponding to the c1-th frame audio data and the normalized spectrum centroid of the c1-th frame audio data and the product of the sum of the sharpness distribution corresponding to the c2-th frame audio data and the normalized spectrum centroid of the c2-th frame audio data can be understood as the recognition sharpness below, and can be calculated according to Formula 9. The fourth threshold can be Threshold 1 below.

[0025] In this way, the value of the normalized spectrum centroid can be distributed within a relatively small range, such as between 0 and 1, making it easy to determine the fourth threshold; and the normalized spectrum centroid can reduce the impact of different sampling rates and frequency ranges between different audio signals, so that the recognition sharpness can more accurately reflect the center position of the energy distribution in the spectrum, which helps to improve the accuracy of dental sound recognition.

[0026] In combination with the first aspect, in certain implementations of the first aspect, the sharpness distribution corresponding to the c1th frame audio data and the sharpness distribution corresponding to the c2th frame audio data respectively include the sharpness of multiple Bark frequency bands, and the multiple Bark frequency bands are Bark frequency bands whose frequencies are greater than or equal to a preset frequency.

[0027] The multiple Bark frequency bands may be the multiple specific Bark frequency bands described below.

[0028] In this way, the sharpness distribution corresponding to the c1th frame audio data and the sharpness distribution corresponding to the c2th frame audio data can reflect the sharpness of the high-frequency audio data, which is convenient for more accurate recognition of sibilance.

[0029] In the second aspect, the embodiment of the present application proposes an audio processing method. The method includes: determining that the x-th frame of audio data in multiple frames of audio data is sibilant; dividing the preset frequency range of the x-th frame of audio data into y frequency bands, the preset frequency range includes the sibilant band, or the preset frequency range partially overlaps with the sibilant band, and y is a positive integer; based on the sharpness distribution and band sound pressure level distribution corresponding to each frequency band in the y frequency bands, calculating the gain of each frequency band in the y frequency bands, the sharpness distribution includes the sharpness of at least one frequency band, and the band sound pressure level distribution includes the band sound pressure level of at least one frequency band; based on the gain of each frequency band in the y frequency bands, suppressing the y frequency bands to obtain audio data after sibilant suppression.

[0030] Among them, the sharpness distribution corresponding to each of the y frequency bands may include: the sharpness of at least one frequency band corresponding to each frequency band; the band sound pressure level distribution corresponding to each of the y frequency bands includes: the band sound pressure level of at least one frequency band corresponding to each frequency band.

[0031] In the audio processing method of the present application, the terminal device can perform frequency-band sibilance suppression on the preset frequency units of sibilance. This fine-grained sibilance suppression can reduce the damage to the timbre. In addition, the terminal device calculates the gain corresponding to each frequency band based on the sharpness distribution and the frequency band sound pressure level distribution, wherein the sharpness distribution can reflect the sharpness of the frequency band, and the frequency band sound pressure level distribution can reflect the energy of the frequency band. In this way, the terminal device can suppress frequency bands with higher sharpness and greater energy. The probability of suppressing frequency bands with lower sharpness can be reduced, thereby reducing the timbre damage caused during the sibilance suppression process.

[0032] In combination with the second aspect, in certain implementations of the second aspect, the gain of each frequency band in y frequency bands is calculated, including: calculating the sum of the first frequency band sound pressure level distributions corresponding to the k-th frequency band in the y frequency bands to obtain a first value; calculating the sum of the first sharpness distributions corresponding to the k-th frequency band in the y frequency bands to obtain a second value; calculating the suppression sharpness of the k-th frequency band based on the first value and the second value, the suppression sharpness of the k-th frequency band being used to indicate the magnitude of the first value and the second value; and determining the first gain of the k-th frequency band based on the suppression sharpness.

[0033] The first value can be The second value can be calculated by the following formula 13 The first gain may be the band gain g of the kth frequency band as follows: k .

[0034] In this way, the suppression sharpness of the kth frequency band can reflect the sharpness and energy of the kth frequency band. Determining the first gain based on the suppression sharpness can reduce the probability of suppressing frequency bands with lower sharpness, thereby reducing the timbre damage of sibilant suppression.

[0035] In combination with the second aspect, in certain implementations of the second aspect, a first gain of the kth frequency band is determined based on suppression sharpness, including: when the suppression sharpness is less than or equal to a first threshold, the first gain is 0; and / or, when the suppression sharpness is greater than the first threshold, the first gain is calculated based on the suppression sharpness, the first threshold and the second threshold, and the second threshold is used to represent the suppression capability.

[0036] The first threshold may be the threshold 4 mentioned below. The second threshold may be the threshold 5 mentioned below. The threshold 5 may also be referred to as suppression capability or suppression gain.

[0037] In this way, when the suppression sharpness is small, it means that the sharpness of the kth frequency band is small and / or the energy is low, and the terminal device may not suppress the kth frequency band, so that the timbre is less damaged.

[0038] In combination with the second aspect, in some implementations of the second aspect, the first gain g k Satisfies the following formula:

[0039]

[0040] Among them, sharp k To suppress sharpness, thres sharpnes is the first threshold, progain i is the second threshold.

[0041] In this way, the first gain increases with the increase of the sharpness of the suppression of the kth frequency band, so that the greater the sharpness of the kth frequency band, the greater the amplitude (the sum of the sound pressure level distribution of the frequency band), the greater the sharpness of the suppression of the kth frequency band, the greater the first gain, and the greater the degree of suppression of the kth frequency band by the terminal device based on the first gain, thereby reducing the sharpness of the kth frequency band. Conversely, if the sharpness of the suppression of the kth frequency band is small, the smaller the first gain, the smaller the degree of suppression of the kth frequency band by the terminal device based on the first gain, so that the hearing of the suppressed kth frequency band will not become dull.

[0042] In addition, the second threshold is a preset value that can reflect the degree of suppression or suppression ability, which can also be called suppression gain. When the suppression sharpness is constant, the larger the second threshold, the larger the gain calculated by the terminal device, and the greater the suppression ability for the frequency band; the smaller the second threshold, the smaller the gain calculated by the terminal device, and the smaller the suppression ability for the frequency band. Therefore, the second threshold can be a value determined based on experience or experiments, etc., which enables the terminal device to reasonably suppress sibilance.

[0043] In combination with the second aspect, in certain implementations of the second aspect, the suppression sharpness of the kth frequency band is calculated based on the first value and the second value, including: calculating the ratio of the first value to a third threshold value to obtain a third value; based on a preset function, converting the third value to a fourth value, the fourth value being less than or equal to the third value; calculating the product of the fourth value and the second value, the suppression sharpness being the product of the fourth value and the second value.

[0044] The third threshold may be the threshold 3 mentioned below. The third value may be p' mentioned below. The fourth value may be f(p') mentioned below.

[0045] In this way, the larger first value can be converted into a smaller fourth value, which makes it easier to control the suppression sharpness of the frequency band within a smaller fluctuation range, thereby facilitating the setting of the first threshold.

[0046] In combination with the second aspect, in certain implementations of the second aspect, obtaining the first value includes: dividing the amplitude spectrum of the x-th frame audio data into E frequency bands, where E is a positive integer; calculating the second frequency band sound pressure level distribution of the x-th frame audio data, the second frequency band sound pressure level distribution including the frequency band sound pressure level of each frequency band in the E frequency bands; and calculating the sum of the first frequency band sound pressure level distributions based on the second frequency band sound pressure level distribution, the first frequency band sound pressure level distribution including the frequency band sound pressure level of at least one frequency band in the E frequency bands.

[0047] The E frequency bands may be the e1 frequency bands mentioned below; the second frequency band sound pressure level distribution may be the frequency band sound pressure level p below. x The E frequency bands may be divided based on 1 / 3 octave filtering, for example.

[0048] In combination with the second aspect, in certain implementations of the second aspect, in order from low to high frequency, at least one of the E frequency bands includes a first frequency band to a second frequency band, the first frequency band is a frequency band to which the lower limit frequency of the kth frequency band in the E frequency bands belongs, and the second frequency band is a frequency band to which the upper limit frequency of the kth frequency band in the E frequency bands belongs.

[0049] The first frequency band is, for example, one of the E frequency bands indicated by h=k_fl below; the second frequency band is, for example, one of the E frequency bands indicated by h=k_fh below. h=k_fl and h=k_fh may be the same or different.

[0050] In this way, the frequency band sound pressure level distribution corresponding to the kth frequency band can be determined based on the frequency band sound pressure level distribution of the E frequency bands, and then the sum of the first frequency band sound pressure level distributions can be calculated. The sum of the first frequency band sound pressure level distributions is related to the amplitude of the kth frequency band, and can reflect the amplitude or energy of the kth frequency band.

[0051] Exemplarily, the sum Psum of the sound pressure level distribution in the first frequency band satisfies the following formula:

[0052]

[0053] Among them, k_fl is the lower limit frequency of the kth frequency band, h=k_fl is the index of a frequency band among the E frequency bands to which the lower limit frequency belongs, k_fh is the upper limit frequency of the kth frequency band, h=k_fh is the index of a frequency band among the E frequency bands to which the upper limit frequency belongs, h is the index of the frequency band among the E frequency bands, p(h) is the band sound pressure level of the frequency band with index h in the second frequency band sound pressure level distribution, and the index of the frequency band among the E frequency bands increases as the frequency of the frequency band increases.

[0054] In combination with the second aspect, in certain implementations of the second aspect, obtaining the second value includes: calculating a second sharpness distribution of the x-th frame of audio data, the second sharpness distribution including the sharpness of each Bark frequency band; based on the second sharpness distribution, calculating the sum of the first sharpness distributions, the first sharpness distribution including the sharpness of at least one Bark frequency band.

[0055] The second sharpness distribution may be the Bark domain sharpness distribution described below. The first sharpness distribution belongs to the second sharpness distribution.

[0056] In this way, the first sharpness distribution is consistent with the perception characteristics of the human ear, and sibilance suppression based on the first sharpness distribution enables the user to have a better hearing experience.

[0057] Exemplarily, the sum Ssum of the first sharpness distribution satisfies the following formula:

[0058]

[0059] Among them, k_fl is the lower limit frequency of the kth frequency band, k_fh is the upper limit frequency of the kth frequency band, z is the index of the Bark frequency band, z=bark() is a function for determining the index of the Bark frequency band based on the lower limit frequency or the upper limit frequency, and sp(z) is the sharpness of the Bark frequency band with index z in the second sharpness distribution.

[0060] Wherein, z=bark() is similar to Formula 11 and Formula 12 below, and z may be the product of bark1 and 10 in Formula 12.

[0061] In this way, the sum of the first sharpness distributions can reflect the sharpness of the kth frequency band heard by the human ear, and the timbre damage of sibilance suppression based on the sum of the first sharpness distributions is relatively small.

[0062] In combination with the second aspect, in certain implementations of the second aspect, determining that the x-th frame of audio data in multiple frames of audio data is a sibilant sound includes: determining that the x-th frame of audio data is a sibilant sound based on the sum of the third sharpness distribution of the x-th frame of audio data and the spectral centroid of the x-th frame of audio data.

[0063] The sum of the third sharpness distribution may be the sum of the sharpness of a plurality of specific Bark frequency bands described below.

[0064] In this way, the sum of the third sharpness distribution can reflect the sharpness of the x-th frame audio data, and the spectral centroid of the x-th frame audio data can reflect the center position of the energy distribution of the x-th frame audio data. Therefore, the sharpness and energy distribution of the x-th frame audio data are combined to determine whether the x-th frame audio data is a sibilant. In this way, it can be identified that the audio data with energy distribution concentrated in high frequencies and high sharpness is a sibilant, so that the accuracy of sibilant recognition is higher.

[0065] In combination with the second aspect, in certain implementations of the second aspect, the third sharpness distribution includes the sharpness of each Bark frequency band in a plurality of Bark frequency bands, and the plurality of Bark frequency bands are Bark frequency bands having a frequency greater than or equal to a preset frequency.

[0066] The Bark frequency band having a frequency greater than or equal to the preset frequency may be a plurality of specific Bark frequency bands described below.

[0067] Optionally, the multiple Bark frequency bands may also be: Bark frequency bands with an index greater than or equal to a preset index (bark). Since the index of a Bark frequency band increases with increasing frequency, the Bark frequency band with an index greater than or equal to the preset index may also be a Bark frequency band with a frequency higher than a certain threshold.

[0068] In this way, the third sharpness distribution can also reflect the sharpness of the high-frequency part in the x-th frame audio data.

[0069] In combination with the second aspect, in certain implementations of the second aspect, determining that the x-th frame of audio data is a sibilant sound includes: determining that the x-th frame of audio data is a sibilant sound based on the sum of the third sharpness distribution and the normalized spectrum centroid of the x-th frame of audio data.

[0070] In this way, the centroid of the spectrum of the x-th frame of audio data can be within a standardized range, and the influence of different sampling rates of the audio signal can be reduced, so that the accuracy of audio recognition is higher.

[0071] In combination with the second aspect, in certain implementations of the second aspect, determining that the x-th frame of audio data is a sibilant sound includes: calculating the product of the sum of the third sharpness distribution and the centroid of the normalized spectrum to obtain the recognition sharpness of the x-th frame of audio data; when the recognition sharpness is greater than a fourth threshold, determining that the x-th frame of audio data is a sibilant sound.

[0072] The fourth threshold may be the threshold 1 mentioned below.

[0073] In this way, the larger the sum of the third sharpness distribution and the normalized spectrum centroid are, the greater the recognition sharpness is. Based on the recognition sharpness, audio with high sharpness and the center of energy distribution at a high frequency can be identified as sibilant sound, which makes the accuracy of sibilant sound recognition higher.

[0074] In combination with the second aspect, in certain implementations of the second aspect, the method also includes: when the recognition sharpness is less than or equal to a fourth threshold, determining whether the x-1th frame of audio data is a dental sound; when the x-1th frame of audio data is a dental sound, determining whether the recognition sharpness is greater than a fifth threshold, and the fifth threshold is less than the fourth threshold; when the recognition sharpness is greater than the fifth threshold, determining that the x-1th frame of audio data is a dental sound.

[0075] The fifth threshold may be the threshold 2 mentioned below.

[0076] In this way, for multiple frames of continuous sibilants, during the stage when the sibilant amplitude decreases, sibilants with smaller amplitude (smaller recognition sharpness) can be identified, which helps to further improve the accuracy of sibilant recognition.

[0077] In combination with the second aspect, in certain implementations of the second aspect, the method further includes: when the audio data of the x-1th frame is not a sibilant, determining that the audio data of the xth frame is not a sibilant; or, when the recognition sharpness is less than or equal to a fifth threshold, determining that the audio data of the xth frame is not a sibilant.

[0078] If the x-1th frame audio data is not a sibilant, it means that the xth frame audio data is not a sibilant in the amplitude decreasing stage. If the recognition sharpness is less than the fourth threshold, it means that the xth frame audio data is not a sibilant. If the recognition sharpness is less than or equal to the fifth threshold, it means that the xth frame audio data has a smaller sharpness or a smaller high-frequency position energy, which means that the xth frame audio data is not a sibilant. In this way, non-sibilants can be accurately identified.

[0079] In conjunction with the second aspect, in some implementations of the second aspect, the method further includes:

[0080] The audio data after sibilance suppression is filtered by a filter, wherein a first parameter of the filter is determined based on a frequency response curve of the audio playback module, and the first parameter includes one or more of the following: center frequency, quality factor or gain.

[0081] The first parameter is the center frequency (FC), Q value, gain and other parameters of the filter processing module described below.

[0082] In this way, through the filtering processing of the filtering processing module, the amplification effect of the audio playback module such as the speaker on the sibilance band can be reduced, thereby helping to further improve the sound quality of the audio data and improve the user experience.

[0083] In a third aspect, an embodiment of the present application provides an audio processing device, which may be an electronic device or a chip or chip system in an electronic device. The audio processing device may include an audio playback unit and a processing unit. When the audio processing device is an electronic device, the audio playback unit may be a speaker, and the processing unit may be a processor. The audio processing device may also include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit so that the electronic device implements the audio processing method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect. When the audio processing device is a chip or chip system in an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit so that the electronic device implements the audio processing method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect. The storage unit may be a storage unit in the chip (for example, a register, a cache, etc.), or a storage unit located outside the chip in the electronic device (for example, a read-only memory, a random access memory, etc.).

[0084] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, the memory being used to store code instructions, and the processor being used to run the code instructions to execute the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0085] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is run on a computer, the computer executes the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0086] In a sixth aspect, an embodiment of the present application provides a computer program product comprising a computer program. When the computer program runs on a computer, the computer executes the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0087] In a seventh aspect, the present application provides a chip or a chip system, the chip or chip system comprising at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect. The communication interface in the chip can be an input / output interface, a pin, or a circuit, etc.

[0088] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0089] It should be understood that the third to seventh aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 The figure is a schematic diagram of an interface of an application scenario;

[0091] Figure 2A A schematic diagram of a time domain waveform and spectrum of audio data;

[0092] Figure 2B A time domain waveform diagram and a spectrum diagram of audio data;

[0093] Figure 3A A schematic diagram of a standard sharpness distribution and spectrum of audio data;

[0094] Figure 3B A time domain waveform diagram, a standard sharpness distribution diagram and a spectrum diagram of audio data;

[0095] Figure 4 A schematic block diagram of the hardware architecture of a terminal device provided in an embodiment of the present application;

[0096] Figure 5 A schematic block diagram of a software architecture of a terminal device provided in an embodiment of the present application;

[0097] Figure 6 A flowchart of an audio processing method provided in an embodiment of the present application;

[0098] Fig. 7A A schematic diagram of the standard sharpness distribution and the identified sharpness distribution of audio data provided in an embodiment of the present application;

[0099] Figure 7B A standard sharpness distribution diagram and a recognition sharpness distribution diagram of audio data provided in an embodiment of the present application;

[0100] Figure 8 A schematic diagram of a process for correcting and merging frequency band sound pressure levels provided in an embodiment of the present application;

[0101] Fig. 9 A schematic diagram of a process for determining loudness distribution in the Bark domain provided in an embodiment of the present application;

[0102] Fig.10 A schematic diagram of a frequency band characteristic loudness variation curve of audio data provided in an embodiment of the application;

[0103] Fig.11 A schematic diagram of a process flow of a dental sound recognition method provided in an embodiment of the present application;

[0104] Fig. 12A A schematic diagram of the original sequence spectrum and the spectrum after sibilance suppression provided in an embodiment of the present application;

[0105] Fig. 12B The original sequence spectrogram and the spectrogram after sibilance suppression provided in the embodiment of the present application;

[0106] Fig.13A A schematic diagram for illustrating a sibilance suppression part provided in an embodiment of the present application;

[0107] Fig. 13B A spectrum diagram for displaying the sibilance suppression part provided in an embodiment of the present application. DETAILED DESCRIPTION

[0108] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0109] 1. Frame

[0110] In the field of audio signal processing, a frame refers to the segmentation of continuous audio data into short time segments. Each frame usually contains a certain number of sampling points, which represent the amplitude changes of the audio signal in the time period. A frame can be audio data within a period of time, such as 10ms, 20ms or 30ms. This period of time can be called the frame length, that is, the time length of each frame.

[0111] 2. Sibilance

[0112] Refers to the sound produced when air flows through the narrow channel between the teeth and tongue during pronunciation, such as the pronunciation of initials such as "x", "s", "q" and "c".

[0113] 3. Sharpness

[0114] Sharpness usually refers to the harshness of a sound or the intensity of the high-frequency components in the sound spectrum. A high-sharpness sound will make people feel harsh and uncomfortable.

[0115] 4. Clarity

[0116] Sharpness usually refers to the brightness of a sound, and is usually related to the spectral energy distribution of the sound. A sound is perceived as brighter if it has more energy in the high frequency band.

[0117] 5. Frequency band energy

[0118] Refers to the energy contained in the audio signal within a specific frequency band (i.e. a certain range of frequencies), measured in dB.

[0119] For example, the audio signal can be decomposed into different frequency components by Fourier transform or other spectrum analysis methods. The frequency band energy is the integration or summation of the energy of these frequency components in a certain frequency band. For example, the audio signal can be decomposed into several frequency bands (such as 20 Hz to 100 Hz, 100 Hz to 200 Hz, etc.), and then the energy in each frequency band is calculated.

[0120] 6. Sound pressure level (SPL)

[0121] It is a logarithmic measure of sound pressure relative to a reference sound pressure.

[0122] The band sound pressure level is the sound pressure level calculated within a specific frequency range (i.e. frequency band).

[0123] 7. Frequency range

[0124] It refers to the amplitude value of the audio signal at a specific frequency.

[0125] The relationship between frequency amplitude and sound pressure level: The sound pressure level can be calculated based on the frequency amplitude. Specifically, the relationship between frequency amplitude and sound pressure level is logarithmic, that is, the sound pressure level is the logarithmic representation of the frequency amplitude.

[0126] 8. Bark domain (Bark scale)

[0127] It can also be called Bark spectrum, Bark scale or Bark frequency scale. It is a frequency scale based on human auditory perception. The Bark frequency scale is designed to better reflect the human ear's ability to perceive different frequencies. The traditional Hertz (Hz) frequency scale is linear, while the Bark frequency scale is nonlinear and more in line with human auditory characteristics.

[0128] Bark bands are discrete frequency intervals in the Bark domain. The entire auditory frequency range (approximately 20Hz to 20kHz) is usually divided into 24 Bark bands. Each Bark band corresponds to a specific frequency range. These Bark bands are divided based on the human auditory system's ability to resolve frequencies.

[0129] Furthermore, each of the 24 Bark frequency bands corresponds to one Bark value, which can also be understood as an index or sequence number of the Bark frequency band.

[0130] Exemplarily, the 24 Bark frequency bands and the Bark value corresponding to each Bark frequency band may be shown in Table 1. The frequency range of the Bark frequency band corresponding to the Bark value 1 is 20-100 Hz; the frequency range of the Bark frequency band corresponding to the Bark value 2 is 100-200 Hz, and so on.

[0131] Table 1

[0132]

[0133]

[0134] It should be noted that when the frequency ranges of two Bark frequency bands overlap, the overlapping frequency can belong to either of the two Bark frequency bands. For example, the frequency range of the Bark frequency band corresponding to Bark value 1 is 20-100Hz; the frequency range of the Bark frequency band corresponding to Bark value 2 is 100-200Hz, then the frequency 100Hz can belong to the Bark frequency band corresponding to Bark value 1, and can also belong to the Bark frequency band corresponding to Bark value 2. This application does not make specific restrictions on this.

[0135] 9. Equalizer (EQ)

[0136] It is an audio data processing tool used to adjust the gain of different frequency bands in audio data. It can enhance or reduce the volume of a specific frequency range to improve the sound quality or achieve a specific sound effect.

[0137] 10. Multiband dynamic range compressor (MBDRC)

[0138] It is an advanced dynamics processing tool that can compress different frequency bands of audio data separately.

[0139] 11. Threshold of hearing

[0140] It can also be called the hearing threshold, which refers to the minimum sound intensity that the human ear can hear in a very quiet environment. This minimum sound intensity varies with frequency and is usually expressed in a quiet hearing range. The quiet hearing range is a frequency-dependent curve called the threshold of hearing curve, which describes the minimum audible sound pressure level at different frequencies.

[0141] 12. Excitation level

[0142] Refers to the sound energy or loudness within a certain frequency band. It reflects the degree of stimulation of the sound within the frequency band to the human ear. The stimulation level is usually expressed in decibels (dB).

[0143] 13. Spectral centroid

[0144] It can be understood as the "center of mass" of the spectrum, that is, the central position of the energy distribution in the spectrum. The center of mass of the spectrum is obtained by calculating the weighted average of each frequency component of the spectrum.

[0145] 14. Normalized spectral centroid

[0146] The centroid of the spectrum is normalized so that its value is within a standardized range (usually between 0 and 1). This normalization process can eliminate the effects of different sampling rates and frequency ranges between different audio signals, making it easier to compare.

[0147] 15. Frequency response

[0148] It can also be called frequency response. It refers to the relationship between the output sound pressure level of an audio playback device (such as a speaker) at different frequencies and the frequency of the input audio signal. It is usually used to describe the performance of a speaker within the entire audio spectrum.

[0149] Frequency response is usually expressed in the form of a frequency response curve (or frequency response curve), where the horizontal axis (x-axis) represents frequency or frequency point (usually in Hertz) and the vertical axis (y-axis) represents sound pressure level (usually in decibels).

[0150] The frequency response curve shows the output sound pressure level of an audio playback device (such as a speaker, etc.) at different frequencies and can be used to evaluate the uniformity and accuracy of the audio playback device (such as a speaker, etc.) over the entire frequency range.

[0151] 16. Center frequency of the filter

[0152] It is the main frequency parameter of the filter, indicating the position where the filter reaches maximum gain or minimum attenuation on the frequency response curve.

[0153] 17. Quality factor (Q value) of the filter

[0154] Describes the frequency selectivity or bandwidth of the filter. The higher the Q value, the narrower the filter bandwidth and the more selective it is; the lower the Q value, the wider the bandwidth and the less selective it is.

[0155] 18. Filter Gain

[0156] Indicates the degree to which the filter amplifies or attenuates the audio signal at a specific frequency.

[0157] 19. Other terms

[0158] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0159] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0160] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0161] 20. Electronic devices

[0162] The electronic devices of the embodiments of the present application may include handheld devices, vehicle-mounted devices, etc. with face recognition functions. For example, some electronic devices are: mobile phones, tablet computers, PDAs, laptop computers, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, vehicle-mounted devices, wearable devices, terminal devices in 5G networks or future evolved public land mobile communication networks (PLANs), and wireless terminals in smart cities. The embodiments of the present application do not limit this.

[0163] As an example but not limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also powerful functions achieved through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include full-featured, large-sized, and fully or partially independent of smartphones, such as smart watches or smart glasses, as well as devices that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various types of smart bracelets and smart jewelry for vital sign monitoring.

[0164] In addition, in the embodiments of the present application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network that interconnects people and machines and things.

[0165] The electronic devices in the embodiments of the present application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0166] In an embodiment of the present application, the electronic device or each network device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux operating system, Unix operating system, Android operating system, iOS operating system, or Windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0167] With the development of the terminal technology field, the functions that terminal devices can provide are becoming more and more abundant. For example, users can listen to songs, record and play recordings, watch videos, play games, and make calls through terminal devices. However, there may be sibilance in the audio played by the terminal device, which may cause a bad subjective hearing experience for the user. Therefore, before the terminal device plays the audio data, the acquired audio data can be processed by sibilance suppression, etc., so that the sound effect of the played audio data is better, which can give the user a good audio and video experience.

[0168] For example, Figure 1 This is a schematic diagram of an interface of an application scenario provided by an embodiment of the present application. Figure 1 As shown in interface (a) in the figure, the interface (a) can be understood as a music playing interface of a music application. Interface (a) includes a control 101. In response to a user input operation on the control 101, such as clicking the control 101, the terminal device can switch from interface (a) to interface (b), and the terminal device starts playing the audio data of the song.

[0169] The audio data of the song played when the terminal device displays interface (b) is processed audio data obtained by processing such as sibilance suppression. That is, in response to the user's input operation on the control 101, the terminal device can obtain the audio data of the song. And the terminal device can perform processing such as sibilance suppression on the obtained audio data to obtain the processed audio data.

[0170] Dental sounds can be understood as part of the pronunciation of human voice. For example, when the initial consonants are "x", "s", "q" and "c", the tip of the tongue presses against the upper front teeth, and the friction between the airflow and the teeth produces dental sounds. Figure 2A and Figure 2B ,in Figure 2B It is a time domain waveform and spectrum diagram of a piece of audio data. Figure 2A For Figure 2B Schematic diagram of the time domain waveform and spectrum is shown.

[0171] in, Figure 2A The part circled in box 201 is Figure 2B The time domain waveform and frequency spectrum corresponding to the pronunciation of "x" circled in box 205 are shown. Figure 2A The part circled in box 202 is Figure 2B Schematic representation of the time domain waveform and frequency spectrum corresponding to the pronunciation of "c" circled in box 206. Figure 2A The part circled in box 203 is Figure 2B The time domain waveform and frequency spectrum corresponding to the pronunciation of "x" circled in box 207 are shown. Figure 2A The part circled in box 204 is Figure 2B Schematic representation of the time domain waveform and frequency spectrum corresponding to the "s" pronunciation circled in box 208.

[0172] Combination Figure 2A and Figure 2B In a spectrum diagram or spectrum schematic, the vertical axis is frequency, and the frequency increases evenly from bottom to top. Figure 2A In the spectrum diagram of , the lighter the color (white part), the higher the energy; Figure 2B In the spectrum diagram, the yellower the color, the higher the energy.

[0173] From the spectrum diagrams of the circled parts of boxes 201, 202, 203 and 204, and the spectrum of the circled parts of boxes 205, 206, 207 and 208, it can be seen that in the spectrum of "x", "c" and "s", the energy is higher at higher frequency positions, so the pronunciation of "x", "c" and "s" heard by the user is sharper.

[0174] In addition, a frame of audio data is usually a segment of tens of ms, such as 10ms, 20ms or 30ms, while a sibilant sound is usually a segment of 40-400ms. Therefore, the sibilant sound in a piece of audio data usually appears in the form of multiple consecutive frames. Figure 2A , the circled parts of box 201, box 202, box 203 and box 204 are all multi-frame audio data, which can include multi-frame dental sounds. In addition, combined with the schematic diagram of the time domain waveform of the multi-frame dental sounds circled in box 201, box 202, box 203 and box 204, and the time domain waveform of the multi-frame dental sounds circled in box 205, box 206, box 207 and box 208, it can be seen that the amplitude of the continuous multi-frame dental sounds usually changes in a slowly increasing trend first and then in a slowly decreasing trend.

[0175] exist Figure 2A A schematic diagram showing the spectrum with Figure 2B In the spectrum diagram shown, the spectrum of dental sounds (pronunciations of "x", "c", and "s") is compared with the spectrum of non-dental sounds (e.g., the uncircled portion). It can be seen that for dental sounds, the energy at higher frequency positions is higher. For non-dental sounds, the energy at higher frequency positions is lower.

[0176] Generally, the frequency range (frequency band) of sibilance is 4kHz-12kHz, which belongs to the mid-to-high frequency band and is in the frequency band that the human ear is more sensitive to. Therefore, when the terminal device plays audio, if the audio playback device (such as a speaker, etc.) in the terminal device amplifies the energy of the audio, the user will hear a high-pitched sound, which may lead to a poor user experience.

[0177] Therefore, in order to provide good sound quality, the terminal device can suppress the sibilance in the audio data before playing the audio data. Suppressing the sibilance in the audio data needs to be achieved through two processes, namely, sibilance recognition and sibilance suppression.

[0178] Taking the ath frame of audio data in a segment of audio data as an example, currently the terminal device can identify whether each frame of audio data in the segment of audio data is sibilant sound in the following manner.

[0179] S1. The terminal device calculates the Bark domain loudness distribution of the a-th frame of audio data.

[0180] The Bark domain loudness distribution can also be understood as the loudness of each Bark frequency band corresponding to the a-th frame audio data, that is, the characteristic loudness of each Bark frequency band. The Bark frequency bands can refer to the 24 Bark frequency bands shown in Table 1 above, for example. Each of the 24 Bark frequency bands can correspond to a loudness.

[0181] S2. The terminal device calculates the sum of the Bark domain loudness distribution of the a-th frame audio data, and obtains the total loudness N of the Bark domain loudness distribution of the a-th frame audio data. a .

[0182] For example, the total loudness N of the Bark domain loudness distribution a The following formula can be satisfied:

[0183] N a =∫1 24 n'(z)dz (Formula 1);

[0184] Wherein, z is the Bark value, that is, the index of the Bark frequency band; n'(z) is the loudness of the Bark frequency band with index z.

[0185] S3, based on Bark domain loudness distribution, total loudness N a , calculate the standard sharpness (S) of the a-th frame audio data.

[0186] Exemplarily, the standard sharpness (S) may satisfy the following formula:

[0187]

[0188] Among them, N a is the total loudness of the Bark domain loudness distribution; n'(z) is the loudness of the Bark frequency band with index z; g(z) is the excitation function, and the excitation function can satisfy the following formula 3.

[0189]

[0190] S4. The terminal device determines whether the standard sharpness is greater than or equal to a preset threshold value 1.

[0191] When the standard sharpness is greater than or equal to the preset threshold value 1, the terminal device determines that the audio data of the ath frame is sibilant sound;

[0192] When the standard sharpness is less than the preset threshold value 1, the terminal device determines that the a-th frame audio data is not sibilant.

[0193] Based on the above method, the terminal device can identify the sibilance in a section of audio data. Furthermore, the terminal device can suppress the sibilance through an equalizer (EQ) or a multiband dynamic range compressor (MBDRC).

[0194] However, processing audio data in the above manner may result in the following problems.

[0195] Question 1: The accuracy of dental sound recognition may be poor.

[0196] Combined with the above formula 2, the standard sharpness is the weighted sum of the loudness of each Bark frequency band of the a-th frame audio data. In this way, in the a-th frame audio data, when the sharpness of the lower-frequency Bark frequency band is large, the standard sharpness calculated by the terminal device may also be large. As a result, the terminal device determines that the a-th frame audio data is sibilant based on the standard sharpness being greater than the preset threshold 1. However, sibilant sounds generally refer to mid- and high-frequency audio data with greater energy. Therefore, this approach may result in poor accuracy in sibilant sound recognition.

[0197] In addition, when the sound source of the audio data is relatively complex, the accuracy of sibilant recognition may be poor based on the preset threshold 1. For example, in complex scenes such as music, the audio data contains a large number of musical instruments and vocals. When the loudness of low-frequency musical instruments is relatively large, the calculated standard sharpness corresponding to the musical instrument sound will also be relatively large. If sibilant recognition is performed based on a higher preset threshold 1, the situation where musical instrument sounds are recognized as sibilant sounds can be reduced, but it will cause some sibilant sounds in other audio (such as vocals) to be misidentified or incompletely recognized; if sibilant recognition is performed based on a lower preset threshold 1, more musical instrument sounds will also be recognized as sibilants because the standard sharpness corresponding to the musical instrument sounds is relatively large.

[0198] For example, in combination Figure 3A and Figure 3B . Figure 3B It shows the time domain waveform, standard sharpness distribution and spectrum of a piece of audio data. Figure 3A yes Figure 3B Schematic diagram of the standard sharpness distribution and the spectrum diagram in . The standard sharpness distribution can be understood as a sequence (or set) of standard sharpness of each frame of audio data in multiple frames of audio data.

[0199] like Figure 3A and Figure 3B As shown, the piece of audio data includes audio data 1 of a piano, audio data 2 of a mixed sound source (human voice and musical instrument sounds), and audio data 3 of a drum set.

[0200] exist Figure 3A In the spectrum diagram shown, the white part represents the part with higher energy. Figure 3B In the spectrum diagram shown, the closer to yellow, the higher the energy. That is, Figure 3A The white in the Figure 3B Schematic diagram of the higher energy part.

[0201] like Figure 3A and Figure 3BAs shown, combined with the frequency spectrum of audio data 1, it can be seen that the energy of audio data 1 is lower as a whole; combined with the frequency spectrum of audio data 2, it can be seen that in audio data 2, some audio data have higher energy and some audio data have lower energy; combined with the frequency spectrum of audio data 3, it can be seen that in audio data 3, the energy of mid- and low-frequency audio data is higher.

[0202] Based on this, the standard sharpness of each frame of audio data in audio data 1 is generally low; the standard sharpness of each frame of audio data in audio data 2 is uneven; and the standard sharpness of each frame of audio data in audio data 2 is generally high.

[0203] When the terminal device identifies the sibilance based on the preset threshold 1, assuming that the preset threshold 1 is relatively large, for example, x1, since the standard sharpness distribution of the audio data 1 is all smaller than x1, the terminal device cannot identify the sibilance in the audio data 1. Moreover, for the audio data 2, the terminal device may only be able to identify a part of the sibilance with a relatively high standard sharpness in the audio data 2, and may miss some sibilance with a relatively low standard sharpness.

[0204] That is, since the sibilance usually appears in multiple consecutive frames, and the amplitude of the multiple frames of continuous sibilance usually shows a trend of first slowly increasing and then slowly decreasing, the standard sharpness distribution of the multiple frames of continuous sibilance also shows a trend distribution from low to high and then from high to low. In this way, when the terminal device recognizes the sibilance of the audio data 2 based on the larger preset threshold 1 (x1), the sibilance recognized by the terminal device may be a few frames with larger standard sharpness among the multiple frames of continuous sibilance of the audio data 2, while the multiple frames of sibilance with smaller standard sharpness are not recognized. When the terminal device subsequently suppresses the sibilance, it also suppresses the few frames with larger standard sharpness among the multiple frames of continuous sibilance. The user may hear a sudden change in the audio data, resulting in a poor user experience.

[0205] In addition, assuming that the preset threshold 1 is relatively small, for example, x2, since the standard sharpness distribution of audio data 3 is mostly greater than x3, the terminal device may recognize most of the audio data in audio data 3 as sibilant sounds. The terminal device suppresses most of the audio data in audio data 3, resulting in timbre distortion of audio data 3, distorted drum sounds heard by the user, and poor user experience.

[0206] It can be seen that for Figure 3A and Figure 3B If the preset threshold 1 is high, the terminal device may not be able to recognize all the sibilants in the audio data 2; if the preset threshold 2 is low, the terminal device may recognize the sound of the instrument as sibilants. Therefore, the accuracy of sibilant recognition based on the preset threshold 1 is poor.

[0207] Question 2: Sibilance suppression may cause timbre distortion.

[0208] When the terminal device performs sibilance suppression through EQ or MBDRC, the measurement indicator is the energy of each frequency band (also known as amplitude, in dB).

[0209] However, for different frequency bands, the frequency ranges are different, and the sensitivity of human ear perception is different. Using amplitude as the judgment standard alone may suppress frequency bands with lower sensitivity to human ear perception, or suppress frequency bands with less sharpness, causing the audio tone heard by the user to be distorted and dull.

[0210] For example, it is assumed that the amplitudes of sibilant sound 1 and sibilant sound 2 are similar, but sibilant sound 2 is more sharp and sounds sharper; sibilant sound 1 is less sharp and does not sound sharp.

[0211] In this way, if the amplitude is used as the judgment basis only, the amplitudes of sibilance 1 and sibilance 2 are close, and the terminal device may suppress both sibilance 1 and sibilance 2. Since sibilance 2 is more sharp, it is no longer sharp after being suppressed, which makes the user's hearing better; however, since sibilance 1 is less sharp, the timbre of sibilance 1 may be damaged after being suppressed, making the suppressed sibilance 1 not clear or bright enough, which makes the user's hearing worse.

[0212] For example, for wind sound or sibilance in flat fricative sounds "v" and "f", the energy distribution of each frequency band is uniform, and usually, for users, wind sound or flat fricative sounds themselves are not sharp. If the terminal device suppresses sibilance in wind sound or flat fricative sounds based on amplitude, it will cause timbre damage, making the timbre of wind sound or flat fricative sounds heard by users distorted and dull.

[0213] In view of this, the present application provides an audio processing method. After a terminal device identifies at least one frame of sibilance, for each frame of sibilance, a preset frequency range (e.g., sibilance band 4kHz-12kHz) in each frame of sibilance can be divided into y frequency bands (y is an integer greater than 1). The terminal device can suppress sibilance for each frequency band based on the sharpness distribution and frequency band sound pressure level distribution corresponding to each frequency band in the y frequency bands.

[0214] Among them, the sharpness distribution can reflect the sharpness of the frequency band, and the frequency band sound pressure level distribution can reflect the energy level of the frequency band. In this way, by combining the sharpness distribution and the frequency band sound pressure level distribution, the terminal device can identify the frequency bands with higher sharpness and higher energy among the y frequency bands, and then suppress these frequency bands, so that the probability of the terminal device suppressing sibilance for the non-sharp frequency bands is lower, and thus the terminal device will cause less damage to the timbre of the audio data when suppressing sibilance.

[0215] Combine the following Figures 4 to 13BThe technical solution of the present application and how the technical solution of the present application solves the above technical problems are described in detail with specific embodiments. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0216] The embodiments shown in the present application may be executed by a terminal device, which may be the terminal device itself, or a chip, chip system or processor that supports the terminal device to implement the audio processing method, or a logic module or software that can implement all or part of the terminal device functions.

[0217] It should be understood that the specific form and quantity of each device shown in the embodiments of the present application are only examples and should not constitute any limitation on the implementation of the method provided in the present application.

[0218] In order to facilitate the understanding of this scheme, first combine Figure 4 The hardware structure of the terminal device 400 is described.

[0219] Figure 4 This is a schematic block diagram of the hardware architecture of the terminal device 400 provided in the embodiment of the present application. Figure 4 As shown, the terminal device 400 may include a processor 410, an external memory interface 420, an internal memory 421, a universal serial bus (USB) interface 430, a charging management module 440, a power management module 441, a battery 442, an antenna 1, an antenna 2, a mobile communication module 450, a wireless communication module 460, an audio module 470, a sensor module 480, a button 490, an indicator 492, a camera 493, and a display screen 494, etc.

[0220] Among them, the sensor module 480 can include but is not limited to one or more of the following sensors: pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor and bone conduction sensor, etc.

[0221] The audio module 470 may include: a speaker 470A, a receiver 470B, a microphone 470C, and an earphone interface 470D, etc. The speaker may be used to play audio data from the processor 410 .

[0222] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 400. In other embodiments of the present application, the terminal device 400 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0223] The processor 410 may include one or more processing units, for example, the processor 410 may include an application processor (AP), a modem processor (Modem), a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Optionally, a memory may be provided in the processor 410 for storing instructions and data. Different processing units may be independent devices or integrated in one or more processors.

[0224] Optionally, the processor 410 may obtain audio data, and perform sibilance recognition on the audio data to determine at least one frame of sibilance in the audio data; the processor 410 may also perform sibilance suppression on at least one frame of sibilance to obtain audio data after sibilance suppression; and the processor 410 may be pre-set with a peak filter, and the parameters corresponding to the peak filter (such as center frequency, Q value, and gain, etc.) may be determined based on the frequency response curve of the speaker 470A, and the processor 410 may filter the audio data after sibilance suppression through the peak filter to obtain filtered audio data. Afterwards, the processor 410 may transmit the filtered audio data to the audio module 470 so that the audio module 470 (such as the speaker 470A) plays the filtered audio data.

[0225] The wireless communication function of the terminal device 400 can be implemented through antenna 1, antenna 2, mobile communication module 450, wireless communication module 460, modem processor and baseband processor.

[0226] The terminal device 400 implements the display function through a GPU, a display screen 494, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 494 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 410 may include one or more GPUs, which execute program instructions to generate or change display information.

[0227] The display screen 494 is used to display images, videos, etc. In some embodiments, the terminal device 400 may include 1 or N display screens 494, where N is a positive integer greater than 1.

[0228] The external memory interface 420 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 400. The internal memory 421 may be used to store computer executable program codes, which include instructions.

[0229] Combine the following Figure 5 , taking the Android system as an example, the software architecture of the terminal device 400 is illustrated.

[0230] It should be understood that the software system of the terminal device 400 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture.

[0231] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: application layer, Java framework layer, system service layer, system runtime and system library, hardware abstraction layer, and kernel layer.

[0232] 1. Application layer

[0233] The application layer may include a series of application packages. For example, the application package may include a first application, through which the user can interact with the terminal device 400, and the terminal device 400 can play audio through the first application. The first application may be, but is not limited to, a call, a music application, a game application, a video application, a social application, etc. For example, if the first application is a music application, the terminal device 400 can play songs through the first application. The process can be referred to in the above description of Figure 1 Description.

[0234] During the process of playing audio through the first application, the first application can instruct the audio processing module in the FWK layer to obtain and process audio data.

[0235] 2. Java framework layer, also known as application framework (framework, FWK) layer

[0236] The Java framework layer can provide an application programming interface (API) and a programming framework for the application programs in the application layer. The Java framework layer includes some predefined functions. The Java framework layer can include, but is not limited to, an activity manager, an input manager, a phone manager, a window manager, and an audio processing module.

[0237] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0238] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0239] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0240] The phone manager is used to provide communication functions of the terminal device 400, such as management of call status (including connection, disconnection, etc.).

[0241] The audio processing module can be used to obtain audio data from the application layer or other layers, and can perform sibilance recognition, sibilance suppression and filtering on the audio data in sequence to obtain filtered audio data. The audio processing module is also used to write the filtered audio data into a buffer.

[0242] It should be understood that the audio processor module may be a software module or may include multiple software modules. For example, the audio processing module may include a sibilance recognition and suppression module and a filtering processing module.

[0243] Among them, the sibilance recognition and suppression module is used to perform sibilance recognition and sibilance suppression on audio data, and transmit the audio data after sibilance suppression to the filtering processing module.

[0244] The filtering processing module may also be called a peak filter, which is used to filter the audio data after sibilance suppression to obtain filtered audio data.

[0245] It should be noted that the audio processing module (or sibilance recognition and suppression module) can identify sibilance based on the sharpness distribution 1 and the normalized spectrum centroid of the audio data, and / or can suppress sibilance based on the sharpness distribution 2 and the frequency band sound pressure level distribution corresponding to each frequency band in the sibilance. For details, please refer to the description below.

[0246] 3. System services layer

[0247] The system service layer includes various system services, which interact with the application framework layer through the Binder IPC mechanism. The system service layer manages and coordinates system resources and provides core functions for applications.

[0248] The system service layer includes an audio mixer (audio flinger). The audio flinger can read the filtered audio data from the cache, and perform mixing, sound effects and other processing on the filtered audio data to obtain audio data a.

[0249] 4. Hardware Abstraction Layer

[0250] The hardware abstraction layer (HAL) provides hardware-related abstract interfaces so that upper-level system services and libraries can interact with different hardware.

[0251] HAL includes multiple library modules such as Bluetooth HAL, camera HAL, audio HAL and sensor HAL, among which each library module can provide an interface for the corresponding hardware component implementation.

[0252] The audio HAL can obtain the audio data a from the audio flinger. The audio HAL can further process the audio data a based on the hardware requirements for playing the audio data, such as converting the format of the audio data a, adjusting the sampling rate and volume control, etc., to obtain the audio data b. The audio HAL can transmit the audio data b to the hardware (speaker, receiver or headset, etc.) in the terminal device through the kernel layer, so that the hardware in the terminal device can play the audio.

[0253] 6. Kernel layer (Kernel), also known as Linux Kernel

[0254] The kernel layer is a layer between hardware and software. The kernel layer is used to drive the hardware to make the hardware work. The kernel layer includes display driver, screen driver, graphics processing unit (GPU) driver, camera, and audio driver, etc., which are not limited in the embodiments of the present application.

[0255] The audio driver can obtain the audio data b from the audio HAL, and can drive the digital-to-analog converter to play the audio data b. For example, the audio driver can convert the audio data b from an electrical signal (or a digital signal) to a sound signal (or an analog signal) through a digital-to-analog converter, and play the sound signal through an output device such as a speaker or a headset.

[0256] It should be understood that Figure 5 The layered architecture shown is only an example, and the operating system of the terminal device 400 may also include more or fewer layers. For example, the operating system also includes an Android Runtime and a system library, etc. And each layer may include more or fewer software modules, for example, the FWK layer may also include an activity manager, etc.

[0257] Combination Figure 5 The software architecture shown in the figure can Figure 6 The method 600 shown performs audio processing, which is described in detail below.

[0258] Figure 6 FIG. 6 is a flow chart of an audio processing method 600 provided in an embodiment of the present application. Figure 6 As shown, the method 600 includes the following steps:

[0259] S601. The sibilance recognition and suppression module obtains M frames of audio data.

[0260] Wherein, M is a positive integer. The M frames of audio data may be time domain signals.

[0261] S602: The sibilance recognition and suppression module converts the M frames of audio data from time domain signals to frequency domain signals.

[0262] The frequency domain signal of the M frames of audio data may also be understood as the frequency spectrum or amplitude spectrum of the M frames of audio data.

[0263] Exemplarily, the sibilance recognition and suppression module may convert the M frames of audio data from a time domain signal to a frequency domain signal by means of fast Fourier transform (FFT) or the like.

[0264] The sibilance recognition and suppression module can perform sibilance recognition on each frame of the M frames of audio data in turn. Below, taking the xth frame of audio data as an example, the sibilance recognition process (S603 to S608) of the sibilance recognition and suppression module is described in detail.

[0265] As x takes all integers between 1 and M, the sibilance recognition and suppression module can recognize sibilance in M ​​frames of audio data.

[0266] S603, the sibilance recognition and suppression module calculates the frequency band sound pressure level p of the x-th frame audio data (frequency domain signal) x .

[0267] Among them, the frequency band sound pressure level p x It can be understood as a sequence of multiple frequency band sound pressure levels. That is, the entire frequency band of the x-th frame audio data can be divided into e1 frequency bands, each of the e1 frequency bands can correspond to a frequency band sound pressure level, and there are e1 frequency band sound pressure levels in total. The e1 frequency band sound pressure levels constitute the frequency band sound pressure level p of the x-th frame audio data. x . e1 is a positive integer.

[0268] Exemplarily, the sibilance recognition and suppression module may filter the spectrum of the x-th frame audio data by 1 / 3 octave filtering to obtain e1 frequency bands, where e1 is, for example, 28. The width of each of the e1 frequency bands is 1 / 3 times the center frequency of the 1 / 3 octave filtering.

[0269] Taking the band sound pressure level of one of the e1 bands as an example, the band sound pressure level can be calculated using the following formula:

[0270]

[0271] Among them, p0 is the band sound pressure level of the frequency band; id1 is the lower limit frequency of the frequency band; id2 is the upper limit frequency of the frequency band; mag i is the frequency amplitude; p re1f The standard sound pressure may also be referred to as the reference sound pressure, for example, 20 microPascals (μPa).

[0272] It should be understood that for each of the e1 frequency bands, the frequency band sound pressure level can be calculated in the above manner, thereby obtaining the e1 frequency band sound pressure levels corresponding to the e1 frequency bands. For example, e1 can be 28, and through the above process, the sibilance recognition and suppression module can obtain the frequency band sound pressure level of each of the 28 frequency bands.

[0273] It should be noted that the frequency band division through 1 / 3 octave filtering shown above is only an example. The sibilant recognition and suppression module can also perform frequency band division through 1 / 1 octave filtering, etc., so that the divided frequency bands can more accurately describe the spectral characteristics of the noise.

[0274] S604, based on the frequency band sound pressure level p x The sibilance recognition and suppression module calculates the Bark domain loudness distribution of the x-th frame audio data.

[0275] The Bark domain loudness distribution may include the characteristic loudness of each frequency band in the Bark frequency band, that is, the Bark domain loudness distribution may be understood as a set or sequence including multiple characteristic loudnesses.

[0276] S605 . Based on the Bark domain loudness distribution of the x-th frame of audio data, the sibilance recognition and suppression module calculates sharpness distribution 1 of the x-th frame of audio data.

[0277] Among them, sharpness distribution 1 may include the sharpness of each of the 24 Bark frequency bands.

[0278] The sibilance recognition and suppression module can, for example, calculate the sharpness distribution 1 of the x-th frame audio data in the following manner:

[0279] First, the sibilance recognition and suppression module calculates the sum of the Bark domain loudness distribution of the x-th frame of audio data, that is, the total loudness (N sum ).

[0280] For example, the total loudness N of the Bark domain loudness distribution sum The following formula can be satisfied:

[0281]

[0282] Where n(z) is the characteristic loudness of the Bark frequency band with index z.

[0283] Afterwards, the sibilance recognition and suppression module calculates the sharpness distribution 1 based on the total loudness and the Bark domain loudness distribution.

[0284] For the x-th frame of audio data, the sharpness of the Bark band with index z in the Bark band can satisfy the following formula:

[0285]

[0286] Where sp(z) is the sharpness of the Bark band with index z, n(z) is the characteristic loudness of the Bark band with index z, and N sum is the total loudness, and g(z) is the activation function.

[0287] The activation function can satisfy the following formula:

[0288]

[0289] In this way, when z traverses from 1 to 24, the sibilance recognition and suppression module can determine the sharpness of each of the 24 Bark frequency bands, that is, the sharpness distribution 1 of the x-th frame audio data.

[0290] S606: The sibilance recognition and suppression module calculates the normalized spectrum centroid NSC of the x-th frame of audio data.

[0291] Exemplarily, the normalized spectrum centroid NSC of the x-th frame of audio data satisfies the following formula:

[0292]

[0293] Among them, n is the nth frequency point, N is the total number of frequency points included in the spectrum of the xth frame audio data, f(n) is the frequency corresponding to the nth frequency point, E(n) is the frequency energy of the nth frequency point, and fs is the sampling rate.

[0294] S607 . The sibilance recognition and suppression module calculates the recognition sharpness of the x-th frame of audio data based on the normalized spectrum centroid and sharpness distribution 1 of the x-th frame of audio data.

[0295] The recognition sharpness can be understood as the product of the normalized spectrum centroid and the sum of the sharpnesses of multiple specific Bark bands. The multiple specific Bark bands can be, for example, multiple Bark bands with higher frequencies, such as multiple Bark bands with z ranging from 17 to 24.

[0296] Exemplarily, when the multiple specific Bark frequency bands are multiple Bark frequency bands where z is 17 to 24, the recognition sharpness of the x-th frame audio data satisfies the following formula:

[0297]

[0298] Wherein, NSC is the normalized spectral centroid of the x-th frame of audio data, and sp(z) is the sharpness of the Bark band with index z.

[0299] It should be understood that in Formula 9, 17 may also be replaced by other values, such as 16 or 18, etc. This does not constitute a limitation on the embodiments of the present application.

[0300] By combining the sum of the sharpness distribution and the normalized spectrum centroid to calculate the recognition sharpness, and based on the recognition sharpness, the sibilance recognition is performed. In the audio recognition process, not only the sharpness of the audio data is considered, but also the frequency of the higher energy position in the audio data is considered, so that the terminal device can recognize the sibilance with high energy distribution in the middle and high frequency positions. This can improve the distinction between the spectral features of sibilance and non-sibilance, which is conducive to improving the recognition effect in complex scenes such as music and video.

[0301] S608: The sibilance recognition and suppression module determines whether the x-th frame of audio data is sibilance based on the recognition sharpness of the x-th frame of audio data.

[0302] Exemplarily, when the recognition sharpness of the x-th frame audio data is greater than threshold 1, it can be determined that the x-th frame audio data is sibilant; when the recognition sharpness of the x-th frame audio data is less than or equal to threshold 1, it can be determined that the x-th frame audio data is not sibilant.

[0303] Among them, threshold 1 can be a preset value.

[0304] It should be understood that since multiple specific Bark bands are high-frequency bands, the recognition sharpness of the x-th frame audio data can reflect the sharpness of the high-frequency bands; in addition, by normalizing the spectrum centroid, it can also reflect whether the position with higher energy in the x-th frame audio data has a higher frequency. And since the normalized spectrum centroid has been normalized, it can also eliminate the influence of different sampling rates and frequency ranges between different audio signals, making it more convenient to compare.

[0305] Therefore, by identifying the sibilance through recognition sharpness, the sharpness of the higher frequency band in the audio data and the frequency of the higher energy position are comprehensively considered. When the recognition sharpness of the x-th frame audio data is large, it means that the x-th frame audio data has a high sharpness and the frequency of the higher energy position is large, indicating that the probability that the x-th frame audio data is a sibilance is high, so it can be determined that the x-th frame audio data is a sibilance.

[0306] Compared with the standard sharpness for sibilance recognition, the recognition sharpness takes into account the sharpness of the higher frequency band and the frequency of the higher energy position, making the accuracy of sibilance recognition higher. In addition, since the normalized spectrum centroid has been normalized, the influence of different frequency ranges of different audio data can be eliminated, so that the recognition sharpness of different audio data can be distributed at a relatively more stable level, so that sibilance recognition based on recognition sharpness can be applied to various types of audio data, such as musical instrument sounds and human voices.

[0307] For example, in combination Fig. 7A and Figure 7B ,in Figure 7B The time domain waveform, standard sharpness distribution, identification sharpness distribution and spectrum diagram of a piece of audio data are shown. Figure 3B , Figure 7B Added identification of sharpness distribution. Fig. 7A For Figure 7B Schematic diagram of standard sharpness distribution and identified sharpness distribution is shown. Fig. 7A The standard sharpness distribution in Figure 3A The standard sharpness distribution shown is the same.

[0308] Combination Fig. 7A or Figure 7B, comparing the recognition sharpness distribution with the standard sharpness distribution, it can be seen that the recognition sharpness distribution of this segment of audio data is relatively more concentrated, making it easier to identify dental sounds based on a preset value (threshold 1). In addition, referring to the standard sharpness distribution and recognition sharpness distribution of audio data 1 and audio data 3, it can be seen that the recognition sharpness distribution of audio data 1 and audio data 3 is almost stable at around 0. In this way, when dental sounds are identified by recognition sharpness, the probability of identifying the audio data emitted by the instrument as dental sounds is low, making the accuracy of dental sound recognition higher.

[0309] Based on the above embodiment, S604 can be implemented in the following manner, that is, based on the frequency band sound pressure level p x The sibilance recognition and suppression module can determine the Bark domain loudness distribution of the x-th frame audio data in the following manner.

[0310] S01, the sibilance recognition and suppression module simulates the filtering characteristics of the human ear, and corrects and merges the band sound pressure level of the low-frequency band in the e1 band to obtain the band sound pressure level of each band in the e2 band, where e2 is a positive integer less than e1. The band sound pressure level of each band in the e2 band constitutes a new band sound pressure level p x , the new frequency band sound pressure level p x Includes e2 frequency band sound pressure levels.

[0311] like Figure 8 As shown, assuming that in the order of frequency from low to high, among e1 frequency bands, the band sound pressure level of the first frequency band is p1, the band sound pressure level of the second frequency band is p2, and so on, the band sound pressure level of the eth frequency band is pe.

[0312] The sibilance recognition and suppression module combines the first to sixth frequency bands to obtain a new first frequency band, and the frequency range of the new first frequency band may be 25-80 Hz. Correspondingly, the band sound pressure level (p1') of the new first frequency band is the sum of the band sound pressure levels of the first frequency band to the sixth frequency band, that is, p1'=p1+p2+p3+p4+p5+p6.

[0313] The sibilance recognition and suppression module combines the 7th to 9th frequency bands to obtain a new 2nd frequency band, and the frequency range of the new 2nd frequency band can be 100-160Hz. Correspondingly, the band sound pressure level (p2') of the new 2nd frequency band is the sum of the band sound pressure level of the 7th frequency band to the band sound pressure level of the 7th frequency band, that is, p2'=p7+p8+p9.

[0314] The sibilance recognition and suppression module combines the 10th frequency band and the 11th frequency band to obtain a new 3rd frequency band, and the frequency range of the new 3rd frequency band can be 200-250Hz. Correspondingly, the band sound pressure level (p3') of the new 3rd frequency band is the sum of the band sound pressure level of the 10th frequency band and the band sound pressure level of the 11th frequency band, that is, p3'=p10+p11.

[0315] The band sound pressure level of each band from the 12th band to the e1th band remains unchanged. In this way, the e1 bands are merged into e2 bands, and the band sound pressure level p x Also updated to the new frequency band sound pressure level p x . Wherein, e1 can be, for example, 20, and e2 can be, for example, 20.

[0316] S02, new band sound pressure level p based on e2 bands x , the sibilance recognition and suppression module calculates the characteristic loudness of each of the e2 frequency bands.

[0317] Taking one of the e2 frequency bands as an example, based on the frequency band sound pressure level p0 of the frequency band, the sibilance recognition and suppression module can calculate the characteristic loudness N0 of the frequency band by the following formula: p0 is one of the new frequency band sound pressure levels.

[0318]

[0319] Among them, E TQ is the quiet listening range excitation level, E0 is the reference sound intensity, I0 is 10 -12 Watt per square meter (W / m 2 ), E is the excitation level of the frequency band. When calculating the characteristic loudness N1, E can be determined based on the frequency band sound pressure level p0 of the frequency band, for example, E=10 0.1×p0 wait.

[0320] The unit of the characteristic loudness may be song per bark, that is, song / Bark.

[0321] It should be understood that for each of the e2 frequency bands, the characteristic loudness of the frequency band can be calculated in the above manner, thereby obtaining e2 characteristic loudnesses corresponding to the e2 frequency bands.

[0322] S03, the sibilance recognition and suppression module converts the e2 characteristic loudnesses into Bark domain loudness distribution.

[0323] For example, Fig. 9As shown, the characteristic loudness of the new first frequency band in the e2 frequency bands is N1; the characteristic loudness of the new second frequency band is N2, and so on, the characteristic loudness of the new e2 frequency band is Ne2. Each of the e2 frequency bands is a frequency range, corresponding to an upper frequency limit (the maximum frequency in the frequency range) and a lower frequency limit (the minimum frequency in the frequency range).

[0324] Process 1: For each frequency band in the e2 frequency band, the index of the Bark subband to which the frequency lower limit belongs can be calculated based on the frequency lower limit corresponding to the frequency band, and the index of the Bark subband to which the frequency upper limit belongs can also be calculated based on the frequency upper limit corresponding to the frequency band, thereby determining multiple Bark subbands corresponding to the frequency band.

[0325] The number of Bark subbands may be 240, and according to the frequency from low to high, the index of the 240 Barks may be, for example, an integer between 1 and 240. The Bark subband may correspond to the Bark frequency band, for example, the first Bark frequency band may correspond to the first Bark subband to the tenth Bark subband, and so on.

[0326] For example, taking the new first frequency band in the e2 frequency band as an example, the frequency lower limit corresponding to the new first frequency band is hz1, and the frequency lower limit corresponding to the new first frequency band is hz2. Based on hz1 (or hz2) and the following formula, the initial Bark subband index (bark0) corresponding to hz1 (or hz2) can be calculated.

[0327]

[0328] Here, hz can be the upper frequency limit or the lower frequency limit.

[0329] Afterwards, the initial Bark subband index (bark0) can be corrected by the following formula to obtain the Bark subband index (bark1).

[0330]

[0331] For example, substituting hz1 into formula 11, bark0 is calculated to be equal to r1', and then r1' is substituted into formula 12 to obtain bark1, where r1 is the product of bark1 multiplied by 10; substituting hz2 into formula 11, bark0 is calculated to be equal to r2', and then r2' is substituted into formula 12 to obtain bark1 equal to 0.1×r2, where r2 is the product of bark1 multiplied by 10. Therefore, the Bark subband corresponding to the new first frequency band is the r1th Bark subband (the Bark subband with index r1) to the r2th Bark subband (the Bark subband with index r2). By analogy, multiple Bark subbands corresponding to each of the e2 frequency bands can be determined.

[0332] In process 2, the sibilance recognition and suppression module determines the characteristic loudness corresponding to each Bark subband.

[0333] Exemplarily, in order from low to high frequency, for e2 frequency bands, when the characteristic loudness corresponding to the new i-th frequency band is greater than or equal to the characteristic loudness of the new i-1th frequency band, the characteristic loudness of the multiple Bark subbands corresponding to the new i-th frequency band are all the characteristic loudness of the new i-th frequency band; when the characteristic loudness corresponding to the new i-th frequency band is less than the characteristic loudness of the new i-1th frequency band, the multiple Bark subbands corresponding to the new i-th frequency band introduce a masking effect to obtain new characteristic loudness of the multiple Bark subbands corresponding to the new i-th frequency band. i takes integers from 1 to e2. And when i is 1, if the characteristic loudness N1 of the new 1st frequency band is greater than 0, the r1th Bark subband to the r2th Bark subband corresponding to the new 1st frequency band all correspond to the characteristic loudness N1.

[0334] For example, if the characteristic loudness N2 of the new second frequency band is greater than the characteristic loudness of the new first frequency band, then the r3th Bark sub-band to the r4th Bark sub-band corresponding to the new second frequency band all correspond to the characteristic loudness N2.

[0335] If the characteristic loudness N3 of the new third frequency band is less than the characteristic loudness N2 of the new second frequency band, a masking effect is introduced into the r5th Bark subband to the r6th Bark subband corresponding to the new third frequency band.

[0336] It can be understood that for a piece of audio data, when the characteristic loudness of the audio data decreases, the characteristic loudness tends to decrease slowly. Fig.10 ,After point A, the characteristic loudness decreases, and the decreasing trend of the characteristic loudness is slow.

[0337] Therefore, when N3 is less than N2, the characteristic loudness corresponding to the r5th Bark subband to the r6th Bark subband slowly decreases from N2 to N3. That is, for the r5th Bark subband to the r6th Bark subband, the characteristic loudness of each Bark subband decreases in order from low to high frequency, that is, the characteristic loudness of the r5th Bark subband is the highest, close to N2, and the characteristic loudness of the r6th Bark subband is the lowest, close to N3. Exemplarily, the characteristic loudness of the r5th Bark subband may be N2×0.9+N3×0.1; the characteristic loudness of the r5+1th Bark subband may be N2×0.8+N3×0.2; and so on, the characteristic loudness of the r6th Bark subband may be N2×0.1+N3×0.9.

[0338] After the sibilant sound recognition and suppression module traverses the characteristic loudness of the e2 frequency bands, the characteristic loudness corresponding to each Bark sub-band in the multiple Bark sub-bands corresponding to the e2 frequency bands can be determined.

[0339] In process 3, the sibilance recognition and suppression module may determine the characteristic loudness of each of the 24 Bark frequency bands, that is, the Bark domain loudness distribution, based on the characteristic loudness corresponding to each of the Bark sub-bands in the multiple Bark sub-bands corresponding to the e2 frequency bands.

[0340] It should be understood that the Bark subband corresponding to each Bark frequency band is determined. For example, the Bark frequency band with z=1 may correspond to the 1st to 10th Bark subbands; the Bark frequency band with z=2 may correspond to the 11th to 20th Bark subbands, etc. The characteristic loudness of each Bark frequency band is the sum of the characteristic loudnesses of multiple Bark subbands corresponding to the Bark frequency band. For example, the characteristic loudness of the Bark frequency band with z=1 (the 1st Bark frequency band) is the sum of the characteristic loudnesses of the 1st to 10th Bark subbands, etc.

[0341] Therefore, when the characteristic loudness corresponding to each Bark sub-band in the multiple Bark sub-bands corresponding to the e2 frequency bands is determined, the characteristic loudness of each Bark frequency band, that is, the Bark domain loudness distribution, can be calculated.

[0342] On the basis of the above embodiment, in order to further improve the accuracy of dental sound recognition, S608 can also be performed by Fig.11 The described process implements:

[0343] S1101. The sibilance recognition and suppression module determines whether the recognition sharpness of the x-th frame audio data is greater than or equal to a threshold value 1.

[0344] If the recognition sharpness of the x-th frame of audio data is greater than the threshold 1, it is determined that the x-th frame of audio data is sibilant.

[0345] If the recognition sharpness of the x-th frame of audio data is less than or equal to threshold 1, execute S1102.

[0346] S1102: The sibilance recognition and suppression module determines whether the x-1th frame of audio data is sibilance.

[0347] If the audio data of the x-1th frame is not a sibilant sound, it is determined that the audio data of the xth frame is not a sibilant sound.

[0348] If the audio data of the x-1th frame is sibilant, execute S1103.

[0349] S1103 . The sibilance recognition and suppression module determines whether the recognition sharpness of the x-th frame audio data is greater than or equal to a threshold value 2.

[0350] If the recognition sharpness of the x-th frame of audio data is greater than the threshold 2, it is determined that the x-th frame of audio data is sibilant.

[0351] If the recognition sharpness of the x-th frame audio data is less than or equal to the threshold 2, it is determined that the x-th frame audio data is not sibilant.

[0352] It should be understood that threshold 1 and threshold 2 are preset values, and threshold 2 is smaller than threshold 1.

[0353] Combination Figure 2A and Figure 2B It can be seen from the time domain waveform of the dental sound that for multiple consecutive frames of dental sounds, the amplitude of the dental sound first shows a slow increase trend, and then shows a slow decrease trend. Therefore, in the amplitude decrease stage, the amplitude of the dental sound in the next frame is usually lower than the amplitude of the previous frame of dental sounds, that is, the energy of the dental sound in the next frame is usually lower than the energy of the dental sound in the previous frame, which causes the recognition sharpness of the dental sound in the next frame to be usually lower than the recognition sharpness of the dental sound in the previous frame. Therefore, when the recognition sharpness of the x-th frame of audio data is less than or equal to threshold 1 and the x-1th frame of audio data is a dental sound, it can be determined whether the x-th frame of audio data is a dental sound based on the recognition sharpness of the x-th frame of audio data and a smaller threshold 2. In this way, for multiple frames of continuous dental sounds, in the stage where the amplitude of the dental sounds decreases, dental sounds with smaller amplitudes can be identified, which helps to further improve the accuracy of dental sound recognition.

[0354] The above describes the process of the sibilance recognition and suppression module performing sibilance recognition on the xth frame of audio data. When the sibilance recognition and suppression module traverses each frame of audio data in M ​​frames of audio data (x is an integer from 1 to M), the sibilance recognition and suppression module can recognize all sibilance in the M frames of audio data.

[0355] Further, in the case that the x-th frame audio data contains sibilant sound, the sibilant sound recognition and suppression module may suppress the sibilant sound through S610 to S615 .

[0356] When the x-th frame audio data is not sibilant, the sibilant recognition and suppression module executes S616, ie, converting the x-th frame audio data from a frequency domain signal into a time domain signal.

[0357] The process of sibilance suppression is described in detail below.

[0358] S610: The sibilance recognition and suppression module divides a preset frequency range in the x-th frame of audio data into y frequency bands.

[0359] The preset frequency range may include a sibilance frequency band, that is, the preset frequency range includes 4-12kHz, so that the sibilance recognition and suppression module can suppress the sibilance frequency band in the x-th frame of audio data. Alternatively, the preset frequency range may also partially overlap with the sibilance frequency band. For example, the preset frequency range is 5-13kHz, etc., so that the sibilance recognition and suppression module can suppress the sibilance frequency band in other frequency bands in the x-th frame of audio data.

[0360] y can be a preset integer greater than 1, such as 3, 9 or 10.

[0361] S611. Based on the sharpness distribution 1, the sibilance recognition and suppression module calculates the suppression sharpness of each of the y frequency bands.

[0362] Among them, the sharpness part 1 is the sharpness part of the x-th frame audio data calculated in S605, that is, the sharpness of each Bark frequency band. In the y frequency bands corresponding to the x-th frame audio data, the suppression sharpness of each frequency band is not only related to the sharpness distribution 1, but also to the frequency band sound pressure level p of the x-th frame audio data. x Related. Band sound pressure level p x It may be the frequency band sound pressure level of each frequency band in the e1 frequency bands corresponding to the x-th frame audio data calculated in S603.

[0363] For example, taking the kth frequency band among the y frequency bands as an example, the sharpness of the kth frequency band is k ) satisfies the following formula:

[0364]

[0365] Wherein, z is the index of the bark frequency band, k_fl is the lower limit frequency of the kth frequency band (the minimum frequency in the frequency range), z=bark(k_fl) is the index of the Bark frequency band to which the lower limit frequency of the kth frequency band belongs, k_fh is the upper limit frequency of the kth frequency band (the maximum frequency in the frequency range), z=bark(k_fh) is the index of the Bark frequency band to which the upper limit frequency of the kth frequency band belongs, z=bark() is a function for determining the index of the Bark frequency band based on the lower limit frequency or the upper limit frequency, and f(p') is the band sound pressure level p based on the xth frame audio data xThe frequency band sound pressure level scores corresponding to the determined k frequency bands.

[0366] Then the suppression sharpness of the kth frequency band can be understood as the product of f(p') and the sum of the sharpnesses of one or more Bark frequency bands corresponding to the kth frequency band. The one or more Bark frequency bands corresponding to the kth frequency band are the Bark frequency band from the lower limit frequency of the kth frequency band to the Bark frequency band to the upper limit frequency of the kth frequency band.

[0367] In Formula 13, z=bark() is similar to Formula 11 and Formula 12 above, and z in z=bark() is the product of bark1 and 10 in Formula 12. That is, the method of determining z based on the lower limit frequency or the upper limit frequency is similar to the method of determining r1 and r2 above, and the above description can be referred to, which will not be repeated here.

[0368] f(p') can be determined based on the sum of the sound pressure levels of one or more frequency bands in the e1 frequency bands corresponding to the kth frequency band. f(p') can satisfy the following formula:

[0369]

[0370] Among them, h is the index of the frequency band in the e1 frequency band, k_fl is the lower limit frequency of the k-th frequency band (the minimum frequency in the frequency range), h=k_fl is the index of a frequency band in the e1 frequency band to which the lower limit frequency of the k-th frequency band belongs, k_fh is the upper limit frequency of the k-th frequency band (the maximum frequency in the frequency range), h=k_fh is the index of a frequency band in the e1 frequency band to which the upper limit frequency of the k-th frequency band belongs, and p(h) is the band sound pressure level p of the h-th frequency band in the e1 frequency band. x The sound pressure level of a frequency band in the ), thres_p is the preset threshold value 3.

[0371] That is, the method for determining one or more frequency bands in the e1 frequency band corresponding to the k-th frequency band is similar to the method for determining one or more Bark frequency bands corresponding to the k-th frequency band. If the lower limit frequency of the k-th frequency band is within the frequency range of a frequency band in the e1 frequency band, then the one frequency band is the frequency band corresponding to the lower limit frequency of the k-th frequency band; if the upper limit frequency of the k-th frequency band is within the frequency range of a frequency band in the e1 frequency band, then the one frequency band is the frequency band corresponding to the upper limit frequency of the k-th frequency band. The one or more frequency bands in the e1 frequency band corresponding to the k-th frequency band are: the frequency band corresponding to the lower limit frequency of the k-th frequency band to the frequency band corresponding to the upper limit frequency of the k-th frequency band.

[0372] For one or more of the e1 frequency bands corresponding to the kth frequency band, each frequency band corresponds to a frequency band sound pressure level, That is, the sum of one or more frequency band sound pressure levels corresponding to the one or more frequency bands, wherein the one or more frequency band sound pressure levels corresponding to the one or more frequency bands can also be understood as the frequency band sound pressure level distribution corresponding to the kth frequency band.

[0373] The larger the value, the greater the sharpness corresponding to the kth frequency band. The calculated sharpness of the k-th frequency band k ) may be larger; The smaller the value, the smaller the sharpness corresponding to the kth frequency band. The calculated sharpness of the k-th frequency band k ) may be smaller.

[0374] f(p') can be determined based on p'. For example, f(p') can be determined in any of the following ways.

[0375] For example, f(p') may satisfy the following formula:

[0376]

[0377] f(p') can also satisfy the following formula:

[0378]

[0379] Combining Formula 15 or Formula 16, it can be seen that when p' is less than the threshold 3 (thres_p), f(p') can be set to 0. That is, When it is less than the threshold value 3 (thres_p), it means In this case, even if the sharpness corresponding to the kth frequency band is large, the kth frequency band has a small energy, so that the user's subjective hearing of the kth frequency band is not sharp, and there is no need to suppress the kth frequency band. Therefore, f(p') can be set to 0 at this time. Among them, the threshold 3 (thres_p) can be, for example, a value determined by experience and preset in the terminal device.

[0380] In addition, f(p') can also be equal to p'. In this way, there is no need to convert p', which reduces the amount of calculation of the terminal device.

[0381] That is, the changing trends of f(p') and p' can be the same, that is, the larger p' is, the larger f(p') is; the smaller p' is, the smaller f(p') is. When p' is small, it can reflect that the energy of the kth frequency band is low, then f(p') can be small, for example, it can be 0, or equal to p'. So that the sharpness of the kth frequency band calculated based on f(p') is kOn the contrary, when p' is larger, it can be reflected that the energy of the kth frequency band is higher, and f(p') can be larger, so that the sharpness of the kth frequency band calculated based on f(p') is smaller. k ) is larger.

[0382] It can be seen that based on f(p') and The calculated sharpness of the k-th frequency band k ) can reflect the sharpness and energy level of the kth frequency band.

[0383] In this way, the sibilance recognition and suppression module can determine the frequency bands with higher sharpness and energy among the y frequency bands based on the suppression sharpness of each frequency band, thereby suppressing the frequency bands with higher sharpness and energy. Compared with suppressing the frequency bands with higher energy, the embodiment of the present application also takes sharpness into consideration, so that the probability of the sibilance recognition and suppression module suppressing the frequency bands with higher energy and lower sharpness is lower, thereby helping to reduce the damage of sibilance suppression to the timbre. It can improve the sound quality of the processed audio data and improve the user experience.

[0384] S612: The sibilance recognition and suppression module determines whether the suppression sharpness of the kth frequency band is greater than a threshold value 4. The threshold value 4 is a preset value.

[0385] If the suppression sharpness of the kth frequency band is less than or equal to the threshold value 4, it means that the sharpness and / or energy of the kth frequency band is low, and S613 may be executed, ie, the frequency band gain of the kth frequency band is set to 0.

[0386] If the suppression sharpness of the kth frequency band is greater than the threshold value 4, it means that the sharpness and energy of the kth frequency band are relatively high, and S614 can be executed, that is, the band gain g of the kth frequency band is calculated. k .

[0387] Where, the band gain g k It can be determined based on the suppression sharpness of the kth frequency band, threshold 4, and threshold 5. For example, the frequency band gain g k The following formula can be satisfied:

[0388]

[0389] Among them, sharp k is the suppression sharpness of the kth frequency band, thres sharpnes For threshold 4, progain i The threshold is 5.

[0390] Threshold 5 is a preset value less than 0, which can be used to indicate suppression capability. The smaller the threshold 5 is, the smaller the suppression strength of the sibilance recognition and suppression module on the kth frequency band is; the larger the threshold 5 is, the greater the suppression strength of the sibilance recognition and suppression module on the kth frequency band is.

[0391] It can be understood that, in combination with formula 17, it can be determined that the gain of the kth frequency band is positively correlated with the suppression sharpness of the kth frequency band; combined with formulas 13 to 16, it can be determined that the suppression sharpness of the kth frequency band is positively correlated with the sum of the sharpness of one or more Bark frequency bands corresponding to the kth frequency band. That is, for two frequency bands, assuming that the f(p') corresponding to the two frequency bands are equal or very close (for example, the difference is less than a threshold), then the frequency band with a larger sum of the sharpness of one or more Bark frequency bands in the two frequency bands has a larger suppression sharpness, and the gain corresponding to the frequency band is also larger.

[0392] Exemplarily, for the kth frequency band and the wth frequency band, k and w may be different, and the kth frequency band and the wth frequency band may belong to one frame of audio data (identified as sibilant); or, k and w are the same, and the kth frequency band and the wth frequency band belong to different frames of audio data (identified as sibilant). For example, the kth frequency band belongs to the xth frame of audio data (identified as sibilant), the wth frequency band belongs to the x+hth frame of audio data (identified as sibilant), and k and w are equal.

[0393] Assume that the amplitude of the kth frequency band is the same or similar to that of the wth frequency band, and that f(p') corresponding to the kth frequency band is equal to or very close to f(p') corresponding to the wth frequency band. And the sum of the sharpness of one or more Bark frequency bands corresponding to the kth frequency band is S1, then the suppression sharpness of the kth frequency band is the product of S1 and f(p'), and the product is, for example, sharp1; the sum of the sharpness of one or more Bark frequency bands corresponding to the wth frequency band is S2, then the suppression sharpness of the wth frequency band is the product of S2 and f(p'), and the product is, for example, sharp2. Then, when S1 is less than S2, sharp1 is less than sharp2. Substituting sharp1 and sharp2 into formula 17 respectively, the calculated gain of the kth frequency band is also less than the gain of the wth frequency band. Alternatively, if sharp1 is less than the threshold 4, the gain of the kth frequency band is 0.

[0394] Therefore, the gain is determined based on the suppression sharpness. For the frequency band with higher sharpness, the gain calculated by the terminal device is relatively large; for the frequency band with lower sharpness, the gain calculated by the terminal device is relatively small. This allows the terminal device to suppress the sibilant signal with higher sharpness to a higher degree, and suppress the sibilant signal with lower sharpness to a lower degree. This makes the sibilant suppression less damaging to the timbre.

[0395] Based on the above manner (S611 to S614), when k takes all integers between 1 and y, the sibilant recognition and suppression module can calculate the frequency band gain of each frequency band in the y frequency bands corresponding to the x-th frame audio data.

[0396] S615 . The sibilance recognition and suppression module performs time-frequency domain gain smoothing on the y frequency bands based on the frequency band gains of the y frequency bands.

[0397] That is, the frequency band gains of y frequency bands may be smoothed from both the time domain and the frequency domain, so that the processed x-th frame audio data sounds smoother and more natural.

[0398] Exemplarily, the sibilance recognition and suppression module may perform time-frequency domain gain smoothing in the following manner.

[0399] Assume that y is 3, and from low to high frequency, the band gain of the first band in the three bands is g1, the band gain of the second band is g2, and the band gain of the third band is g3. There may be 0 or no 0 in g1, g2, and g3.

[0400] Then, in the order of frequency from low to high, the sibilance recognition and suppression module can perform smoothing on the latter frequency band based on the frequency band gain of the previous frequency band and the frequency band gain of the latter frequency band in the y frequency bands, and the frequency band gain g2 of the second frequency band can be replaced by g1×0.9+g2×0.1; the frequency band gain g2 of the third frequency band can be replaced by g2×0.9+g3×0.1, etc. In this way, after suppressing the y frequency bands, the suppressed audio data is smoother and more natural.

[0401] It should be understood that the y frequency bands shown above are all the y frequency bands corresponding to the x-th frame of audio data.

[0402] In addition, the sibilance recognition and suppression module may also perform smoothing processing on each of the y frequency bands corresponding to the x-1th frame of audio data based on the frequency band gain of each of the y frequency bands corresponding to the x-1th frame of audio data.

[0403] Assume that, in the three frequency bands corresponding to the x-1th frame audio data, the frequency band gain of the first frequency band is g1', the frequency band gain of the second frequency band is g2', and the frequency band gain of the third frequency band is g3'. There may be 0 or no 0 in g1', g2' and g3'.

[0404] Then, in order from low to high frequency, the sibilance recognition and suppression module can perform smoothing processing on the band gain of the fth frequency band corresponding to the xth frame of audio data based on the band gain of the fth frequency band corresponding to the x-1th frame of audio data, where f is a positive integer. For example, the band gain g1 of the first frequency band in the xth frame of audio data can be replaced by g1'×0.9+g1×0.1; the band gain g2 of the second frequency band in the xth frame of audio data can be replaced by g2'×0.9+g2×0.1; the band gain g3 of the third frequency band in the xth frame of audio data can be replaced by g3'×0.9+g3×0.1, etc.

[0405] In this way, the sibilance recognition and suppression module can determine the gain of each frequency band in the y frequency bands (corresponding to the x-th frame of audio data) after smoothing.

[0406] Afterwards, the sibilance recognition and suppression module may suppress y frequency bands corresponding to the x-th frame of audio data based on the gains after smoothing.

[0407] S616: Convert the x-th frame audio data from a frequency domain signal to a time domain signal.

[0408] It should be understood that no matter whether the x-th frame of audio data is sibilant sound or not, the x-th frame of audio data is converted from a frequency domain signal to a time domain signal.

[0409] When the x-th frame audio data is not sibilant, the sibilant recognition and suppression module does not execute S610 to S615; when the x-th frame audio data is sibilant, the sibilant recognition and suppression module executes S610 to S615 to suppress sibilant on the x-th frame audio data.

[0410] Through S610 to S616, when x takes all integers between 1 and M, the sibilance recognition and suppression module can suppress sibilance for all recognized sibilance in the M frames of audio data, thereby making the processed M frames of audio data smoother.

[0411] Furthermore, when the audio playback module such as the speaker of the terminal device plays the audio data after the sibilance is suppressed, the audio data after the sibilance is suppressed may still sound sharp due to the characteristics of the audio playback module itself.

[0412] For example, when the audio playback module such as a speaker is small, such as the speaker of a mobile phone, the energy of the sibilance band in the audio data may be amplified when playing audio data, so that even if the audio data played is sibilance-suppressed audio data, the audio data played by the terminal device may still sound sharp.

[0413] Therefore, the terminal device may also perform filtering processing on the audio data processed by the sibilance recognition and suppression module through the filtering processing module. The details are as follows.

[0414] S617: The sibilance recognition and suppression module transmits the converted time domain signal to the filtering processing module.

[0415] Taking the example that the time domain signal transmitted by the sibilant recognition and suppression module to the filtering processing module is the time domain signal of the x-th frame audio data, the filtering processing module can execute S618, that is, filter the time domain signal of the x-th frame audio data to obtain the filtered audio data of the x-th frame audio data.

[0416] It is understood that the center frequency (FC), Q value, gain and other parameters of the filter processing module can be preset, and the center frequency (FC), Q value and gain and other parameters of the filter processing module are determined based on the frequency response curve of the audio playback module such as the speaker.

[0417] For example, for the peak in the frequency response curve, the gain is high, so that the filter processing module may amplify the frequency band energy corresponding to the frequency point of the peak (the horizontal coordinate of the peak). If the amplified frequency band coincides with the sibilance in the audio data, the energy of the sibilance will be amplified. Therefore, it is necessary to set the center frequency (FC), Q value, gain and other parameters of the filter processing module to make the frequency response curve of the filter processing module smoother.

[0418] For example, the center frequency (FC) can be set to the frequency point of the peak in the initial frequency response curve. The Q value can be set as the ratio of the center frequency (FC) to the effective bandwidth. The effective bandwidth can be set based on the width of the peak in the initial frequency response curve. The larger the width of the peak in the initial frequency response curve (the difference in the horizontal coordinates), the larger the effective bandwidth can be; the smaller the width of the peak in the initial frequency response curve, the smaller the effective bandwidth can be. The gain of the filtering processing module can be determined based on the vertical coordinate of the peak in the initial frequency response curve. For example, the gain of the filtering processing module can be the difference between the vertical coordinate of the peak and the vertical coordinate of the horizontal line. The horizontal line is a horizontal straight line determined based on the relatively stable part of the initial frequency response curve.

[0419] In this way, through the filtering processing of the filtering processing module, the amplification effect of the audio playback module such as the speaker on the sibilance band can be reduced, thereby helping to further improve the sound quality of the audio data and improve the user experience.

[0420] The audio data (filtered audio data) that has been processed by sibilance suppression and filtering can be played in the following manner (S619 to S625). The following manner is only an example. In some possible implementations, the terminal device can also play the audio data (filtered audio data) that has been processed by sibilance suppression and filtering in other ways, which does not constitute a limitation on the embodiments of the present application.

[0421] S619: The filtering processing module writes the filtered audio data of the x-th frame of audio data into the cache.

[0422] It can be understood that when x takes all integers between 1 and M, the filtering processing module can write all the filtered audio data of the M frames of audio data into the cache.

[0423] S620: The audio mixer obtains one or more frames of filtered audio data from the buffer, wherein the one or more frames of filtered audio data belong to part or all of the filtered audio data of the M frames of audio data.

[0424] S621. The audio mixer performs mixing, sound effect processing, etc. on the filtered audio data of one or several frames of audio data to obtain audio data a.

[0425] S622: The audio mixer transmits the audio data a to the audio HAL.

[0426] S623: The audio HAL further processes the audio data a by converting the format, adjusting the sampling rate and volume control, etc., to obtain audio data b.

[0427] S624: The audio HAL transmits the audio data b to the audio driver.

[0428] S625: The audio driver plays the audio data b through an audio playback module such as a speaker.

[0429] It can be understood that S620 to S625 can be executed multiple times in a loop so that the terminal device can play all the audio data in the M frames of audio data.

[0430] The method of the present application can accurately identify sibilants in audio data, and can effectively suppress the identified sibilants with little damage to the timbre. Fig. 12A and Fig. 12B ,in Fig. 12B The spectrogram of the audio data before and after sibilance suppression provided in the embodiment of the present application is shown in FIG. Fig. 12A For Fig. 12B A schematic diagram of a spectrogram is shown.

[0431] Fig. 12B (a) in FIG. 1 shows the original sequence spectrogram, that is, the spectrogram before sibilant suppression. Correspondingly, Fig. 12A (a) shows a schematic diagram of the original sequence spectrogram; Fig. 12B (b) shows a spectrogram after sibilance recognition and sibilance suppression by the method of the present application. Correspondingly, Fig. 12B (b) in FIG. 1 is a schematic diagram of a spectrogram after sibilant suppression.

[0432] exist Fig. 12A In the figure, the white part represents the part with higher energy, and the closer to white, the higher the energy. Fig. 12B The closer the color is to yellow, the higher the energy. Fig. 12A The part circled in box 1201 in (a) can be understood as Fig. 12B The portion circled in box 1205 in (a) shows a portion of the sibilant sound. Fig. 12A The part circled in box 1202 in (b) can be understood as Fig. 12B The diagram of the portion circled in box 1206 in (b) shows the sibilance after sibilance suppression.

[0433] Combination Fig. 12A Compare the selected words in box 1201 with those in box 1202, or Fig. 12B By comparing the spectra circled in box 1205 with those in box 1206, and comparing the spectra circled in box 1207 with those in box 1208, it can be seen that the audio data after the sibilance recognition and sibilance suppression by the method of the present application has the sibilance with higher frequency and greater energy significantly suppressed, so that Fig. 12A or Fig. 12B In (b), the energy of the sibilant sound at higher frequency positions is reduced.

[0434] And, combined with Fig. 12A The circled portion of box 1203 in (a) is Fig. 12A Circle the part in box 1204 in (a), or combine Fig. 12B From the spectrograms of the low-frequency part in (a) and (b), it can be seen that the method of the present application will not suppress the audio data with lower frequency and higher energy, so that the timbre damage of the sibilant sound suppression is relatively small.

[0435] In addition, the method of the present application can identify sibilance with a higher accuracy than that of sibilance identified by using standard sharpness and a preset threshold of 1. The method of the present application can suppress sibilance, which can reduce the damage to the timbre, compared with suppressing sibilance by using EQ or MBDRC.

[0436] For example, in combination Fig.13A and Fig. 13B ,in Fig. 13B The spectrum diagram for showing the sibilance suppression part provided in the embodiment of the present application is as follows: Fig.13A For Fig. 13B 's hint. Fig.13A (a) and Fig. 13B(a) in FIG. 1 shows the original sequence of the same audio data, which includes the sound of musical instruments and human voice. If the sibilance is identified based on the amplitude of the audio data and suppressed based on the EQ method, the suppressed audio part can be as follows: Fig.13A (b) (the white part) or Fig. 13B As shown in (b) (the red and yellow parts). It can be seen that this audio processing method also suppresses the sound of musical instruments (piano and drums), and also suppresses the low-frequency part of voice and accompaniment, resulting in timbre loss. The audio timbre heard by the user is distorted and the listening experience is poor.

[0437] When the method of the present application is used to perform sibilance recognition and sibilance suppression, the suppressed audio portion can be as follows: Fig.13A (c) (the white part) or Fig. 13B As shown in (c) in the figure (the red or yellow part). It can be seen that the method of the present application has a high accuracy in audio recognition and is not easy to identify musical instrument sounds as dental sounds, so that the musical instrument sounds are almost not suppressed; and the low-frequency part in the audio data is also almost not suppressed. By comparison, it can be seen that the method of the present application can reduce the damage to the timbre.

[0438] It should be understood that the size of the sequence numbers in the above embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic.

[0439] The audio processing method of the embodiment of the present application has been described above, and the device for executing the above method provided by the embodiment of the present application is described below. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the relevant device provided by the embodiment of the present application can execute the steps in the above list sorting method.

[0440] The embodiment of the present application also provides an audio processing device. The audio processing device may include a processor, a communication interface and a memory. The processor, the communication interface and the memory communicate with each other through an internal connection path, the memory is used to store instructions, and the processor is used to execute the instructions stored in the memory. The communication interface can be used to send signals to other devices (such as a processor or a touch screen of a terminal device), and can also be used to receive signals from other devices (such as a memory). Exemplarily, the communication interface reads the instructions stored in the memory and sends the instructions to the processor.

[0441] It should be understood that the audio processing device can be specifically the terminal device in the above embodiment, and can be used to execute the various steps and / or processes corresponding to the terminal device in the above method embodiment. Optionally, the memory can include a read-only memory and a random access memory, and provide instructions and data to the processor. A part of the memory can also include a non-volatile random access memory. For example, the memory can also store information about the device type. The processor can be used to execute the instructions stored in the memory, and when the processor executes the instructions stored in the memory, the processor is used to execute the various steps and / or processes of the above method embodiment.

[0442] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0443] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor executes the instructions in the memory, and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0444] The application startup method provided in the embodiment of the present application can be applied to electronic devices with communication functions. The electronic devices include terminal devices, and the specific device form of the terminal devices can refer to the above related descriptions, which will not be repeated here.

[0445] An embodiment of the present application provides an electronic device, which terminal device includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the above method.

[0446] The embodiment of the present application provides a chip. The chip includes a processor, and the processor is used to call a computer program in a memory to execute the technical solution in the above embodiment. Its implementation principle and technical effect are similar to those of the above related embodiments, and will not be repeated here.

[0447] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. The above method is implemented when the computer program is executed by the processor. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0448] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that is intended to carry or store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave) is used to transmit software from a website, server or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disks and optical disks as used herein include optical disks, laser disks, optical disks, digital versatile disks (DVD), floppy disks and Blu-ray disks, where disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.

[0449] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above method.

[0450] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0451] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. An audio processing method, characterized in that: Applied to a terminal device, the method comprises: Display a first interface of a first application, wherein the first interface includes a first control; Based on a first operation of the user on the first control, obtaining a first audio, the first audio including a first sibilant signal and a second sibilant signal, the amplitude of the first sibilant signal is consistent with the amplitude of the second sibilant signal, and a first sharpness of the first sibilant signal is less than a second sharpness of the second sibilant signal; Play a second audio, wherein the second audio is the audio obtained by suppressing sibilance of the first audio, the second audio includes a third sibilance signal and a fourth sibilance signal, the third sibilance signal is the processed first sibilance signal, the fourth sibilance signal is the processed second sibilance signal, and a first difference between the amplitude of the first sibilance signal and the amplitude of the third sibilance signal is smaller than a second difference between the amplitude of the second sibilance signal and the amplitude of the fourth sibilance signal.

2. The method according to claim 1, characterized in that The first sharpness is the sum of the sharpness distributions corresponding to the first sibilant sound signals, and the second sharpness is the sum of the sharpness distributions corresponding to the second sibilant sound signals.

3. The method according to claim 1 or 2, characterized in that: The third sibilant signal is a signal obtained by processing the first sibilant signal based on the gain of the first sibilant signal, and the fourth sibilant signal is a signal obtained by processing the second sibilant signal based on the gain of the second sibilant signal, and the gain of the first sibilant signal is smaller than the gain of the second sibilant signal.

4. The method according to claim 3, characterized in that The gain of the first sibilant sound signal is determined based on the sum of the first sharpness and the frequency band sound pressure level distribution corresponding to the first sibilant sound signal, and the gain of the second sibilant sound signal is determined based on the sum of the second sharpness and the frequency band sound pressure level distribution corresponding to the second sibilant sound signal. The sum of the frequency band sound pressure level distribution corresponding to the first sibilant sound signal is positively correlated with the amplitude of the first sibilant sound signal, and the sum of the frequency band sound pressure level distribution corresponding to the second sibilant sound signal is positively correlated with the amplitude of the second sibilant sound signal.

5. The method according to any one of claims 1 to 4, characterized in that The first sibilant signal belongs to the c1th frame audio data, the second sibilant signal belongs to the c2th frame audio data, the c1th frame audio data and the c2th frame audio data are sibilants, the c1th frame audio data is identified by the terminal device based on the sharpness distribution corresponding to the c1th frame audio data and the frequency spectrum centroid of the c1th frame audio data, and the c2th frame audio data is identified by the terminal device based on the sharpness distribution corresponding to the c2th frame audio data and the frequency spectrum centroid of the c2th frame audio data.

6. The method according to claim 5, characterized in that The product of the sum of the sharpness distribution corresponding to the c1th frame audio data and the normalized spectral centroid of the c1th frame audio data is greater than the fourth threshold, and the product of the sum of the sharpness distribution corresponding to the c2th frame audio data and the normalized spectral centroid of the c2th frame audio data is greater than the fourth threshold.

7. The method according to claim 5 or 6, characterized in that: The sharpness distribution corresponding to the c1-th frame audio data and the sharpness distribution corresponding to the c2-th frame audio data respectively include sharpnesses of multiple Bark frequency bands, and the multiple Bark frequency bands are Bark frequency bands whose frequencies are greater than or equal to a preset frequency.

8. An audio processing method, characterized in that: include: Determine that the xth frame of audio data among the multiple frames of audio data is sibilant sound; Divide the preset frequency range of the x-th frame audio data into y frequency bands, the preset frequency range includes the sibilant frequency band, or the preset frequency range partially overlaps with the sibilant frequency band, and y is a positive integer; Calculate the gain of each of the y frequency bands based on the sharpness distribution and the band sound pressure level distribution corresponding to each of the y frequency bands, wherein the sharpness distribution includes the sharpness of at least one frequency band, and the band sound pressure level distribution includes the band sound pressure level of at least one frequency band; Based on the gain of each frequency band in the y frequency bands, the y frequency bands are suppressed to obtain audio data after sibilance suppression.

9. The method according to claim 8, characterized in that The calculating the gain of each frequency band in the y frequency bands comprises: Calculating the sum of the first frequency band sound pressure level distributions corresponding to the kth frequency band in the y frequency bands to obtain a first value; Calculate the sum of the first sharpness distributions corresponding to the kth frequency band in the y frequency bands to obtain a second value; Calculating the suppression sharpness of the k-th frequency band based on the first value and the second value, where the suppression sharpness of the k-th frequency band is used to represent the magnitude of the first value and the second value; Based on the suppression sharpness, a first gain of the k-th frequency band is determined.

10. The method according to claim 9, characterized in that The step of determining a first gain of the kth frequency band based on the suppression sharpness comprises: When the suppression sharpness is less than or equal to a first threshold, the first gain is 0; and / or, When the compression sharpness is greater than the first threshold, the first gain is calculated based on the compression sharpness, the first threshold, and a second threshold, where the second threshold is used to represent the compression capability.

11. The method according to claim 10, characterized in that The first gain g k Satisfies the following formula: Among them, sharp k For the suppression sharpness, thres sharpnes For the first threshold, progain i is the second threshold.

12. The method according to any one of claims 9 to 11, characterized in that The calculating the suppression sharpness of the k-th frequency band based on the first value and the second value comprises: Calculating a ratio of the first value to a third threshold value to obtain a third value; Based on a preset function, convert the third value into a fourth value, wherein the fourth value is less than or equal to the third value; The product of the fourth value and the second value is calculated, and the suppression sharpness is the product of the fourth value and the second value.

13. The method according to any one of claims 9 to 12, characterized in that The obtaining of the first value comprises: Divide the amplitude spectrum of the x-th frame of audio data into E frequency bands, where E is a positive integer; Calculate a second frequency band sound pressure level distribution of the x-th frame audio data, where the second frequency band sound pressure level distribution includes a frequency band sound pressure level of each frequency band in the E frequency bands; Based on the second frequency band sound pressure level distribution, a sum of the first frequency band sound pressure level distributions is calculated, where the first frequency band sound pressure level distribution includes the frequency band sound pressure level of at least one frequency band among the E frequency bands.

14. The method according to claim 13, characterized in that In order from low to high frequency, at least one of the E frequency bands includes a first frequency band to a second frequency band, the first frequency band is a frequency band to which the lower limit frequency of the k-th frequency band in the E frequency bands belongs, and the second frequency band is a frequency band to which the upper limit frequency of the k-th frequency band in the E frequency bands belongs.

15. The method according to any one of claims 9 to 14, characterized in that The obtaining of the second value comprises: Calculate a second sharpness distribution of the x-th frame of audio data, where the second sharpness distribution includes the sharpness of each Bark frequency band; Based on the second sharpness distribution, a sum of the first sharpness distribution is calculated, wherein the first sharpness distribution includes the sharpness of at least one Bark frequency band.

16. The method according to any one of claims 8 to 15, characterized in that The step of determining that the xth frame of audio data among the multiple frames of audio data is sibilant sound comprises: Based on the sum of the third sharpness distributions of the x-th frame of audio data and the frequency spectrum centroid of the x-th frame of audio data, it is determined that the x-th frame of audio data is sibilant.

17. The method according to claim 16, characterized in that The third sharpness distribution includes the sharpness of each Bark frequency band in a plurality of Bark frequency bands, where the plurality of Bark frequency bands are Bark frequency bands having a frequency greater than or equal to a preset frequency.

18. The method according to claim 16 or 17, characterized in that The determining that the x-th frame of audio data is sibilant sound comprises: Based on the sum of the third sharpness distributions and the normalized frequency spectrum centroid of the x-th frame of audio data, it is determined that the x-th frame of audio data is sibilant.

19. The method according to claim 18, characterized in that The determining that the x-th frame of audio data is sibilant sound comprises: Calculate the product of the sum of the third sharpness distributions and the centroid of the normalized spectrum to obtain the recognition sharpness of the x-th frame of audio data; When the recognition sharpness is greater than a fourth threshold, it is determined that the x-th frame of audio data is sibilant sound.

20. The method according to claim 19, characterized in that The method further comprises: When the recognition sharpness is less than or equal to the fourth threshold, determining whether the x-1th frame of audio data is sibilant; In the case where the x-1th frame of audio data is sibilant sound, determining whether the recognition sharpness is greater than a fifth threshold, and the fifth threshold is less than the fourth threshold; When the recognition sharpness is greater than the fifth threshold, it is determined that the x-th frame of audio data is sibilant sound.

21. The method according to claim 20, characterized in that The method further comprises: In the case that the x-1th frame audio data is not a sibilant, determining that the xth frame audio data is not a sibilant; or, When the recognition sharpness is less than or equal to the fifth threshold, it is determined that the x-th frame audio data is not sibilant.

22. The method according to any one of claims 8 to 21, characterized in that The method further comprises: The audio data after the sibilance suppression is filtered by a filter, wherein a first parameter of the filter is determined based on a frequency response curve of the audio playback module, and the first parameter includes one or more of the following: center frequency, quality factor or gain.

23. An electronic device, characterized in that: The electronic device comprises: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method as described in any one of claims 1 to 22.

24. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 22.

25. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 22.

26. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 22.

Images