Signal processing method for enhancing human voice, electronic device, and storage medium

Abstract

The application provides a signal processing method for enhancing human voice, an electronic device and a storage medium, and relates to the technical field of signal processing. In the scene of playing audio and video by the electronic device, the human voice gain parameter can be determined according to the environmental noise and the background voice frequency domain signal and the human voice frequency domain signal corresponding to the multi-channel audio stream, then the human voice frequency domain signal is gain adjusted according to the human voice gain parameter and the normalized spectral flux, and the enhanced human voice frequency domain signal is obtained. Since the gain of the human voice signal can be dynamically adjusted for the input multi-channel audio signal under different environmental noises, the effect of enhancing the human voice is achieved, and the phenomenon of large and small hearing is avoided, so the application scheme can achieve the effect that the video voice can be heard clearly without disturbing others when playing the video in a quiet environment, or the user can hear the video voice clearly when playing the video in a noisy environment, and the user experience can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of signal processing, in particular to a signal processing method for enhancing human voice, an electronic device and a storage medium. BACKGROUND

[0002] At present, users often play videos such as films and television series through electronic devices with loudspeakers such as mobile phones, tablet computers, notebook computers and smart screens. However, when playing videos through electronic devices, due to factors such as hardware capability of the electronic device, listening ability of the user himself, listening environment and content of the material, the user may not be able to clearly hear the human voice dialogue in the video, reducing the user experience. SUMMARY

[0003] The present application provides a signal processing method for enhancing human voice and an electronic device, which can dynamically adjust the gain of the human voice signal according to the environmental noise condition for the input multi-channel audio signal, achieve the effect of enhancing human voice, and make the output human voice stable, thereby avoiding the phenomenon of large and small on the hearing.

[0004] In a first aspect, the present application provides a signal processing method for enhancing human voice, which comprises: acquiring an mth frame of audio signal in a multi-channel audio stream; calculating a normalized spectral flux corresponding to the mth frame of audio signal; acquiring a human voice frequency domain signal and a background sound frequency domain signal from the mth frame of audio signal; determining a human voice gain parameter according to the environmental noise, the background sound frequency domain signal and the human voice frequency domain signal; and adjusting the gain of the human voice frequency domain signal according to the human voice gain parameter and the normalized spectral flux to obtain an enhanced human voice frequency domain signal.

[0005] Through the signal processing method for enhancing human voice provided by the present application, in the scene of playing audio and video through an electronic device, the human voice gain parameter can be determined according to the environmental noise and the background sound frequency domain signal and the human voice frequency domain signal corresponding to the multi-channel audio stream, and then the gain of the human voice frequency domain signal is adjusted according to the human voice gain parameter and the normalized spectral flux to obtain an enhanced human voice frequency domain signal. In different environmental noise conditions, the gain of the human voice signal can be dynamically adjusted for the input multi-channel audio signal to achieve the effect of enhancing human voice, and the output human voice is stable, thereby avoiding the phenomenon of large and small on the hearing. Therefore, through the present application, the effect of being able to clearly hear the human voice of the video when playing the video in a quiet environment without disturbing others, or the effect of being able to clearly hear the human voice of the video when playing the video in a noisy environment can be achieved, which can improve the user experience.

[0006] It should be noted that the traditional Wiener filtering algorithm significantly improves the gain of signals with high signal spectrum intensity (referred to as large signals), but does not significantly improve the gain of signals with low signal spectrum intensity (referred to as small signals). It can be understood that in this case, the frequency spectrum range of the human voice signal is relatively large. The frequency spectrum range of the human voice signal refers to the frequency spectrum range formed by the small signals and the large signals. That is, through the traditional Wiener filtering algorithm, the gain of the large signals is significantly improved, and the gain of the small signals is too small, which may cause the phenomenon of large and small on the final listening. In order to solve the problem of large and small on the listening, the embodiment of the present application proposes an improved method of dynamically adjusting the gain of the human voice.

[0007] In some possible implementation manners, the determination of the human voice gain parameter according to the environmental noise, the background sound frequency domain signal and the human voice frequency domain signal includes: determining a first gain adjustment coefficient and a second gain adjustment coefficient according to the average amplitude of the environmental noise; and calculating the human voice gain parameter according to the first gain adjustment coefficient, the second gain adjustment coefficient, the background sound frequency domain signal and the human voice frequency domain signal by using the following equation:

[0008] G'(m, k) = p(1-G(m, k))+qG(m, k)

[0009]

[0010] SNR(m, k) = C(m, k) / S(m, k)

[0011] Wherein, G'(m, k) represents the human voice gain parameter, p represents the first gain adjustment coefficient, q represents the second gain adjustment coefficient, C(m, k) represents the human voice frequency domain signal, and S(m, k) represents the background sound frequency domain signal.

[0012] It can be understood that the human voice gain estimation formula used in the present application scheme is a formula improved based on the traditional Wiener filtering formula. Through the human voice gain estimation formula used in the present application scheme, the gain of human voice signals with different intensities in the frequency spectrum range of the human voice signal can be adjusted.

[0013] Through the embodiment scheme of the present application, the values of the two coefficients p and q can be dynamically adjusted according to the current environmental noise level, so that the gain of the large signals and the gain of the small signals can be adjusted, or the gain of the large signals and the gain of the small signals can be adjusted at the same time, which can avoid the phenomenon of large and small on the listening. By dynamically adjusting the gain of the human voice, the effect of being able to hear the human voice of the video clearly without disturbing others when playing the video in a quiet environment can be achieved, and the effect of being able to hear the human voice of the video clearly when playing the video in a noisy environment can be achieved.

[0014] Through the scheme, in a quiet environment, the values of the two coefficients p and q are dynamically adjusted, the gain of a large signal is appropriately increased, the gain of a small signal is appropriately increased, even when the volume is low when the electronic device plays a video, it can be ensured that the user can clearly hear the strong voice dialogue in the video, and can also clearly hear the weak voice dialogue in the video. Thus, when playing a video in a quiet environment, the user can clearly hear the voice of the video without disturbing others.

[0015] Through the scheme, in a noisy environment, the values of the two coefficients p and q are dynamically adjusted, the gain of a large signal is appropriately increased, the gain of a small signal is appropriately increased, the user can clearly hear the strong voice dialogue in the video, and can also clearly hear the weak voice dialogue in the video. Thus, when playing a video in a noisy environment, the user can clearly hear the voice of the video.

[0016] Compared with only increasing the gain of a large signal in the related art, the scheme appropriately increases the gain of a small signal while increasing the gain of a large signal, so as to compress the dynamic range of the voice signal, thereby avoiding the phenomenon of large and small in hearing.

[0017] The following describes possible cases of specific values of the first gain adjustment coefficient p and the second gain adjustment coefficient q.

[0018] In some possible implementation manners, the determination of the first gain adjustment coefficient and the second gain adjustment coefficient according to the average amplitude of the environmental noise includes:

[0019] When N∈[a,b), p=0.02×N-0.8, q=1;

[0020] When N∈[b,c), p=0.15×N-9.25, q=-0.04×N+3.6;

[0021] Wherein, N represents the average amplitude of the environmental noise, in decibels dB, a is less than b, and b is less than c.

[0022] In some possible implementation manners, a is 40 dB, b is 65 dB, and c is 75 dB.

[0023] Through the scheme, the values of p and q can be determined according to the average amplitude N of the environmental noise based on the linear mapping relationship between p, q and the average amplitude of the environmental noise.

[0024] It should be noted that the linear mapping relationship between p, q and the average amplitude of the ambient noise is exemplary, and the application does not limit the linear mapping relationship between p, q and the average amplitude of the ambient noise. It can be understood that in actual implementation, the corresponding relationship between p, q and the average amplitude of the ambient noise can be set according to actual use requirements.

[0025] In some possible implementation manners, the method provided in the application is applied to an electronic device including a microphone. The method further includes: determining an ambient noise signal according to an audio signal collected by the microphone; and determining an average amplitude of the ambient noise according to the ambient noise signal, a sampling time length, a sampling frequency, and a minimum sound pressure intensity.

[0026] Through the scheme of the application, the microphone of the electronic device (such as a mobile phone) is called regularly to record ambient noise signals, and then the noise level of the current environment of the electronic device is estimated by analyzing the ambient noise signals. Considering power consumption, the microphone is called every x minutes to collect y seconds of data. Exemplarily, the microphone is called every 30 minutes to collect 10 seconds of data.

[0027] In some possible implementation manners, the electronic device further includes a loudspeaker, and the method of determining the ambient noise signal according to the audio signal collected by the microphone includes: when the loudspeaker is working, determining the ambient noise signal according to the audio signal collected by the microphone and the audio signal played by the loudspeaker; and when the loudspeaker is not working, determining the audio signal collected by the microphone as the ambient noise signal.

[0028] It should be noted that when the microphone collects the ambient noise signal, if the electronic device is playing a video, the video sound is played through the loudspeaker, and then the video sound is propagated through the environment, received by the microphone and recorded. That is, the microphone recording result includes both the ambient noise and the sound played by the loudspeaker. Therefore, the signal played by the loudspeaker needs to be eliminated, and then the noise level of the current environment is estimated to improve the accuracy of calculating the ambient noise.

[0029] In some possible implementation manners, the method of determining the ambient noise signal according to the audio signal collected by the microphone and the audio signal played by the loudspeaker includes:

[0030] The following equation is used to calculate the ambient noise signal according to the audio signal collected by the microphone and the audio signal played by the loudspeaker: s(t) = d(t) - x(t) x S spk x S mic . Wherein s(t) represents the ambient noise signal, d(t) represents the audio signal collected by the microphone, x(t) represents the audio signal played by the loudspeaker, and S spk represents the sensitivity of the loudspeaker, Smic This indicates the microphone's sensitivity.

[0031] According to the solution proposed in this application, if the speaker is not working (e.g., no audio or video is playing) when the microphone is collecting ambient noise signals, the average amplitude of the data collected within y seconds can be directly calculated as the result of the ambient noise estimation. If the speaker is working when the microphone is collecting ambient noise signals, the current ambient noise situation can be estimated based on the sensitivity of the speaker and the sensitivity of the microphone, which can improve the accuracy of the ambient noise calculation.

[0032] In some possible implementations, the determination of the average amplitude of the environmental noise based on the environmental noise signal, sampling duration, sampling frequency, and minimum sound pressure level includes:

[0033] The average amplitude of the environmental noise is calculated using the following equation, based on the environmental noise signal, sampling duration, sampling frequency, and minimum sound pressure level:

[0034]

[0035] Where N represents the average amplitude of the environmental noise, s(t) represents the environmental noise signal, T represents the acquisition duration, F represents the sampling frequency, and P0 represents the minimum sound pressure level.

[0036] In some possible implementations, the above-mentioned gain adjustment of the human voice frequency domain signal based on the human voice gain parameter and normalized spectral flux to obtain an enhanced human voice frequency domain signal includes:

[0037] The enhanced human voice signal spectrum is calculated using the following equation, based on the human voice gain parameter and the normalized spectral flux:

[0038] C′(m,k)=C(m,k)+C(m,k)*G′(m,k)*V(m)

[0039] Where C′(m,k) represents the enhanced human voice frequency domain signal, C(m,k) represents the human voice frequency domain signal, G′(m,k) represents the human voice gain parameter, and V(m) represents the normalized spectral flow corresponding to the m-th frame audio signal.

[0040] This application's solution adjusts the gain of the human voice frequency domain signal based on the human voice gain parameters and normalized spectral flow, resulting in an enhanced human voice frequency domain signal. By dynamically adjusting the human voice gain, it is possible to achieve the effect of clearly hearing the human voice in a quiet environment without disturbing others, and to ensure that users can clearly hear the human voice in a noisy environment. Furthermore, it avoids fluctuations in perceived volume, thus improving the user experience.

[0041] In some possible implementation manners, the obtaining of the vocal frequency domain signal from the mth frame of audio signal includes:

[0042] When the mth frame of audio signal is a two-channel stereo signal, the vocal frequency domain signal is obtained from the mth frame of audio signal by using the following equation:

[0043] C(m, k) = ψ(m, k) × [F L (m, k) + F R (m, k)] / 2

[0044]

[0045]

[0046]

[0047]

[0048] F L (m, k) = STFT(L)

[0049] F R (m, k) = STFT(R)

[0050] Wherein, C(m, k) represents the vocal frequency domain signal, L represents the left channel time domain signal, F L (m, k) represents the left channel frequency domain signal, represents the conjugate function of the left channel frequency domain signal, R represents the right channel time domain signal, F R (m, k) represents the right channel frequency domain signal, represents the conjugate function of the right channel frequency domain signal; wherein, STFT() represents a short-time Fourier transform function. Wherein, φ LR (m, k) represents the cross-correlation function between the left channel frequency domain signal and the right channel frequency domain signal, φ LL (m, k) represents the autocorrelation function of the left channel frequency domain signal, φ RR (m, k) represents the autocorrelation function of the right channel frequency domain signal; wherein, ψ(m, k) represents a similarity function between the mth frame of left channel frequency domain signal and the right channel frequency domain signal.

[0051] In some possible implementation manners, the method provided by the application further includes: obtaining the optimized ψ(m, k) by using the following equation:

[0052] ψ(m, k) = (1-α)ψ(m-1, k) + αψ(m, k)

[0053] Wherein, a represents the inter-frame smoothing coefficient, a e (0, 1) ; wherein, ψ (m, k) represents the similarity function between the left channel frequency domain signal and the right channel frequency domain signal of the mth frame, and ψ (m-1, k) represents the similarity function between the left channel frequency domain signal and the right channel frequency domain signal of the (m-1) th frame.

[0054] In some possible implementation manners, the above-mentioned obtaining the background sound frequency domain signal from the mth frame audio signal comprises:

[0055] When the mth frame audio signal is a two-channel stereo signal, the following equation is used to obtain the background sound frequency domain signal from the mth frame audio signal:

[0056] s (t) = L-R

[0057] S (m, k) = STFT [s (t) ]

[0058] Wherein, L and R represent the left channel time domain signal and the right channel time domain signal in the mth frame audio signal respectively, and s (t) and S (m, k) represent the background sound time domain signal and the background sound frequency domain signal corresponding to the mth frame audio signal respectively.

[0059] In some possible implementation manners, the above-mentioned obtaining the vocal frequency domain signal from the mth frame audio signal comprises: when the mth frame audio signal is a 5.1 channel or 7.1 channel stereo signal, performing short-time Fourier transform on the center channel time domain signal in the mth frame audio signal to obtain the vocal frequency domain signal.

[0060] In some possible implementation manners, the above-mentioned obtaining the background sound frequency domain signal from the mth frame audio signal comprises: when the mth frame audio signal is a 5.1 channel stereo signal, the following equation is used to obtain the background sound frequency domain signal from the mth frame audio signal:

[0061] s (t) = (L+Ls)-(R+Rs)

[0062] S (m, k) = STFT [s (t) ]

[0063] Wherein, L, Ls, R, Rs represent the front left channel time domain signal, the side surround left channel time domain signal, the front right channel time domain signal, and the side surround right channel time domain signal in the mth frame audio signal respectively; s (t) and S (m, k) represent the background sound time domain signal and the background sound frequency domain signal corresponding to the mth frame audio signal respectively.

[0064] When the mth frame audio signal is a 7.1 channel stereo signal, the following equation is used to obtain the background sound frequency domain signal from the mth frame audio signal:

[0065] s (t) = (L+Ls+Lb)-(R+Rs+Rb)

[0066] S(m, k) = STFT [s(t)]

[0067] wherein L, Ls, Lb, R, Rs, Rb represent front left channel time domain signal, side surround left channel time domain signal, back left channel time domain signal, front right channel time domain signal, side surround right channel time domain signal, back right channel time domain signal in the mth frame of audio signal respectively; s(t) and S(m, k) represent background sound time domain signal and background sound frequency domain signal corresponding to the mth frame of audio signal respectively.

[0068] In some possible implementation manners, the calculating of the normalized spectral flux corresponding to the mth frame of audio signal comprises:

[0069] The normalized spectral flux V(m) corresponding to the mth frame of audio signal is calculated by using the following equation:

[0070]

[0071]

[0072]

[0073] C(m, k) = STFT [c(t)]

[0074] S(m, k) = STFT [s(t)]

[0075] wherein STFT() represents a short-time Fourier transform function. Wherein c(t) and C(m, k) represent voice time domain signal and voice frequency domain signal corresponding to the mth frame of audio signal respectively, C(m-1, k) represents voice frequency domain signal corresponding to the (m-1)th frame of audio signal, δ C (m, k) represents spectral flux of voice frequency domain signal. Wherein s(t) and S(m, k) represent background sound time domain signal and background sound frequency domain signal corresponding to the mth frame of audio signal respectively, S(m-1, k) represents background sound frequency domain signal corresponding to the (m-1)th frame of audio signal, δ S (m, k) represents spectral flux of background sound frequency domain signal.

[0076] In some possible implementation manners, the method provided by the present application further comprises: when the mth frame of audio signal is a two-channel stereo signal, c(t) and s(t) are calculated according to the following equation:

[0077]

[0078] s(t) = L - R

[0079] wherein L and R represent left channel time domain signal and right channel time domain signal in the mth frame of audio signal respectively, and β is a constant.

[0080] When the mth frame audio signal is a 5.1 channel signal, c(t) and s(t) are calculated according to the following equations:

[0081] c(t) = C

[0082] s(t) = (L + Ls) - (R + Rs)

[0083] wherein C, L, Ls, R, Rs represent the middle channel time domain signal, the front left channel time domain signal, the side surround left channel time domain signal, the front right channel time domain signal, the side surround right channel time domain signal in the mth frame 5.1 channel signal respectively;

[0084] When the mth frame audio signal is a 7.1 channel signal, c(t) and s(t) are calculated according to the following equations:

[0085] c(t) = C

[0086] s(t) = (L + Ls + Lb) - (R + Rs + Rb)

[0087] wherein C, L, Ls, Lb, R, Rs, Rb represent the middle channel time domain signal, the front left channel time domain signal, the side surround left channel time domain signal, the back left channel time domain signal, the front right channel time domain signal, the side surround right channel time domain signal, the back right channel time domain signal in the mth frame 7.1 channel signal respectively.

[0088] In some possible implementation manners, after the gain adjustment is performed on the vocal frequency domain signal according to the vocal gain parameter and the normalized spectral flux to obtain the enhanced vocal frequency domain signal, the method further includes: performing downmix processing on the enhanced vocal frequency domain signal to obtain a vocal enhanced stereo frequency domain signal.

[0089] By performing downmix processing on the enhanced vocal frequency domain signal and the multi-channel signal to obtain the vocal enhanced stereo frequency domain signal, and further converting the vocal enhanced stereo frequency domain signal into a vocal enhanced stereo time domain signal, a signal supported by a loudspeaker or a headphone is converted.

[0090] In some possible implementation manners, the downmix processing performed on the enhanced vocal frequency domain signal to obtain the vocal enhanced stereo frequency domain signal includes:

[0091] When the mth frame audio signal is a two-channel stereo signal, the downmix processing performed on the enhanced vocal frequency domain signal to obtain the vocal enhanced stereo frequency domain signal is according to the following equations:

[0092] F′ L = F L + C′(m, k)

[0093] F′ R = F R + C′(m, k)

[0094] wherein F′ L and F′ R represent the voice-enhanced stereo frequency domain signal; wherein C′(m, k) represents the enhanced voice frequency domain signal, F L and F R represent the frequency domain signals corresponding to the left / right channel time domain signals in the mth frame of audio signal.

[0095] In some possible implementation manners, the down-mix processing according to the enhanced voice frequency domain signal to obtain the voice-enhanced stereo frequency domain signal comprises:

[0096] When the mth frame of audio signal is a 5.1 channel stereo signal, the down-mix processing according to the enhanced voice frequency domain signal to obtain the voice-enhanced stereo frequency domain signal is according to the following equation:

[0097] F′ L = 0.4143*(F L + 0.707C′(m, k) + 0.707F Ls )

[0098] F′ R = 0.4143*(F R + 0.707C′(m, k) + 0.707F Rs )

[0099] wherein F′ L and F′ R represent the voice-enhanced stereo frequency domain signal; wherein C′(m, k) represents the enhanced voice frequency domain signal, F L , F R , F Ls , F Rs represent the frequency domain signals corresponding to the left channel, right channel, side surround left channel and side surround right channel time domain signals in the mth frame of 5.1 channel signal respectively.

[0100] In some possible implementation manners, the down-mix processing according to the enhanced voice frequency domain signal to obtain the voice-enhanced stereo frequency domain signal comprises:

[0101] When the mth frame of audio signal is a 7.1 channel stereo signal, the down-mix processing according to the enhanced voice frequency domain signal to obtain the voice-enhanced stereo frequency domain signal is according to the following equation:

[0102] F′ L= 0.3204 * (F L + 0.707C'(m, k) + 0.707F Ls + 0.707F Lb )

[0103] F' R = 0.3204 * (F R + 0.707C'(m, k) + 0.707F Rs + 0.707F Rb )

[0104] wherein, F' L and F' R represent the voice-enhanced stereo frequency domain signal respectively; wherein, C'(m, k) represents the enhanced voice frequency domain signal, F L , F R , F Ls , F Rs , F Lb , F Rb represent the frequency domain signal corresponding to the left channel, the right channel, the side surround left channel, the side surround right channel, the rear left channel, the rear right channel time domain signal of the mth frame 7.1 channel signal respectively.

[0105] In some possible implementation manners, after the above downmix processing according to the enhanced voice frequency domain signal to obtain the voice-enhanced stereo frequency domain signal, the method provided by the present application further includes: converting the voice-enhanced stereo frequency domain signal into a voice-enhanced stereo time domain signal through a Fourier transform algorithm; outputting the voice-enhanced stereo time domain signal through a sound output device; wherein, the sound output device is a loudspeaker or a headset.

[0106] The scheme provided by the present application can process multi-channel audio signals such as dual-channel stereo, 5.1 channel and 7.1 channel, and finally output voice-enhanced audio signals by dynamically adjusting the voice gain, so as to achieve the effect that the user can hear the voice of the video clearly in a noisy environment when playing the video.

[0107] In some possible implementation manners, before the above obtaining the voice frequency domain signal and the background sound frequency domain signal from the mth frame audio signal, the method provided by the present application further includes: judging whether the normalized spectral flux is greater than a first preset threshold; when the normalized spectral flux is greater than the first preset threshold, determining that the mth frame audio signal contains speech components.

[0108] It should be noted that whether the mth audio signal contains speech components can be determined by judging whether the normalized spectral flux is greater than a first preset threshold, and then whether the mth multi-channel audio signal needs to be subjected to the signal processing of enhancing human voice is determined according to whether the mth audio signal contains speech components.

[0109] In one aspect, if the mth audio signal contains speech components, the mth multi-channel audio signal needs to be subjected to the signal processing of enhancing human voice. The signal processing of enhancing human voice includes: obtaining a human voice frequency domain signal and a background sound frequency domain signal from the mth audio signal; determining a human voice gain parameter according to the environmental noise, the background sound frequency domain signal and the human voice frequency domain signal; and performing gain adjustment on the human voice frequency domain signal according to the human voice gain parameter and the normalized spectral flux to obtain an enhanced human voice frequency domain signal.

[0110] In another aspect, if the mth audio signal does not contain speech components, the mth multi-channel audio signal does not need to be subjected to the signal processing of enhancing human voice.

[0111] Specifically, after judging whether the normalized spectral flux is greater than the first preset threshold, the method provided by the present application further includes: when the normalized spectral flux is less than or equal to the first preset threshold, determining that the mth audio signal does not contain speech components, and in this case, the mth multi-channel audio signal does not need to be subjected to the signal processing of enhancing human voice, but can be subjected to normal downmix processing and time-frequency conversion processing / frequency-time conversion processing.

[0112] For example, in the case that the mth audio signal is a stereo signal, the mth stereo time domain signal without human voice enhancement is output through a sound output device; in the case that the mth audio signal is a 5.1 / 7.1 channel signal, the mth 5.1 / 7.1 channel signal is subjected to downmix processing to obtain the mth stereo frequency domain signal without human voice enhancement; the stereo frequency domain signal without human voice enhancement is converted into the mth stereo time domain signal without human voice enhancement, and the mth stereo time domain signal without human voice enhancement is output through the sound output device; wherein the sound output device is a loudspeaker or a headset.

[0113] In a second aspect, the present application provides a signal processing device for enhancing human voice, which comprises units for executing the method in the first aspect described above. The device can correspond to the method described in the first aspect described above, and the related description of the units in the device is referred to the description of the first aspect described above, and will not be repeated here for brevity.

[0114] The method described in the first aspect described above can be implemented by hardware, or can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, processing modules or units, display modules or units, etc.

[0115] In a third aspect, the present application provides an electronic device, comprising a processor, and a computer program or instructions stored in the memory and readable by the processor, wherein the processor is configured to execute the computer program or instructions so that the method in the first aspect is performed.

[0116] In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program (also referred to as instructions or code) for implementing the method in the first aspect. For example, when the computer program is executed by a computer, the computer can perform the method in the first aspect.

[0117] In a fifth aspect, the present application provides a chip comprising a processor. The processor is configured to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation manner thereof. Optionally, the chip further comprises the memory, and the memory is connected to the processor through a circuit or a wire.

[0118] In a sixth aspect, the present application provides a chip system comprising a processor. The processor is configured to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation manner thereof. Optionally, the chip system further comprises the memory, and the memory is connected to the processor through a circuit or a wire.

[0119] In a seventh aspect, the present application provides a computer program product comprising a computer program (also referred to as instructions or code), which, when executed by an electronic device, causes the electronic device to implement the method in the first aspect.

[0120] It can be understood that the beneficial effects of the above-mentioned second aspect to seventh aspect can be referred to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0121] Figure 1 An application scenario diagram of the signal processing method for enhancing human voice provided by an embodiment of the present application;

[0122] Figure 2 An application scenario analysis diagram of the signal processing method for enhancing human voice provided by an embodiment of the present application;

[0123] Figure 3 A processing flow diagram of various channel audio signals in the related art;

[0124] Figure 4 A flow diagram of processing an audio signal by an electronic device in the related art;

[0125] Figure 5A signal processing flowchart provided by an embodiment of the present application when an electronic device plays a video scene;

[0126] Figure 6 A processing flowchart of various channel audio signals provided by an embodiment of the present application;

[0127] Figure 7 A structural diagram of an electronic device provided by an embodiment of the present application;

[0128] Figure 8 A software architecture diagram of an electronic device provided by an embodiment of the present application;

[0129] Figure 9 A flowchart of a signal processing method for enhancing vocals provided by an embodiment of the present application;

[0130] Figure 10 A module interaction timing diagram of a signal processing method for enhancing vocals provided by an embodiment of the present application;

[0131] Figure 11 Another flowchart of a signal processing method for enhancing vocals provided by an embodiment of the present application;

[0132] Figure 12 A waveform diagram when a signal processing method for enhancing vocals provided by an embodiment of the present application detects a vocal signal and extracts the vocal signal for a two-channel stereo signal;

[0133] Figure 13 A diagram of a signal processing method for enhancing vocals provided by an embodiment of the present application in which a microphone collects environmental noise and a loudspeaker plays a sound;

[0134] Figure 14 A waveform diagram of a signal processing method for enhancing vocals provided by an embodiment of the present application in which a loudspeaker plays a sound;

[0135] Figure 15 A waveform diagram of a signal processing method for enhancing vocals provided by an embodiment of the present application in which a microphone collects a signal;

[0136] Figure 16 A curve mapping relationship diagram between coefficients p and q and an environmental noise level in a signal processing method for enhancing vocals provided by an embodiment of the present application;

[0137] Figure 17 A structural diagram of a signal processing device for enhancing vocals provided by an embodiment of the present application. DETAILED DESCRIPTION

[0138] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0139] The term "and / or" used in the present application is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The symbol " / " in the present application means that the associated objects are or, for example, A / B means A or B.

[0140] The terms "first" and "second" and the like in the description and claims of the present application are used to distinguish different objects, and are not used to describe a specific order of the objects. In the description of the embodiments of the present application, unless otherwise specified, "a plurality of" means two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0141] At present, users often play video resources (such as films and television dramas) through electronic devices with loudspeakers such as mobile phones, tablet computers, notebook computers and smart screens. Figure 1 As shown in (a) of FIG. 1, a user watches a video through a mobile phone in a noisy environment. Figure 1 As shown in (b) of FIG. 1, a user watches a video through a smart television in a quiet environment.

[0142] However, in the process of playing a video through an electronic device, the human voice dialogue in the video is drowned out by various factors in different environments, resulting in that the user cannot hear the human voice dialogue in the video. As shown in (a) of FIG. 2, when a video is played, the human voice dialogue in the video is covered by external environmental sound in a noisy environment, resulting in that the user cannot hear the human voice dialogue in the video, even if the volume is increased. As shown in (b) of FIG. 2, in a quiet environment, the volume is reduced when a video is played to avoid disturbing others, at this time, the human voice dialogue is covered by other sounds in the video, resulting in that the user cannot hear the human voice dialogue in the video. Figure 2 Figure 2

[0143] ​​To solve the above problems, an embodiment of the present application provides a signal processing method for enhancing human voice based on digital signal processing, which is applied to an electronic device. The human voice enhancement algorithm provided by the present application is used to process the input multi-channel audio signal, which can improve the intelligibility of human voice and solve the above problems to some extent, thereby improving the user experience.

[0144] To facilitate understanding of the embodiments of the present application, some terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0145] Channel: A channel generally refers to an independent audio signal channel collected or played back at different spatial positions in audio production and playback. The number of channels can be different for different audio / video sources. For example, the currently commonly used multi-channel audio signal adopts dual channels, or 5.1 channels, or 7.1 channels. In the embodiments of the present application, the audio / video source is collectively referred to as an audio source, and the form of the audio source is an audio stream, which includes continuous multi-frame multi-channel audio signals.

[0146] Stereo sound: In sound technology, stereo sound is a stereo surround sound effect obtained through a recording, transmission and playback system. For a dual-channel stereo sound, a 5.1 channel, and a 7.1 channel audio source, a stereo sound effect can be produced when played through a speaker or a headset of an electronic device.

[0147] Multi-channel signal downmixing: A downmixing transfer function is used to process a multi-channel signal, so that the number of output channels is less than the number of input channels. This signal processing is called downmixing processing. In the related art, binaural rendering processing of dual channels to binaural audio is also called downmixing processing.

[0148] Dual channel: As shown in (a) of FIG. 1, a dual channel includes a left channel and a right channel. In the related art, when an electronic device receives a dual-channel audio source, the electronic device outputs a dual-channel stereo sound signal through a speaker or a headset. Figure 3

[0149] 5.1 channel: As shown in (b) of FIG. 1, a 5.1 channel includes five channels, i.e., a central channel C, a front left channel L, a front right channel R, a side surround left channel Ls, and a right surround channel Rs. In the related art, when an electronic device receives a 5.1 channel audio source, the electronic device performs downmixing processing on the five channel signals to obtain a dual-channel stereo sound signal, and then plays the dual-channel stereo sound signal through a speaker or a headset. Figure 3

[0150] 7.1 channel: As shown in (c) of FIG. 1, a 7.1 channel includes seven channels, i.e., a central channel C, a front left channel L, a front right channel R, a side surround left channel Ls, a side surround right channel Rs, a back surround left channel Lr, and a back surround right channel Rr. In the related art, when an electronic device receives a 7.1 channel audio source, the electronic device performs downmixing processing on the seven channel signals to obtain a dual-channel stereo sound signal, and then plays the dual-channel stereo sound signal through a speaker or a headset. Figure 3 ​​As shown in (c) in FIG. 7.1 channel includes seven channels, that is, a central channel C, a front left channel L, a front right channel R, a side surround left channel Ls, a right surround channel Rs, a back left channel Lb, and a back right channel Rb. In the related art, when an electronic device receives an audio source of a 7.1 channel, the electronic device performs downmix processing on the seven channel signals to obtain a two-channel stereo signal, and then plays the two-channel stereo signal through a speaker or a headset.

[0151] It should be noted that the signal processing method for enhancing human voice provided in the embodiments of the present application can be applied to processing the above-mentioned two-channel audio source, and can also process a 5.1 channel audio source or a 7.1 channel audio source. It can be understood that in actual implementation, the signal processing method for enhancing human voice provided in the embodiments of the present application can also be applied to processing other multi-channel audio sources. In order to facilitate the description, the signal processing method for enhancing human voice provided in the embodiments of the present application will be exemplarily described below by taking processing a two-channel audio source, a 5.1 channel audio source, and a 7.1 channel audio source as examples.

[0152] In order to facilitate the improvement points of the embodiments of the present application relative to the related art, the audio processing strategy of the electronic device in the related art when playing a video will be described first. Figure 4 The audio output strategy of the electronic device in the related art when playing a video is shown. Exemplarily, the functional modules used in the related art include software modules of the electronic device such as an audio / video application and a downmix module, and hardware modules of the electronic device such as a speaker / headset.

[0153] S101, the audio / video application receives an operation triggered by a user to play a video.

[0154] S102, in response to the user operation, the audio / video application acquires an audio stream.

[0155] S103, the audio / video application determines whether the acquired audio stream is a two-channel audio stream.

[0156] S104, when the acquired audio stream is a two-channel audio stream, the audio / video application sends the two-channel audio stream to the speaker / headset.

[0157] S105, the speaker / headset outputs the two-channel audio stream frame by frame.

[0158] S106, when the audio stream is not a two-channel audio stream, the audio / video application sends the acquired audio stream to the downmix module.

[0159] S107, the downmix module performs downmix processing on the multi-channel audio stream to obtain a two-channel audio stream.

[0160] For ease of illustration, an audio stream with a number of channels greater than 2 is referred to as a multi-channel audio stream. Exemplarily, the multi-channel audio stream can be a 5.1 channel audio stream or a 7.1 channel audio stream.

[0161] In S108, the downmix module transmits the downmixed two-channel audio stream to the loudspeaker / earphone.

[0162] In S109, the loudspeaker / earphone outputs the two-channel audio stream frame by frame.

[0163] By Figure 4 It can be learned that in the related art, when the electronic device plays a two-channel or multi-channel audio stream, the human voice / speech in the audio stream is not processed for enhancement, and thus in a noisy environment, even if the volume is increased, the human voice in the audio stream can be covered by the surrounding noise, or in a quiet environment and in a scenario where the media volume of the electronic device is low, the human voice in the audio stream can be covered by the background sound in the audio stream, resulting in that the user cannot hear the human voice in the audio stream clearly.

[0164] In order to better understand the embodiments of the present application, the embodiments of the present application are briefly described as follows:

[0165] The present application aims to enable the user to hear the human voice dialogue in a video clearly without disturbing others in a quiet environment, and to enable the user to hear the human voice dialogue in a video clearly by resisting the environmental noise in a noisy environment.

[0166] The signal processing method for enhancing human voice provided by the embodiments of the present application can be applied to an audio output scenario in which the input signal is a multi-channel audio signal, for example, a video playing scenario. In a video playing scenario (for example, playing a video), the audio includes human voice dialogue between actors and background music, the human voice dialogue between the actors in the audio is collected to obtain a human voice signal, and then the human voice signal appearing in the video can be processed for enhancement by the present application, so that the user can clearly hear the human voice dialogue when the video is played.

[0167] It should be noted that in actual implementation, the signal processing method for enhancing human voice provided by the embodiments of the present application can also be applied to other more possible scenarios, for example, the embodiments of the present application can also be applied to a scenario of playing multi-channel music through an audio application, which is not limited by the embodiments of the present application. For ease of illustration, the signal processing scenario for enhancing human voice to which the present application is applied is described by taking the electronic device playing a video as an example.

[0168] For example, as shown in FIG. 1, a video is played on an electronic device, and the audio of the video includes human voice dialogue between actors and background music. Figure 5As shown, after the user triggers the electronic device to play the video, the electronic device first acquires the video file and places the video file in the memory buffer. The video file is encapsulated format data, including an audio stream and a video stream. The encapsulated format can be AVI, MOV, or MP4, and the video file can be understood as a container that accommodates various encoded data. After acquiring the video file, the electronic device performs encapsulated format processing on the video file to separate the video stream and the audio stream, and obtain audio compression data and video compression data. Further, the electronic device decodes the audio compression data to obtain audio sampling data, and decodes the video compression data to obtain video pixel data.

[0169] After obtaining the audio sampling data, the common signal processing manner in the related art is that the electronic device synchronously processes the audio sampling data and the video pixel data to obtain video data to be played. Finally, based on the video data to be played, a video picture is displayed on the display and video sound is output through the loudspeaker / earphone. In this case, when the video is played, the human voice dialogue in the video is covered due to the influence of ambient sound or noise such as background sound in the video, resulting in that the user cannot clearly hear the human voice dialogue in the video.

[0170] Unlike the related art, the signal processing step of enhancing the human voice is added in the present application. Specifically, the electronic device performs signal processing on the audio sampling data through a preset human voice enhancement algorithm to enhance the human voice. Then, the processed audio sampling data and the video pixel data are synchronously processed to obtain video data to be played. Finally, based on the video data to be played, a video picture is displayed on the display and video sound is output through the loudspeaker / earphone. In this case, since the signal enhancement processing is performed on the human voice in the video, even if there is the influence of ambient sound or noise such as background sound in the video, the user can clearly hear the human voice dialogue in the video when the video is played.

[0171] The above describes that the signal processing process of enhancing the human voice is added in the video playing scenario. Next, possible implementation manners of the embodiments of the present application for signal processing on different channel audio signals are described in combination with the drawings.

[0172] Figure 6 FIG. 1 shows a schematic diagram of the embodiments of the present application for signal processing on different channel audio signals.

[0173] As shown in FIG. 1, the electronic device receives a video file, and the video file is encapsulated format data, including an audio stream and a video stream. The encapsulated format can be AVI, MOV, or MP4, and the video file can be understood as a container that accommodates various encoded data. After acquiring the video file, the electronic device performs encapsulated format processing on the video file to separate the video stream and the audio stream, and obtain audio compression data and video compression data. Further, the electronic device decodes the audio compression data to obtain audio sampling data, and decodes the video compression data to obtain video pixel data. Figure 6As shown in (a) of FIG. 1, the audio stream is a two-channel stereo signal, and the two channels include a left channel (denoted as L) and a right channel (denoted as R). By performing a voice extraction process on the two-channel stereo signal, a voice signal is obtained, and then the voice signal is enhanced, and then the enhanced voice signal is down-mixed with a background sound signal, and then the down-mixed two-channel stereo signal is output through a loudspeaker or a headset.

[0174] As shown in (b) of FIG. 1, the audio stream is a 5.1-channel audio signal, and the 5.1 channels include a center channel (denoted as C), a left channel L, a right channel R, a left surround channel Ls, and a right surround channel Rs. Since the center channel C in the 5.1-channel audio signal is used to carry a voice signal, the center channel C is directly taken as the voice signal, and then the voice signal is enhanced, and then the enhanced voice signal is down-mixed with a background sound signal (i.e., a 4-channel signal), and then the down-mixed two-channel stereo signal is output through a loudspeaker or a headset. Figure 6 As shown in (c) of FIG. 1, the audio stream is a 7.1-channel audio signal, and the 7.1 channels include a center channel C, a left channel L, a right channel R, a left surround channel Ls, a right surround channel Rs, a back left channel (denoted as Lb), and a back right channel (denoted as Rb). Since the center channel C in the 7.1-channel audio signal is used to carry a voice signal, the center channel C is directly taken as the voice signal, and then the voice signal is enhanced, and then the enhanced voice signal is down-mixed with a background sound signal (i.e., a 6-channel signal), and then the down-mixed two-channel stereo signal is output through a loudspeaker or a headset.

[0175] Figure 6 The signal processing method for enhancing a voice provided in the embodiments of the present application can be applied to an electronic device with a loudspeaker or a headset playing function. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a notebook computer, an Ultra-mobile Personal Computer (UMPC), a handheld computer, a netbook, a Personal Digital Assistant (PDA), a wearable electronic device, a smart watch, and the like, and the specific form of the electronic device is not specially limited in the present application.

[0176] The signal processing method for enhancing a voice provided in the embodiments of the present application can be applied to an electronic device with a loudspeaker or a headset playing function. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a notebook computer, an Ultra-mobile Personal Computer (UMPC), a handheld computer, a netbook, a Personal Digital Assistant (PDA), a wearable electronic device, a smart watch, and the like, and the specific form of the electronic device is not specially limited in the present application. Figure 6 Compared with the related art described above, the signal processing process of extracting a voice signal and enhancing the voice signal is added in the embodiments of the present application. The specific implementation process of extracting a voice signal and enhancing the voice signal will be described in detail below. Figure 3 The signal processing method for enhancing a voice provided in the embodiments of the present application can be applied to an electronic device with a loudspeaker or a headset playing function. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a notebook computer, an Ultra-mobile Personal Computer (UMPC), a handheld computer, a netbook, a Personal Digital Assistant (PDA), a wearable electronic device, a smart watch, and the like, and the specific form of the electronic device is not specially limited in the present application.

[0177] The signal processing method for enhancing a voice provided in the embodiments of the present application can be applied to an electronic device with a loudspeaker or a headset playing function. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a notebook computer, an Ultra-mobile Personal Computer (UMPC), a handheld computer, a netbook, a Personal Digital Assistant (PDA), a wearable electronic device, a smart watch, and the like, and the specific form of the electronic device is not specially limited in the present application.

[0178] ​Figure 7 A structural schematic diagram of an electronic device is shown.

[0179] As shown in the figure, the electronic device includes a processor, a memory, a display screen, an audio module, a speaker, a receiver, a microphone, and a headphone interface. Figure 7

[0180] It can be understood that the structure shown in the embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0181] The memory can be used to store computer executable program code, which includes instructions. The processor invokes the instructions stored in the memory to perform various functional applications and data processing of the electronic device.

[0182] The processor can include one or more processing units, for example, in the present application, the processor executing instructions in the memory can implement the signal processing method for enhancing human voice provided by the embodiments of the present application.

[0183] The display screen is used to display images, videos, etc. The electronic device can include 1 or N display screens, where N is an integer greater than 1.

[0184] The electronic device can implement audio / video functions through the audio module, the speaker, the receiver, the microphone, the headphone interface, and the application processor, etc. For example, music playing, video playing, recording, etc. The electronic device can include 2 or M speakers, where M is a positive integer greater than 2.

[0185] For example, in the scenario where the user listens to music, watches videos, plays games, etc. using the speaker or the earphone, stereo sound playback can be achieved through at least two speakers.

[0186] When a person's ears listen to sound, they will process the sound information obtained on the left and right sides respectively, so as to distinguish the main direction from which the sound is transmitted. Different sounds transmitted through at least two speakers enable the user's two ears to read different sound information, thereby achieving stereo sound playback.

[0187] In addition, an operating system runs on the above components. For example, a Hongmeng operating system, an iOS operating system, an Android open source operating system, a Windows operating system, etc. Application programs can be installed and run on the operating system.

[0188] The embodiments of the present application take the Android system with a layered architecture as an example to exemplarily illustrate the software structure of the electronic device.​

[0189] Figure 8 is a software structure block diagram of an electronic device of an embodiment of the present application.

[0190] The layered architecture divides software into several layers, each of which has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, as shown in Figure 8 Android system can be divided into four layers from top to bottom, namely, an application layer, an application framework layer, a system library, and a kernel layer.

[0191] The application layer can include a series of application packages. In an embodiment of the present application, the application layer can include an audio and video application, which refers to an application program capable of providing an audio signal, such as music, video, games, browsers, instant messaging, etc.

[0192] The application framework layer provides an application programming interface (API) and a programming framework for the application programs of the application layer, which can also be referred to as a service (Service) layer. The application framework layer includes some pre-defined functions.

[0193] In an embodiment of the present application, the application framework layer includes an audio service module. The audio service module includes a human voice detection module, a human voice extraction module, and a channel number detection module.

[0194] The human voice detection module is used to detect whether the audio source contains a human voice signal, that is, whether the audio source contains a speech component.

[0195] The human voice extraction module is used to extract the human voice signal in the audio source.

[0196] The channel number detection module is used to detect the number of channels contained in the audio source. For example, the channel number detection module is used to determine whether the audio signal in the audio source is a two-channel stereo signal or a 5.1 channel or 7.1 channel audio signal.

[0197] For example, for a two-channel stereo signal, a human voice signal can be extracted from the two-channel stereo signal according to a preset human voice extraction algorithm.

[0198] For example, for a 5.1 channel or 7.1 channel audio signal, since the center channel is used to transmit a human voice signal, the signal transmitted by the center channel can be directly obtained.

[0199] The system library can include a plurality of functional modules. For example: a media library, a three-dimensional graphics processing library, a two-dimensional graphics engine, etc.

[0200] In the embodiment of the present application, the system library includes a signal processing module such as an ambient noise estimation module, a voice gain estimation module, and a downmix module.

[0201] The ambient noise estimation module is used to detect the noise level of the environment in which the electronic device is currently located, also referred to as the ambient noise level. The ambient noise level can be set according to actual use requirements, and the specific setting method will be described below.

[0202] The voice gain estimation module is used to adjust the gain of the voice signal extracted by the voice extraction module according to the ambient noise level, to enhance the voice signal.

[0203] The downmix module is used to downmix the enhanced voice signal and the background sound signal to obtain a two-channel stereo signal.

[0204] The kernel layer is the layer between hardware and software. In the embodiment of the present application, the kernel layer includes an audio driver and a display driver. The audio driver is used to trigger the loudspeaker and / or earphone to output the mixed audio signal.

[0205] For ease of illustration, in the Figure 8 The hardware layer that interacts with the above software structure is also embodied. In the embodiment of the present application, the hardware layer includes a display, a loudspeaker, and an earphone interface. The earphone interface is a physical interface provided on the electronic device for connecting an earphone.

[0206] It should be noted that, although the Android system is taken as an example for illustration in the embodiment of the present application, the basic principle is also applicable to electronic devices based on iOS or Windows operating systems.

[0207] The execution subject of the signal processing method for enhancing voice provided in the embodiment of the present application can be the electronic device described above, or a functional module and / or functional entity in the electronic device that can implement the signal processing method for enhancing voice, and the present application scheme can be implemented in a hardware and / or software manner, and the specific implementation can be determined according to actual use requirements, which is not limited in the embodiment of the present application. The signal processing method for enhancing voice provided in the embodiment of the present application will be exemplarily described below with reference to the electronic device and the accompanying drawings.

[0208] The signal processing method for enhancing voice provided in the embodiment of the present application will be described below in combination with specific embodiments.

[0209] Referring back to the above Figure 8As shown, the functional modules used in this application include audio and video applications, a voice detection module, a voice extraction module, a channel number detection module, an environmental noise estimation module, a voice gain estimation module, and a downmixing module. Compared to the functional modules used in related technologies, this application adds a voice detection module, a voice extraction module, a channel number detection module, an environmental noise estimation module, and a voice gain estimation module.

[0210] It should be noted that in the embodiments of this application, human voice can also be referred to as speech. For example, human voice detection can also be referred to as speech detection, human voice extraction can also be referred to as speech extraction, and human voice gain can also be referred to as speech gain.

[0211] exist Figure 8 On this basis, Figure 9 This illustration shows the interaction diagram of each functional module in the signal processing method for enhancing human voice provided in the embodiments of this application.

[0212] like Figure 9 As shown, the audio and video application acquires an audio stream, which includes multiple consecutive frames of multi-channel audio signals. It is assumed that the current frame signal acquired by the audio and video application is the m-th frame multi-channel audio signal, wherein the m-th frame multi-channel audio signal includes, but is not limited to, two-channel stereo signals, 5.1-channel and 7.1-channel audio signals.

[0213] The audio and video application will input the acquired m-th frame of multi-channel audio signal into the human voice detection module.

[0214] The voice detection module detects whether the multi-channel audio signal (current frame signal) in the m-th frame contains speech components.

[0215] On the one hand, if the detection result shows that the multi-channel audio signal of the m-th frame (current frame signal) contains speech components, then the multi-channel audio signal of the m-th frame is subjected to voice enhancement processing.

[0216] First, the channel number detection module determines whether the current frame signal is stereo or 5.1 / 7.1 channels.

[0217] Then, the voice extraction module extracts the voice signal from the multi-channel audio signal of the m-th frame and sends it to the voice gain estimation module.

[0218] If the current frame signal is two-channel stereo, then the human voice is extracted from the two-channel stereo signal using a human voice extraction algorithm, and the extracted human voice signal is sent to the human voice gain estimation module.

[0219] If the current frame signal is 5.1 / 7.1 channels, the channel number detection module will directly send the center channel of the 5.1 / 7.1 channels as the human voice signal to the human voice gain estimation module.

[0220] Then, the human voice gain estimation module receives the human voice signal sent by the human voice extraction module, and receives the ambient noise level sent by the ambient noise estimation module. The human voice gain estimation module adjusts the gain of the human voice signal according to the ambient noise level, and enhances the human voice. The human voice gain estimation module sends the enhanced human voice signal to the downmix module.

[0221] Then, the downmix module performs downmix processing on the enhanced human voice signal and the multi-channel audio signal, converts the multi-channel audio signal into a two-channel stereo signal, and then outputs the two-channel stereo signal through a speaker or a headset.

[0222] On the other hand, in the case where the detection result is that the mth multi-channel audio signal (current frame signal) does not contain a speech component, the mth multi-channel audio signal is not subjected to human voice enhancement processing.

[0223] The specific processing manner is that the mth multi-channel audio signal is input into the channel number detection module, and the channel number detection module detects the channel number of the mth multi-channel audio signal.

[0224] When it is detected that the mth multi-channel audio signal is a 5.1 channel and a 7.1 channel audio signal, the downmix module performs downmix processing on the 5.1 channel and the 7.1 channel audio signal, and converts the multi-channel audio signal into a two-channel stereo signal. Then the two-channel stereo signal is output through a speaker or a headset.

[0225] When it is detected that the mth multi-channel audio signal is a two-channel stereo signal, the two-channel stereo signal is output through a speaker or a headset.

[0226] The above Figure 9 The flowchart of the interaction of the various functional modules in the signal processing method for enhancing human voice provided by the embodiments of the present application is briefly described below, and the timing diagram of the interaction of the various functional modules is shown in Figure 10 The signal processing method for enhancing human voice provided by the embodiments of the present application is described in detail below with reference to the timing diagram of the interaction of the various functional modules.

[0227] First of all, it should be pointed out that the signal processing method for enhancing human voice provided by the embodiments of the present application can be applied to electronic devices such as mobile phones. The electronic devices such as mobile phones can correspondingly preset the human voice enhancement function. The electronic devices such as mobile phones start the human voice enhancement function, and run the human voice enhancement function when the conditions are met (such as when playing multi-channel audio), so as to achieve the effect of enhancing human voice. The human voice enhancement function can be system default enabled, or can support user-defined setting to be enabled or disabled. For example, a user setting entry of the human voice enhancement function is added in the electronic devices such as mobile phones, so that the user can enable or disable the human voice enhancement function according to the needs.

[0228] For example, Figure 10As shown, the signal processing method for enhancing human voice provided by the embodiments of the present application includes steps S201-S225.

[0229] S201, the audio-video application receives a user operation of triggering playing a video.

[0230] For example, the user operation of triggering playing a video can be an operation of the user clicking a video playing button, and can also be an operation of the user clicking a video application icon. Wherein, the video can refer to on-demand video or live video, and the video format can be regular video or short video. Wherein, the short video refers to a video with a playing time less than or equal to a preset time (such as 15 seconds).

[0231] S202, in response to the user operation, the audio-video application acquires an audio stream.

[0232] For example, in response to the operation of the user clicking the video playing button or the operation of the user clicking the video application icon, the audio-video application acquires an audio stream in a video source. Here it is assumed that the audio stream is a multi-channel audio signal such as a 5.1 / 7.1 channel audio signal.

[0233] It should be noted that the input multi-channel audio signal is a time domain signal, and the time domain signal needs to be Fourier transformed for time-frequency conversion to obtain a frequency domain signal. Then the frequency domain signal processing is performed to realize human voice detection, human voice extraction, and human voice enhancement.

[0234] S203, the audio-video application notifies a human voice detection module, a human voice extraction module, a channel number detection module, a human voice gain estimation module, an environmental noise estimation module, a downmix module, and a post-processing module to start a human voice enhancement algorithm.

[0235] Correspondingly, the human voice detection module, the human voice extraction module, the channel number detection module, the human voice gain estimation module, the environmental noise estimation module, the downmix module initialize and start the human voice enhancement algorithm.

[0236] S204, the audio-video application sends an nth frame of audio signal (referred to as current frame signal) in the audio stream to the human voice detection module.

[0237] It should be noted that the audio-video application sends the audio stream to the human voice detection module frame by frame.

[0238] S205, the human voice detection module detects whether the current frame signal contains speech component according to a preset human voice detection algorithm.

[0239] Wherein, the preset human voice detection algorithm is an algorithm provided by the embodiments of the present application for detecting whether the audio signal contains human voice signal, and the specific algorithm process will be described in detail below.

[0240] It should be noted that in the detection process, the time-domain multi-channel audio signal needs to be converted into a frequency-domain multi-channel audio signal by a time-frequency conversion algorithm, and then the voice detection is performed in the frequency domain. Details will be described below.

[0241] S206, in the case that the voice detection module detects that the current frame signal does not contain speech components, the voice detection module sends the current frame signal to the channel number detection module, and the channel number detection module detects whether the current frame signal is a dual-channel stereo signal.

[0242] On the one hand, if the current frame signal is a dual-channel stereo signal, S207-S208 described below are executed. On the other hand, if the current frame signal is not a dual-channel stereo signal, S209-S212 described below are executed.

[0243] S207, in the case that the channel number detection module detects that the current frame signal is a dual-channel stereo signal, the channel number detection module sends the dual-channel stereo signal to the post-processing module, and the post-processing module performs time-frequency conversion on the dual-channel stereo signal to obtain a time-domain dual-channel stereo signal, and then sends the time-domain dual-channel stereo signal to the loudspeaker / earphone.

[0244] S208, the loudspeaker / earphone outputs the time-domain dual-channel stereo signal.

[0245] S209, in the case that the channel number detection module detects that the current frame signal is not a dual-channel stereo signal, the channel number detection module sends the 5.1 / 7.1 channel audio signal to the downmix module.

[0246] S210, the downmix module downmixes the 5.1 / 7.1 channel audio signal into a dual-channel stereo signal.

[0247] S211, the dual-channel stereo signal is transmitted to the loudspeaker / earphone after time-frequency conversion.

[0248] Since the obtained dual-channel stereo signal after downmix processing is a frequency-domain signal, frequency-domain to time-domain conversion (time-frequency conversion) is needed.

[0249] Specifically, the downmix module sends the dual-channel stereo signal to the post-processing module, and the post-processing module performs time-frequency conversion on the dual-channel stereo signal to obtain a time-domain dual-channel stereo signal, and then the post-processing module sends the time-domain dual-channel stereo signal to the loudspeaker / earphone.

[0250] S212, the loudspeaker / earphone outputs the time-domain dual-channel stereo signal.

[0251] The above describes the audio signal processing flow in the case where it is detected that the current frame signal does not contain a speech component. The following describes the audio signal processing flow in the case where it is detected that the current frame signal contains a speech component.

[0252] S213, in the case where the human voice detection module detects that the current frame signal contains a speech component, the human voice detection module sends the current frame signal to the channel number detection module.

[0253] S214, the channel number detection module detects whether the current frame signal is a two-channel stereo.

[0254] S215, in the case where the channel number detection module detects that the current frame signal is a 5.1 / 7.1 channel, the channel number detection module sends the 5.1 / 7.1 channel to the human voice gain estimation module.

[0255] Among the 5.1 / 7.1 channels, the center channel is the human voice frequency domain signal; and the other channels except the center channel are the background sound frequency domain signals.

[0256] S216, in the case where the channel number detection module detects that the current frame signal is a two-channel stereo, the channel number detection module sends the two-channel stereo to the human voice extraction module.

[0257] S217, the human voice extraction module extracts the human voice frequency domain signal and the background sound frequency domain signal from the current frame signal according to a preset human voice extraction algorithm.

[0258] It should be noted that for a two-channel stereo signal, the human voice frequency domain signal and the background sound frequency domain signal can be extracted from the audio signal according to a preset human voice extraction algorithm. The preset human voice extraction algorithm is an algorithm provided by the embodiments of the present application for extracting a human voice signal from an audio signal, and the specific algorithm process will be described in detail below.

[0259] S218, the human voice extraction module sends the human voice frequency domain signal and the background sound frequency domain signal to the human voice gain estimation module.

[0260] At the same time, the environmental noise estimation module collects the noise of the current environment of the electronic device.

[0261] S219, the environmental noise estimation module collects the environmental noise, and determines the noise level of the current environment according to the environmental noise.

[0262] For example, when the average amplitude of the environmental noise is in the range of [40dB, 45dB], it is determined that the environmental noise level is low. When the average amplitude of the environmental noise is in the range of (45dB, 55dB], it is determined that the environmental noise level is medium. When the average amplitude of the environmental noise is in the range of (55dB, 65dB], it is determined that the environmental noise level is slightly high. When the average amplitude of the environmental noise is in the range of (65dB, 75dB], it is determined that the environmental noise level is very high.

[0263] In S220, the environmental noise estimation module sends the information of the noise level to the voice gain estimation module.

[0264] That is, the voice gain estimation module receives the information of the noise level sent by the environmental noise estimation module in addition to the voice frequency domain signal and the background sound frequency domain signal.

[0265] In S221, the voice gain estimation module adjusts the gain of the voice signal according to the noise level, the voice frequency domain signal and the background sound frequency domain signal, to obtain an enhanced voice signal.

[0266] In S222, the voice gain estimation module sends the background sound signal and the enhanced voice signal to the downmix module.

[0267] In S223, the downmix module downmixes the enhanced voice signal and the background sound signal into a two-channel stereo signal.

[0268] In S224, the two-channel stereo signal is converted from the frequency domain to the time domain, and then transmitted to the loudspeaker / earphone.

[0269] Since the two-channel stereo signal obtained after the downmix processing is a frequency domain signal, frequency-time conversion is needed. Specifically, the downmix module sends the two-channel stereo signal to the post-processing module, the post-processing module performs frequency-time conversion on the two-channel stereo signal to obtain a time-domain two-channel stereo signal, and then the post-processing module sends the time-domain two-channel stereo signal to the loudspeaker / earphone.

[0270] In S225, the loudspeaker / earphone outputs the two-channel stereo signal.

[0271] The signal processing method for enhancing voice provided by the embodiments can be compatible with processing multi-channel audio signals such as two-channel stereo, 5.1 channel and 7.1 channel, and can output an audio signal with enhanced voice by dynamically adjusting the voice gain, so as to achieve the effect that the voice of the video can be heard clearly without disturbing others when playing the video in a quiet environment, and the effect that the voice of the video can be heard clearly by the user when playing the video in a noisy environment.

[0272] The above describes the overall process of the signal processing method for enhancing human voice provided by the embodiments of the present application. The following describes each step of the signal processing method for enhancing human voice provided by the embodiments of the present application, including detecting a human voice signal, extracting a human voice signal, detecting a noise level, gain adjusting a human voice signal to enhance human voice, and multi-channel downmixing to stereo.

[0273] First stage: detecting a human voice signal

[0274] It should be noted that one frame of audio signal in an audio stream can only include a human voice signal, can only include a background sound signal, or can include both a human voice signal and a background sound signal. In order to enhance the human voice signal in the audio stream, the human voice detection module is used to detect whether the human voice signal is included in each frame of audio signal.

[0275] In the embodiments of the present application, the human voice detection module can detect the input audio stream to determine whether the current frame of the input audio stream includes a human voice signal.

[0276] Figure 11 A flowchart of how to detect a human voice signal in the signal processing method for enhancing human voice provided by the embodiments of the present application is shown. As shown in Figure 11 The method includes S301-S308.

[0277] S301, obtaining an mth frame of multi-channel audio signal.

[0278] Wherein, m is a positive integer. That is, the first frame of audio signal in the audio stream is obtained first, and then the second frame of audio signal in the audio stream is obtained, and so on. Each frame of audio signal in the audio stream is obtained one by one, and each frame of audio signal is detected in turn to determine whether the human voice signal is included.

[0279] The mth frame of multi-channel audio signal can be any one of a two-channel stereo signal, a 5.1 channel signal, and a 7.1 channel signal.

[0280] It should be noted that if the input signal is a 5.1 channel or 7.1 channel signal, the mth frame of multi-channel audio signal includes a center channel time domain signal and a side channel time domain signal, wherein the center channel time domain signal is used to carry a human voice, and the side channel time domain signal is used to carry a background sound. For a 5.1 channel, the side channel time domain signal includes a front left channel and a front right channel, and a side surround left channel and a side surround right channel. For a 7.1 channel, the side channel time domain signal includes a front left channel and a front right channel, a side surround left channel and a side surround right channel, a rear left channel and a rear right channel.

[0281] If the input signal is a two-channel stereo signal, the mth frame of multi-channel audio signal includes a left channel time domain signal and a right channel time domain signal.

[0282] The following S302A and S302B are executed alternatively.

[0283] S302A, if the input signal is a two-channel stereo signal, the vocal time-domain signal and the background time-domain signal are calculated according to the left-channel time-domain signal and the right-channel time-domain signal.

[0284] According to the following equation 1-1, the vocal time-domain signal c(t) can be calculated according to the left-channel time-domain signal and the right-channel time-domain signal, and according to the following equation 1-2, the background time-domain signal s(t) can be calculated:

[0285]

[0286] s(t) = L - R (equation 1-2)

[0287] Wherein, L represents the left-channel time-domain signal in the two-channel stereo signal, R represents the right-channel time-domain signal in the two-channel stereo signal. Wherein, β is a constant.

[0288] S302B, if the input signal is a 5.1-channel signal or a 7.1-channel signal, the vocal time-domain signal is calculated according to the center-channel time-domain signal, and the background time-domain signal is calculated according to the side-channel time-domain signal.

[0289] It should be noted that, unlike the processing method of the two-channel stereo signal, for the 5.1-channel signal or the 7.1-channel signal, the center-channel signal can be taken as the vocal signal. The short-time Fourier transform is performed on the vocal time-domain signal, i.e. time-frequency conversion, to obtain the vocal frequency-domain signal.

[0290] For the 5.1-channel audio signal, according to the following equation 2-1, the vocal time-domain signal c(t) of the 5.1-channel audio signal is equal to the center-channel time-domain signal C. According to the following equation 2-2, the background time-domain signal s(t) in the 5.1-channel audio signal can be calculated:

[0291] c(t) = C (equation 2-1)

[0292] s(t) = (L + Ls) - (R + Rs) (equation 2-2)

[0293] Wherein, C represents the center-channel time-domain signal, L represents the front left-channel time-domain signal, Ls represents the side surround left-channel time-domain signal, R represents the front right-channel time-domain signal, and Rs represents the side surround right-channel time-domain signal.

[0294] For a 7.1 channel audio signal, the vocal time domain signal c(t) of the 5.1 channel audio signal is equal to the center channel time domain signal C according to Equation 3-1 below. The background sound time domain signal s(t) of the 5.1 channel audio signal can be obtained according to Equation 3-2 below:

[0295] c(t) = C (Equation 3-1)

[0296] s(t) = (L + Ls + Lb) - (R + Rs + Rb) (Equation 3-2)

[0297] where C represents the center channel time domain signal, L represents the front left channel time domain signal, Ls represents the side surround left channel time domain signal, Lb represents the back left channel time domain signal, R represents the front right channel time domain signal, Rs represents the side surround right channel time domain signal, and Rb represents the back right channel time domain signal.

[0298] S303, the vocal time domain signal is converted into a vocal frequency domain signal, and the background sound time domain signal is converted into a background sound frequency domain signal.

[0299] For a two-channel stereo sound, which includes a left channel time domain signal and a right channel time domain signal, the vocal time domain signal and the background sound time domain signal are first calculated according to the left channel time domain signal and the right channel time domain signal. Then, the vocal time domain signal is changed into a vocal frequency domain signal and the background sound time domain signal is changed into a background sound frequency domain signal by a time-frequency conversion algorithm.

[0300] The vocal time domain signal is changed into a vocal frequency domain signal according to a time-frequency conversion algorithm of Equation 4-1 below:

[0301] C(m, k) = STFT [c(t)] (Equation 4-1)

[0302] where STFT() represents a short-time Fourier transform function. Thus, the vocal time domain signal c(t) is subjected to a short-time Fourier transform to perform time-frequency conversion, thereby obtaining the vocal frequency domain signal C(m, k).

[0303] Meanwhile, the background sound time domain signal is changed into a background sound frequency domain signal according to a time-frequency conversion algorithm of Equation 4-2 below:

[0304] S(m, k) = STFT [s(t)] (Equation 4-2)

[0305] Thus, the background sound time domain signal s(t) is subjected to a short-time Fourier transform to perform time-frequency conversion, thereby obtaining the background sound frequency domain signal S(m, k).

[0306] In addition, the multi-signal frequency domain signal needs to be used for downmixing in the subsequent downmixing process. Therefore, for the two-channel stereo, the left-channel time domain signal and the right-channel time domain signal need to be converted into the left-channel frequency domain signal and the right-channel frequency domain signal, respectively. For the 5.1-channel signal or the 7.1-channel signal, the front left-channel time domain signal, the front right-channel time domain signal, the side surround left-channel time domain signal and the side surround right-channel time domain signal need to be subjected to the short-time Fourier transform, respectively, to obtain the front left-channel frequency domain signal, the front right-channel frequency domain signal, the side surround left-channel frequency domain signal and the side surround right-channel frequency domain signal.

[0307] In S304, the spectral flux of the vocal frequency domain signal (denoted as δc) and the spectral flux of the background sound frequency domain signal (denoted as δs) are calculated.

[0308] The process of calculating the spectral flux will be described in detail below.

[0309] In S305, the normalized spectral flux (denoted as V(m)) is calculated according to the spectral flux δc of the vocal frequency domain signal and the spectral flux δs of the background sound frequency domain signal.

[0310] In the embodiments of the present application, the spectral flux of the vocal frequency domain signal and the spectral flux of the background sound frequency domain signal are calculated according to the spectral flux algorithm, and then the normalized spectral flux value is calculated according to the spectral flux of the vocal frequency domain signal and the spectral flux of the background sound frequency domain signal by using the spectral flux normalization algorithm. The process of calculating the normalized spectral flux value will be described in detail below.

[0311] Then, the normalized spectral flux value V(m) is compared with a preset vocal detection threshold (denoted as V), and whether the current frame of audio signal contains the vocal signal is determined according to the comparison result.

[0312] In S306, it is determined whether V(m) is greater than the vocal detection threshold V.

[0313] In S307, in the case where V(m) is greater than the vocal detection threshold V, it is determined that the mth frame of multi-channel audio signal contains the vocal signal.

[0314] In S308, in the case where V(m) is less than or equal to the vocal detection threshold V, it is determined that the mth frame of multi-channel audio signal does not contain the vocal signal.

[0315] The process of detecting the vocal signal is described above by using the method flowchart, and the calculation process in the above process of detecting the vocal signal will be described below in combination with the preset vocal detection algorithm provided in the embodiments of the present application.

[0316] In the embodiments of the present application, whether the current frame audio signal contains a human voice signal can be detected based on spectral flux (delta spectrum magnitude).

[0317] wherein the spectral flux is defined as:

[0318]

[0319] wherein δ X (m, k) represents the change of the spectrum of the signal over time, m represents that the current frame audio signal is the mth frame in the audio stream, and k represents that the frequency point of the current frame audio signal in the frequency domain is k. X(m, k) represents the frequency domain signal corresponding to the mth frame audio signal.

[0320] For example, m = 20, k = 35, and X(m, k) represents the spectrum of the 35th frequency point of the 20th frame frequency domain signal.

[0321] In order to facilitate calculation, the spectral flux of the human voice frequency domain signal and the spectral flux of the background sound frequency domain signal are normalized to obtain normalized spectral fluxes:

[0322]

[0323]

[0324]

[0325] wherein δ C (m, k) is the spectral flux of the human voice frequency domain signal, and represents the change of the spectrum of the human voice signal over time. C(m, k) represents the spectrum of the kth frequency point of the mth frame human voice signal, and C(m-1, k) represents the spectrum of the kth frequency point of the previous frame human voice signal of the mth frame.

[0326] wherein δ S (m, k) is the spectral flux of the background sound frequency domain signal, and represents the change of the spectrum of the background sound frequency domain signal over time. S(m, k) represents the spectrum of the mth frame background sound signal, and S(m-1, k) represents the spectrum of the kth frequency point of the previous frame background sound signal of the mth frame.

[0327] It should be noted that, since the spectral flux represents the change of the spectrum of the signal over time, and the alternation of consonants and vowels exists in the human voice, the spectral flux δ C (m, k) of the human voice signal is greater than the spectral flux δ S (m, k) of the background sound signal.

[0328] For 5.1 / 7.1 channel audio signals, the voice signal can correspond to the center channel signal, and the background signal can correspond to the side channel signal other than the center channel signal.

[0329] The normalized spectral flux V(m) represents the proportion of the voice signal in the audio signal. V(m) can be used as a probability value for determining whether the audio signal contains a voice signal. V(m) ∈ [0, 1].

[0330] In the embodiments of the present application, a voice detection threshold V can be preset. For example, the voice detection threshold V is set to 0.5. The specific value of the voice detection threshold V can be set according to actual use requirements, and the embodiments of the present application are not limited.

[0331] In the embodiments of the present application, V(m) can be compared with the voice detection threshold V, and whether the audio signal contains a voice signal can be determined according to the comparison result.

[0332] For example, when V(m) > V, it can be considered that the current frame of audio signal contains a voice signal. When V(m) ≤ V, it can be considered that the current frame of audio signal does not contain a voice signal.

[0333] The above describes the calculation of the normalized spectral flux for the input audio stream, which can be a two-channel stereo signal or a 5.1 / 7.1 multi-channel signal. The following describes the implementation of how to detect the voice according to the above equation when the input audio stream is a two-channel stereo signal, a 5.1 channel or a 7.1 channel audio signal.

[0334] Scenario one: detecting voice for a two-channel stereo signal

[0335] For a two-channel stereo signal, each frame of audio signal includes a left channel time domain signal and a right channel time domain signal. Specifically, the voice frequency domain signal C(m, k) in the two-channel stereo signal is substituted into the above equation 7 to obtain the spectral flux δ C (m, k) of the voice signal, the background frequency domain signal S(m, k) in the two-channel stereo signal is substituted into the above equation 8 to obtain the spectral flux δ S (m, k) of the background signal, and then the spectral flux δ C (m, k) of the voice signal and the spectral flux δ S (m, k) of the background signal are substituted into the above equation 6 to obtain the normalized spectral flux V(m).

[0336] When V(m) > V, it can be considered that the current frame of audio signal contains a voice signal. When V(m) ≤ V, it can be considered that the current frame of audio signal does not contain a voice signal.

[0337] Scenario two: detecting human voice from 5.1 multi-channel audio signal

[0338] First of all, it should be noted that, for the convenience of description, other channels except the center channel are called side channels, which include front left channel, front right channel, side surround left channel and side surround right channel.

[0339] Specifically, the spectral flux δ C (m, k) of the human voice signal is obtained by substituting the human voice frequency domain signal C(m, k) in the 5.1 channel signal into the above equation 7, the spectral flux δ S (m, k) of the background sound signal is obtained by substituting the background sound frequency domain signal S(m, k) in the 5.1 channel signal into the above equation 8, then the spectral flux δ C (m, k) of the human voice signal and the spectral flux δ S (m, k) of the background sound signal are substituted into the above equation 6 to obtain the normalized spectral flux V(m).

[0340] When V(m) > V, it can be considered that the current frame audio signal contains human voice signal. When V(m)≤V, it can be considered that the current frame audio signal does not contain human voice signal.

[0341] Scenario three: detecting human voice from 7.1 multi-channel audio signal

[0342] First of all, it should be noted that, for the convenience of description, other channels except the center channel are called side channels, which include front left channel, front right channel, side surround left channel, side surround right channel, back left channel and back right channel.

[0343] Specifically, the spectral flux δ C (m, k) of the human voice signal is obtained by substituting the human voice frequency domain signal C(m, k) in the 7.1 channel signal into the above equation 7, the spectral flux δ S (m, k) of the background sound signal is obtained by substituting the background sound frequency domain signal S(m, k) in the 7.1 channel signal into the above equation 8, then the spectral flux δ C (m, k) of the human voice signal and the spectral flux δ S (m, k) of the background sound signal are substituted into the above equation 6 to obtain the normalized spectral flux V(m).

[0344] When V(m) > V, it can be considered that the current frame audio signal contains human voice signal. When V(m)≤V, it can be considered that the current frame audio signal does not contain human voice signal.

[0345] According to Equation 9 below, the vocal signal time-frequency parameter C(m, k) of the 7.1 channel audio signal is equal to the middle channel frequency domain signal Fc, and according to Equation 10 below, the background sound signal time-frequency parameter S(m, k) in the 7.1 channel audio signal can be obtained:

[0346] C(m, k) = F C (Equation 9)

[0347] S(m, k) = (F L +F Ls +F Lb )-(F R +F Rs +F Rb ) (Equation 10)

[0348] Wherein, F L represents the frequency domain signal of the front left channel, F Ls represents the frequency domain signal of the side surround left channel, F Lb represents the frequency domain signal of the rear left channel, F R represents the frequency domain signal of the front right channel, F Rs represents the frequency domain signal of the side surround right channel, and F Rb represents the frequency domain signal of the rear right channel.

[0349] Second stage: extracting vocal signal

[0350] In the embodiment of the present application, the vocal signal extraction module can extract the vocal signal based on the similarity between different channel signals (specifically, the left channel signal and the right channel signal).

[0351] First of all, it should be pointed out that the cross-correlation function can be used to represent the similarity between two channel signals. Among them, the cross-correlation function and the similarity function are respectively represented as:

[0352]

[0353]

[0354] In the embodiment of the present application, the above equation can be used to calculate the similarity between different channel signals.

[0355] Specifically, for a two-channel stereo signal, the similarity between the left channel signal and the right channel signal can be calculated using the following equation:

[0356]

[0357]

[0358]

[0359] wherein F L (m, k) represents a left channel frequency domain signal, represents a conjugate function of the left channel frequency domain signal, F R (m, k) represents a right channel frequency domain signal, represents a conjugate function of the right channel frequency domain signal;

[0360] wherein φ LR (m, k) is a cross-correlation function between the left channel frequency domain signal F L (m, k) and the right channel frequency domain signal F R (m, k) is a cross-correlation function between the left channel frequency domain signal F LL (m, k) and the right channel frequency domain signal F L (m, k) is an autocorrelation function of the left channel frequency domain signal F RR (m, k) is an autocorrelation function of the right channel frequency domain signal F R (m, k) is an autocorrelation function of the right channel frequency domain signal F

[0361] wherein ψ(m, k) is a similarity function between the left channel frequency domain signal and the right channel frequency domain signal.

[0362] The similarity ψ(m, k) of the binaural stereo signal can be calculated by the above equations 11, 12 and 13.

[0363] In some implementations, the embodiments of the present application increase the smoothness of the extracted vocal frequency domain signal by adding an inter-frame smoothing coefficient (denoted as a) to the above equation 11. According to the above equation 11, the following equation 14 can be obtained:

[0364] ψ(m, k) = (1 - a) ψ(m - 1, k) + a ψ(m, k) (Equation 14)

[0365] wherein the inter-frame smoothing coefficient a e (0, 1), the greater the a is, the worse the smoothness is, but the target value can be reached quickly.

[0366] Further, after the similarity ψ(m, k) of the binaural stereo signal is calculated, the vocal frequency domain signal can be calculated by the following equation:

[0367] C(m, k) = ψ(m, k) x [F L (m, k) + F R (m, k)] / 2 (Equation 15)

[0368] wherein C(m, k) represents a spectrum of the extracted vocal signal from the multi-channel audio signal.

[0369] That is, the scheme of the present application extracts the human voice signal by using the inter-channel similarity. The signal with similarity between different channels is the human voice signal. It can be understood that the signal with zero similarity between different channels is the background sound signal.

[0370] The scheme of the present application extracts the human voice signal by using the inter-channel similarity. The signal with greater similarity between different channels is usually the human voice signal.

[0371] Exemplarily, Figure 12 The schematic diagram of human voice detection and human voice extraction is shown. As Figure 12 shown, the input audio source includes left channel signal and right channel signal. On one hand, the human voice detection processing is performed according to the left channel signal and the right channel signal, and the human voice detection result is obtained. The human voice detection result judges the probability of human voice existence by the level, i.e. high level represents human voice and low level represents background sound. On the other hand, the human voice extraction processing is performed according to the left channel signal and the right channel signal, and the human voice extraction result, i.e. human voice signal, is obtained. The signal other than the human voice signal is the background sound signal.

[0372] Third stage: estimate the environmental noise level

[0373] In the embodiment of the present application, the environmental noise estimation module will periodically call the microphone recording of the electronic device (such as mobile phone) to collect the environmental noise signal, and then estimate the noise level of the current environment of the electronic device by analyzing the environmental noise signal.

[0374] In the embodiment of the present application, based on the power consumption consideration, the microphone is called every x minutes to collect y seconds of data. Exemplarily, the microphone is called every 30 minutes to collect 10 seconds of data.

[0375] If the speaker does not work (for example, no audio or video playing) when the microphone collects the environmental noise signal, then the average amplitude of the collected data within y seconds can be directly calculated as the result of the environmental noise estimation.

[0376] If the speaker works when the microphone collects the environmental noise signal, then the noise level of the current environment is estimated according to the sensitivity of the speaker and the sensitivity of the microphone.

[0377] Herein, the scenario of the speaker working when the microphone collects the environmental noise signal is described in combination with the drawings. Figure 13 The process that the audio signal output by the electronic device is first played through the speaker, then propagates through the environment, and then received by the microphone is shown. As Figure 13It is shown that when the microphone collects the environmental noise signal, the electronic device is playing a video, and the video sound is played through the loudspeaker, the video sound is propagated through the environment, received by the microphone and completed recording. That is, the microphone recording result includes both the environmental noise and the sound played by the loudspeaker. Therefore, it is necessary to eliminate the collected signal played by the loudspeaker, and then estimate the noise level of the current environment.

[0378] The calculation process of eliminating the collected signal played by the loudspeaker and then estimating the noise level of the current environment in the embodiment of the application is described in detail below.

[0379] In some embodiments, the signal collected by the microphone includes both the environmental noise and the microphone recording signal. In the embodiment of the application, the signal collected by the microphone can be calculated by the following equation:

[0380] d(t) = y(t) + s(t) (Equation 16)

[0381] Wherein, d(t) represents the signal collected by the microphone, s(t) represents the environmental noise signal, and y(t) represents the microphone recording signal.

[0382] Figure 14 A schematic diagram of the two-channel stereo signal x(t) played by the loudspeaker of the electronic device is shown.

[0383] Figure 15 A schematic diagram of the signal d(t) collected by the microphone of the electronic device is shown.

[0384] The two-channel stereo signal x(t) played by the electronic device is played through the loudspeaker and recorded by the microphone, and the microphone recording signal y(t) is obtained. In the embodiment of the application, the microphone recording signal y(t) can be calculated by the following equation:

[0385] y(t) = x(t) * w(t) (Equation 17)

[0386] Wherein, x(t) is the audio signal played by the loudspeaker of the electronic device, and w(t) represents the system frequency response function of the electronic device.

[0387] It should be noted that the above system frequency response function can be determined according to the loudspeaker sensitivity and the microphone sensitivity. In the embodiment of the application, the system frequency response function w(t) can be calculated by the following equation:

[0388] w(t) = S spk ×S mic ×H (Equation 18)

[0389] S spk represents the loudspeaker sensitivity, Smic wherein S represents the microphone sensitivity, H is an adjustment coefficient. Since the microphone and the speaker are close to each other, the adjustment coefficient H is approximately equal to 1.

[0390] Through the above equation 16, equation 17 and equation 18, the ambient noise signal s(t) can be derived as:

[0391] s(t) = d(t) - x(t) x S spk x S mic (equation 19)

[0392] Then, through the following equation 20, the ambient noise average amplitude can be calculated:

[0393]

[0394] wherein N represents the ambient noise average amplitude, in dB SPL, abbreviated as dB. T is the collection time length, F is the sampling frequency, and P0 is the minimum sound pressure intensity, taking a value of 20 micro-pascal (μPa).

[0395] In the embodiments of the present application, the ambient noise level can be determined according to the ambient noise average amplitude N. The specific determination process will be described in detail below.

[0396] Exemplarily, Table 1 shows the corresponding relationship between the ambient noise average amplitude and the ambient noise level.

[0397] Table 1

[0398]

[0399]

[0400] When the ambient noise average amplitude N takes a value in the range of [40dB, 45dB], it is determined that the ambient noise level is low.

[0401] When the ambient noise average amplitude N takes a value in the range of (45dB, 55dB], it is determined that the ambient noise level is medium.

[0402] When the ambient noise average amplitude N takes a value in the range of (55dB, 65dB], it is determined that the ambient noise level is slightly high.

[0403] When the ambient noise average amplitude N takes a value in the range of (65dB, 75dB], it is determined that the ambient noise level is very high.

[0404] It should be noted that the above table 1 is exemplarily illustrated, and it can be understood that the correspondence between the ambient noise average amplitude and the ambient noise level is not limited to this in actual implementation, and of course the correspondence between the ambient noise average amplitude and the ambient noise level can also be set according to actual use requirements, and the embodiments of the present application do not limit this.

[0405] Fourth stage: enhancing the human voice signal

[0406] In the embodiments of the present application, the human voice gain estimation module adjusts the gain of the human voice signal C(m, k) obtained by the human voice extraction algorithm in the second stage according to the ambient noise level determined in the third stage, so as to realize the effect of dynamically enhancing the human voice.

[0407] First of all, it should be noted that the traditional human voice enhancement algorithm applies the Wiener filtering algorithm to the human voice gain. The calculation equation for adjusting the human voice gain by the Wiener filtering algorithm can be expressed as:

[0408]

[0409] SNR(m, k) = C(m, k) / S(m, k) (Equation 22)

[0410] Wherein, G(m, k) represents the adjusted human voice gain, SNR(m, k) represents the signal-to-noise ratio, C(m, k) represents the spectrum of the mth frame of human voice signal, and S(m, k) represents the spectrum of the mth frame of background sound signal.

[0411] For the convenience of description, some human voice signals with small signal spectrum intensity in the multi-channel audio signal are simply referred to as small signals. Compared with the small signals, some human voice signals with large signal spectrum intensity in the multi-channel audio signal are simply referred to as large signals.

[0412] It should be noted that the traditional Wiener filtering algorithm significantly improves the gain of signals with large signal spectrum intensity, but the gain of signals with small signal spectrum intensity is not obvious. It can be understood that in this case, the spectrum range of the human voice signal is relatively large; wherein the spectrum range of the human voice signal refers to the spectrum range formed by the small signals and the large signals.

[0413] That is, through the traditional Wiener filtering algorithm, the gain of the large signal is obviously small, and the gain of the small signal is too small, which will cause the phenomenon of large and small on the final listening.

[0414] In order to solve the problem of large and small on the listening, the embodiments of the present application propose an improved method to dynamically adjust the human voice gain through the following equation.

[0415] G'(m, k) = p(1-G(m, k))+qG(m, k) (Equation 23)

[0416] wherein G(m, k) represents the voice gain after preliminary adjustment by the Wiener filter algorithm, G'(m, k) represents the voice gain after re-adjustment on the basis of the voice gain after preliminary adjustment, p represents the weight coefficient of the small signal, and q represents the weight coefficient of the large signal.

[0417] It can be understood that the voice gain estimation formula (equation 23) adopted in the scheme of the present application is a formula improved on the basis of the Wiener filter formula (equation 21). Through the voice gain estimation formula adopted in the scheme of the present application, the voice signal gain of different intensities in the voice signal spectrum range can be adjusted.

[0418] Through the scheme of the embodiment of the present application, the values of the two coefficients p and q can be dynamically adjusted according to the current environmental noise level, so that the gain of the large signal can be adjusted, the gain of the small signal can be adjusted, or the gain of the large signal and the gain of the small signal can be adjusted at the same time, and the phenomenon of large and small in hearing can be avoided. By dynamically adjusting the voice gain, the effect of not disturbing others and being able to hear the voice of the video clearly in a quiet environment, and the effect of being able to hear the voice of the video clearly in a noisy environment can be achieved.

[0419] Exemplarily, through the scheme of the present application, in a quiet environment, the values of the two coefficients p and q are dynamically adjusted, the gain of the large signal is appropriately increased, and the gain of the small signal is appropriately increased, so that even if the volume of the electronic device playing the video is low, it can be ensured that the user can clearly hear the voice of the strong voice dialogue in the video, and also can clearly hear the voice of the weak voice dialogue in the video. Thus, the effect of not disturbing others and being able to hear the voice of the video clearly in a quiet environment can be achieved.

[0420] Further exemplarily, through the scheme of the present application, in a noisy environment, the values of the two coefficients p and q are dynamically adjusted, the gain of the large signal is appropriately increased, and the gain of the small signal is appropriately increased, so that the user can clearly hear the voice of the strong voice dialogue in the video, and also can clearly hear the voice of the weak voice dialogue in the video, thereby achieving the effect of being able to hear the voice of the video clearly in a noisy environment.

[0421] Compared with the related art in which only the gain of the large signal is increased, the scheme of the present application appropriately increases the gain of the small signal while increasing the gain of the large signal, thereby achieving the effect of compressing the dynamic range of the voice signal, and thus the phenomenon of large and small in hearing can be avoided.

[0422] The possible cases of specific values of p and q are described in detail below.

[0423] In some embodiments, p and q values can be determined according to the average amplitude N of the environmental noise based on a linear mapping relationship between p, q and the average amplitude of the environmental noise.

[0424] Exemplarily, Figure 16 A schematic diagram showing the linear mapping relationship between p, q and the average amplitude of the environmental noise is shown. As Figure 16 shown, the horizontal coordinate is the average amplitude N of the noise level, in dB SPL, and the vertical coordinate is the p / q value, the thick solid line represents p, and the thin solid line represents q. According to the linear mapping relationship in the figure, in the average amplitude [40, 75) of the noise level, the values of p and q are

[0425] (1) The value rule of p in the average amplitude [40, 75) of the noise level.

[0426] When 40≤N<65, p=0.02×N-0.8.

[0427] When 65≤N<75, p=0.15×N-9.25.

[0428] (2) The value rule of q in the average amplitude [40, 75) of the noise level.

[0429] When 40≤N<65, q=1.

[0430] When 65≤N<75, q=-0.04×N+3.6.

[0431] In the embodiments of the present application, the specific values of p and q can be determined according to actual use requirements. Exemplarily, the value range of p can be set as [0, 2], and the value range of q can be set as [0.25, 1]. The specific values of p and q can be determined according to actual use requirements, which are not limited in the embodiments of the present application.

[0432] The following illustrates a possible implementation manner of how to dynamically adjust the two coefficients p and q at different environmental noise levels.

[0433] Exemplarily, Table 2 shows the values of p and q at different environmental noise levels and the strategy of adjusting the gain of the vocal signal.

[0434] Table 2

[0435]

[0436]

[0437] As can be seen from the above Table 2, the scheme of the present application can adjust the gain control in different environments considering the influence of environmental noise.

[0438] Scenario one: when the environmental noise level is low (such as 40dB-45dB), the parameters of p=0 and q=1 are adopted. At this time, the above equation 23 changes to:

[0439] G'(m, k)=G(m, k)

[0440] Correspondingly, the strategy of adjusting the gain of the human voice signal is to appropriately increase the gain of a large signal and retain the dynamic range of the human voice signal.

[0441] Scenario two: when the environmental noise level is medium (such as 45dB-55dB), the parameters of p=0.25 and q=1 are adopted. At this time, the above equation 23 changes to:

[0442] G'(m, k)=0.25(1-G(m, k))+G(m, k)

[0443] Correspondingly, the strategy of adjusting the gain of the human voice signal is to increase the gain of a small signal and slightly compress the dynamic range of the signal.

[0444] Scenario three: when the environmental noise level is slightly high (such as 55dB-65dB), the parameters of p=0.5 and q=1 are adopted. At this time, the above equation 23 changes to:

[0445] G'(m, k)=0.5(1-G(m, k))+G(m, k)

[0446] Correspondingly, the strategy of adjusting the gain of the human voice signal is to further increase the gain of a small signal and compress the dynamic range.

[0447] Scenario four: when the environmental noise level is very high (such as 65dB-75dB), the parameters of p=2 and q=0.6 are adopted. At this time, the above equation 23 changes to:

[0448] G'(m, k)=2(1-G(m, k))+0.6G(m, k)

[0449] Correspondingly, the strategy of adjusting the gain of the human voice signal is to greatly increase the gain of the signal and greatly compress the dynamic range of the signal.

[0450] The scheme considers the environmental noise factor, automatically adjusts the human voice gain range according to the environmental noise level, and can avoid the phenomenon of large and small on the hearing.

[0451] It should be noted that the above table 2 is an example for illustration. It can be understood that, in actual implementation, the values of p and q and the strategy of adjusting the gain of the human voice signal at different environmental noise levels are not limited to this, and of course the values of p and q and the strategy of adjusting the gain of the human voice signal at different environmental noise levels can also be set according to actual use requirements, and the embodiments of the present application do not limit this.

[0452] In the embodiments of the present application, the human voice gain estimation module can perform human voice enhancement on the human voice signal according to a human voice enhancement algorithm. Specifically, the human voice signal can be enhanced by using the following equation:

[0453] C'(m, k) = C(m, k) + C(m, k) * G'(m, k) * V(m) (Equation 24)

[0454] wherein C'(m, k) represents the enhanced human voice signal spectrum, V(m) is the normalized spectral flux value obtained by the human voice detection algorithm in the first stage, C(m, k) is the human voice signal spectrum obtained by the human voice extraction algorithm in the second stage, and G'(m, k) is the human voice gain obtained by the human voice gain estimation algorithm in the fourth stage.

[0455] Fifth stage: multi-channel signal downmixing

[0456] In the embodiments of the present application, the downmix module can downmix the enhanced human voice signal C'(m, k) and the multi-channel signal into a two-channel stereo signal.

[0457] For example, for the input signal being a two-channel stereo signal, the downmix module downmixes the enhanced human voice signal and the left channel signal to obtain a left channel frequency domain signal, and downmixes the enhanced human voice signal and the right channel signal to obtain a right channel frequency domain signal.

[0458] For another example, for the input signal being a 5.1 / 7.1 channel stereo signal, the downmix module downmixes the enhanced human voice signal and the channel signals other than the center channel to obtain a left channel frequency domain signal and a right channel frequency domain signal.

[0459] After the downmixing is completed, the left channel frequency domain signal and the right channel frequency domain signal are converted by the post-processing module to obtain a left channel time domain signal and a right channel time domain signal, and then the left channel time domain signal and the right channel time domain signal are output through a loudspeaker or a headphone.

[0460] The downmixing and the frequency-time conversion processing will be described in detail below for different input signals.

[0461] (1) The input signal is a two-channel stereo signal.

[0462] After the input two-channel stereo signal is processed by the human voice enhancement algorithm provided in the embodiments of the present application, the enhanced human voice signal is downmixed with the left channel signal and the right channel signal to obtain a left channel frequency domain signal and a right channel frequency domain signal:

[0463] F' L = FL +C'(m, k) (Equation 25)

[0464] F' R = F R +C'(m, k) (Equation 26)

[0465] wherein F' L represents the output left channel frequency domain signal, F' R represents the output right channel frequency domain signal.

[0466] wherein F L represents the input left channel frequency domain signal, F R represents the input right channel frequency domain signal.

[0467] wherein C'(m, k) represents the enhanced vocal signal spectrum obtained by the vocal enhancement algorithm in the fourth stage.

[0468] After the downmix processing is completed, the left channel frequency domain signal F' L is converted from frequency domain to time domain by the post-processing module to obtain a left channel time domain signal; similarly, the right channel frequency domain signal F' R is converted from frequency domain to time domain by the post-processing module to obtain a right channel time domain signal.

[0469] Then, the post-processing module transmits the converted left channel time domain signal and right channel time domain signal to a loudspeaker or earphone, and the loudspeaker or earphone outputs the left channel time domain signal and right channel time domain signal, i.e., outputs a two-channel stereo signal.

[0470] (2) The input signal is a 5.1 channel audio signal.

[0471] After the input 5.1 channel audio signal is processed by the algorithm provided in the above embodiments of the present application, the enhanced vocal signal and the signals of other channels except the center channel are downmixed to obtain a left channel frequency domain signal and a right channel frequency domain signal:

[0472] F' L = 0.4143*(F L + 0.707C'(m, k) + 0.707F Ls ) (Equation 27)

[0473] F' R = 0.4143*(F R + 0.707C'(m, k) + 0.707F Rs ) (Equation 28)

[0474] wherein F' L represents the output left channel frequency domain signal, F'R represents the output right channel frequency domain signal.

[0475] wherein F L represents the input front left channel frequency domain signal, F R represents the input front right channel frequency domain signal, F Ls represents the input side surround left channel frequency domain signal, F Rs represents the input side surround right channel frequency domain signal.

[0476] wherein C'(m, k) represents the enhanced vocal signal spectrum obtained by the vocal enhancement algorithm in the fourth stage.

[0477] After the downmix processing is completed, the left channel frequency domain signal F' L is converted into a left channel time domain signal; similarly, the right channel frequency domain signal F' R is converted into a right channel time domain signal.

[0478] Then, the left channel time domain signal and the right channel time domain signal converted by the post-processing module are transmitted to a loudspeaker or a headphone, and output by the loudspeaker or the headphone.

[0479] (3) The input signal is a 7.1 channel audio signal.

[0480] After the input 7.1 channel audio signal is processed by the algorithm provided in the above embodiments of the present application, the following two-channel stereo signal is output by the loudspeaker or the headphone:

[0481] F' L = 0.3204 * (F L + 0.707C'(m, k) + 0.707F Ls + 0.707F Lb ) (Equation 29)

[0482] F' R = 0.3204 * (F R + 0.707C'(m, k) + 0.707F Rs + 0.707F Rb ) (Equation 30)

[0483] wherein F' L represents the output left channel frequency domain signal, F' R represents the output right channel frequency domain signal.

[0484] wherein F L represents the input front left channel frequency domain signal, F R represents the input front right channel frequency domain signal, FLs denotes an input side surround left channel frequency domain signal, F Rs denotes an input side surround right channel frequency domain signal, F Lb denotes an input back left channel frequency domain signal, F Rb denotes an input back right channel frequency domain signal.

[0485] wherein C'(m, k) denotes a spectrum of the enhanced vocal signal obtained by the vocal enhancement algorithm in the fourth stage.

[0486] Similarly, after the downmix processing is completed, the left channel frequency domain signal F' L , F' R are converted into time domain signals by the post-processing module.

[0487] Then, the post-processing module transmits the converted left channel time domain signal and right channel time domain signal to a loudspeaker or earphone, which outputs the signals.

[0488] The signal processing method for enhancing vocals provided in the embodiments of the present application can be applied to an audio output scene in which the input signal is a multi-channel audio signal, and can dynamically adjust the vocal gain of signals of different intensities in the dynamic range of the vocal signal according to the environmental noise level. By adjusting the vocal gain, the effect of enhancing vocals is achieved, and the output sound remains stable, thereby avoiding the phenomenon of large and small sounds in the hearing.

[0489] In a noisy environment, the present application can take different gain methods for vocals according to the environmental noise level. The algorithm automatically selects the vocal gain parameter in a noisy environment, which can resist noise interference, so that the user can clearly hear the vocal dialogue in the video.

[0490] Similarly, in a quiet environment, in the scenario of watching a television program at night, it is not desirable to disturb people who have fallen asleep. By using the present application, the user can clearly hear the vocal dialogue in the video under the condition that the volume is not obviously increased.

[0491] The signal processing method for enhancing vocals provided in the embodiments of the present application can be applied to a scenario in which an electronic device outputs an audio signal through a loudspeaker, and is also applicable to a scenario in which the electronic device outputs an audio signal through an earphone. For example, in a noisy environment, in the scenario of wearing an earphone and watching a video played by a mobile phone, after the input multi-channel audio signal is processed by the algorithm provided in the present application, the electronic device outputs the enhanced audio signal through the earphone. Even in a noisy environment, the user can clearly hear the vocal dialogue with strong sound and the vocal dialogue with weak sound in the video through the earphone, thereby achieving the effect of clearly hearing the vocal dialogue in the video in a noisy environment.

[0492] It should be noted that in the embodiments of the present application, "greater than" can be replaced by "greater than or equal to", "less than or equal to" can be replaced by "less than", or "greater than or equal to" can be replaced by "greater than", and "less than" can be replaced by "less than or equal to".

[0493] The various embodiments described herein can be independent solutions or combined according to inherent logic, and all fall within the protection scope of the present application.

[0494] The above mainly describes the solutions provided by the embodiments of the present application from the perspective of method steps. It can be understood that, in order to realize the above functions, the electronic device implementing the method contains the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should realize that, in combination with the units and algorithm steps of the examples described in the embodiments disclosed herein, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is realized in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the protection scope of the present application.

[0495] The embodiments of the present application can divide the functional modules of the electronic device according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated in one processing module. The above integrated module can be realized in the form of hardware or software functional module. It should be noted that the division of modules in the embodiments of the present application is illustrative, and is only a logical function division. Actual implementation can have other feasible division manners. The following takes the division of each functional module according to each function as an example for description.

[0496] Figure 17 A schematic block diagram of the signal processing device 400 for enhancing human voice provided by the embodiments of the present application is shown. The device 400 can be used to execute the actions performed by the electronic device in the above method embodiments. The device 400 includes an acquisition unit 410, a processing unit 420 and an output unit 430.

[0497] The receiving unit 410 is configured to acquire the mth frame of audio signal in the multi-channel audio stream. For example, the receiving unit 410 is a microphone.

[0498] The processing unit 420 is configured to calculate a normalized spectral flux corresponding to the mth frame of audio signal; obtain a vocal frequency domain signal and a background sound frequency domain signal from the mth frame of audio signal; determine a vocal gain parameter according to the environmental noise, the background sound frequency domain signal and the vocal frequency domain signal; perform gain adjustment on the vocal frequency domain signal according to the vocal gain parameter and the normalized spectral flux to obtain an enhanced vocal frequency domain signal; and then convert the enhanced vocal frequency domain signal into an enhanced vocal time domain signal.

[0499] The output unit 430 is configured to output the enhanced vocal time domain signal. For example, the output unit 430 is a loudspeaker or a headphone.

[0500] In some possible implementation manners, the processing unit 420 is specifically configured to:

[0501] determine the first gain adjustment coefficient and the second gain adjustment coefficient according to the average amplitude of the environmental noise;

[0502] adopt the following equation to calculate the vocal gain parameter according to the first gain adjustment coefficient, the second gain adjustment coefficient, the background sound frequency domain signal and the vocal frequency domain signal:

[0503] G'(m, k) = p(1 - G(m, k)) + qG(m, k)

[0504]

[0505] SNR(m, k) = C(m, k) / S(m, k)

[0506] wherein G'(m, k) represents the vocal gain parameter, p represents the first gain adjustment coefficient, q represents the second gain adjustment coefficient, C(m, k) represents the vocal frequency domain signal, and S(m, k) represents the background sound frequency domain signal.

[0507] In some possible implementation manners, the processing unit 420 is specifically configured to:

[0508] adopt the following equation to calculate the enhanced vocal signal spectrum according to the vocal gain parameter and the normalized spectral flux:

[0509] C'(m, k) = C(m, k) + C(m, k) * G'(m, k) * V(m)

[0510] wherein C'(m, k) represents the enhanced vocal frequency domain signal, C(m, k) represents the vocal frequency domain signal, G'(m, k) represents the vocal gain parameter, and V(m) represents the normalized spectral flux corresponding to the mth frame of audio signal.

[0511] The signal processing device for enhancing human voice provided in the application can determine a human voice gain parameter according to the environmental noise and the background voice frequency domain signal and the human voice frequency domain signal corresponding to the multi-channel audio stream in the scene of playing audio and video by an electronic device, then perform gain adjustment on the human voice frequency domain signal according to the human voice gain parameter and the normalized spectral flux, and obtain the enhanced human voice frequency domain signal. Since the gain of the human voice signal can be dynamically adjusted according to the input multi-channel audio signal in different environmental noise conditions, the effect of enhancing the human voice is achieved, and the phenomenon of large and small hearing is avoided, so the application scheme can achieve the effect that the human voice of the video can be heard clearly without disturbing others in a quiet environment, or the human voice of the video can be heard clearly by the user in a noisy environment, and the user experience can be improved.

[0512] The apparatus 400 according to the embodiments of the application can correspond to performing the method described in the embodiments of the application, and the above and other operations and / or functions of the units in the apparatus 400 are respectively for realizing the corresponding flows of the method, and for brevity, will not be described here.

[0513] The application further provides a chip coupled with the memory, the chip being used to read and execute the computer program or instructions stored in the memory to execute the method in each of the above embodiments.

[0514] The application further provides an electronic device comprising a chip, the chip being used to read and execute the computer program or instructions stored in the memory so that the method in each of the embodiments is executed.

[0515] The embodiment further provides a computer readable storage medium, the computer readable storage medium storing computer instructions, when the computer instructions are run on an electronic device, the electronic device executes the related method steps to realize the signal processing method for enhancing human voice in the above embodiments.

[0516] The embodiment further provides a computer program product, the computer readable storage medium storing program codes, when the computer program product is run on a computer, the computer executes the related steps to realize the signal processing method for enhancing human voice in the above embodiments.

[0517] In addition, the embodiments of the application further provide a device, which can be a chip, a component or a module, the device can comprise a processor and a memory connected to each other; wherein the memory is used to store computer execution instructions, when the device is running, the processor can execute the computer execution instructions stored in the memory to make the chip execute the signal processing method for enhancing human voice in each of the above method embodiments.

[0518] The electronic device, the computer readable storage medium, the computer program product or the chip provided in the embodiment are used for executing the corresponding method provided in the above, and thus the beneficial effects achieved by the electronic device, the computer readable storage medium, the computer program product or the chip can refer to the beneficial effects of the corresponding method provided in the above, which will not be described here.

[0519] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the division of the apparatus embodiments is only a logical function division, and there can be another division manner for the actual implementation. For example, multiple units or components can be combined or integrated into another apparatus, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, apparatuses or units, and can be in electrical, mechanical or other forms.

[0520] The above describes only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A signal processing method for enhancing human voice, characterized in that, include: Acquire the m-th frame audio signal from a multi-channel audio stream; Calculate the normalized spectral flow corresponding to the m-th frame audio signal. The normalized spectral flow is obtained by normalizing the spectral flow of the human voice frequency domain signal and the spectral flow of the background sound frequency domain signal. The normalized spectral flow represents the proportion of the human voice signal in the audio signal. Obtain the human voice frequency domain signal and the background sound frequency domain signal from the m-th frame audio signal; The human voice gain parameter is determined based on the ambient noise, the background sound frequency domain signal, and the human voice frequency domain signal. Based on the human voice gain parameters and the normalized spectral flux, the gain of the human voice frequency domain signal is adjusted to obtain the enhanced human voice frequency domain signal. The step of determining the human voice gain parameter based on the environmental noise, the background sound frequency domain signal, and the human voice frequency domain signal includes: determining a first gain adjustment coefficient and a second gain adjustment coefficient based on the average amplitude of the environmental noise; and calculating the human voice gain parameter using the following equation based on the first gain adjustment coefficient, the second gain adjustment coefficient, the background sound frequency domain signal, and the human voice frequency domain signal: in, This represents the human voice gain parameter. p This represents the first gain adjustment coefficient. q This represents the second gain adjustment coefficient. This represents the human voice frequency domain signal. This refers to the background sound frequency domain signal.

2. The method according to claim 1, characterized in that, The step of determining the first gain adjustment coefficient and the second gain adjustment coefficient based on the average amplitude of the environmental noise includes: When N∈[a, b) , ; When N∈[b, c) , ; Where N represents the average amplitude of the environmental noise, in decibels (dB), and a is less than b, b is less than c.

3. The method according to claim 2, characterized in that, a is set to 40 dB, b to 65 dB, and c to 75 dB.

4. The method according to claim 1, characterized in that, The method is applied to an electronic device, the electronic device including a microphone; the method further includes: The ambient noise signal is determined based on the audio signal collected by the microphone; The average amplitude of the environmental noise is determined based on the environmental noise signal, sampling duration, sampling frequency, and minimum sound pressure level.

5. The method according to claim 4, characterized in that, The electronic device further includes a speaker, and the step of determining the ambient noise signal based on the audio signal collected by the microphone includes: When the speaker is working, the ambient noise signal is determined based on the audio signal collected by the microphone and the audio signal played by the speaker; When the speaker is not working, the audio signal collected by the microphone is identified as the ambient noise signal.

6. The method according to claim 5, characterized in that, Determining the ambient noise signal based on the audio signal collected by the microphone and the audio signal played by the speaker includes: The ambient noise signal is calculated using the following equation, based on the audio signal captured by the microphone and the audio signal played by the speaker: in, s ( t ) represents the environmental noise signal. This refers to the audio signal captured by the microphone. x ( t () indicates the audio signal played by the speaker. Indicates the sensitivity of the speaker, This indicates the sensitivity of the microphone.

7. The method according to claim 4, characterized in that, The step of determining the average amplitude of the environmental noise based on the environmental noise signal, sampling duration, sampling frequency, and minimum sound pressure level includes: The average amplitude of the environmental noise is calculated using the following equation, based on the environmental noise signal, sampling duration, sampling frequency, and minimum sound pressure level: Wherein, N represents the average amplitude of the environmental noise, s(t) represents the environmental noise signal, T represents the sampling duration, F represents the sampling frequency, and P0 represents the minimum sound pressure level.

8. The method according to any one of claims 1 to 7, characterized in that, The step of adjusting the gain of the human voice frequency domain signal according to the human voice gain parameter and the normalized spectral flux to obtain the enhanced human voice frequency domain signal includes: The enhanced human voice signal spectrum is calculated using the following equation, based on the human voice gain parameter and the normalized spectral flux: in, This represents the enhanced human voice frequency domain signal. This represents the human voice frequency domain signal. V(m) represents the human voice gain parameter, and V(m) represents the normalized spectral flow corresponding to the m-th frame audio signal.

9. The method according to any one of claims 1 to 7, characterized in that, The step of obtaining the human voice frequency domain signal from the m-th frame audio signal includes: When the m-th frame audio signal is a two-channel stereo signal, the human voice frequency domain signal is obtained from the m-th frame audio signal using the following equation: in, The frequency domain signal of the human voice is represented by L, and the time domain signal of the left channel is represented by L. This represents the left channel frequency domain signal. Let R represent the conjugate function of the left channel frequency domain signal, and let R represent the right channel time domain signal. This represents the right channel frequency domain signal. The conjugate function of the right channel frequency domain signal is denoted by ; where STFT() represents the short-time Fourier transform function. in, This represents the cross-correlation function between the left channel frequency domain signal and the right channel frequency domain signal. This represents the autocorrelation function of the left channel frequency domain signal. The autocorrelation function represents the frequency domain signal of the right channel; in, ψ (m,k) represents the similarity function between the left channel frequency domain signal and the right channel frequency domain signal in the m-th frame.

10. The method according to claim 9, characterized in that, The method further includes: The optimized equation is obtained by using the following formula. ψ (m,k): in, α Represents the inter-frame smoothing coefficient. α ∈(0, 1); in, ψ (m,k) represents the similarity function between the left channel frequency domain signal and the right channel frequency domain signal in the m-th frame. ψ (m-1,k) represents the similarity function between the left channel frequency domain signal and the right channel frequency domain signal in the (m-1)th frame.

11. The method according to any one of claims 1 to 7, characterized in that, The step of obtaining the background sound frequency domain signal from the m-th frame audio signal includes: When the m-th frame audio signal is a two-channel stereo signal, the background sound frequency domain signal is obtained from the m-th frame audio signal using the following equation: Where L and R represent the left channel time-domain signal and the right channel time-domain signal in the m-th frame audio signal, respectively, and s(t) and These represent the background sound time-domain signal and the background sound frequency-domain signal corresponding to the m-th frame audio signal, respectively.

12. The method according to any one of claims 1 to 7, characterized in that, The step of obtaining the human voice frequency domain signal from the m-th frame audio signal includes: When the m-th frame audio signal is a 5.1 channel or 7.1 channel stereo signal, the center channel time domain signal in the m-th frame audio signal is subjected to a short-time Fourier transform to obtain the human voice frequency domain signal.

13. The method according to any one of claims 1 to 7, characterized in that, The step of obtaining the background sound frequency domain signal from the m-th frame audio signal includes: When the m-th frame audio signal is a 5.1 channel stereo signal, the background sound frequency domain signal is obtained from the m-th frame audio signal using the following equation: Where L, Ls, R, and Rs represent the time-domain signals of the front left channel, the side surround left channel, the front right channel, and the side surround right channel in the m-th frame audio signal, respectively; s(t) and These represent the background sound time-domain signal and the background sound frequency-domain signal corresponding to the m-th frame audio signal, respectively; When the m-th frame audio signal is a 7.1 channel stereo signal, the background sound frequency domain signal is obtained from the m-th frame audio signal using the following equation: Where L, Ls, Lb, R, Rs, and Rb represent the time-domain signals of the front left channel, the side surround left channel, the rear left channel, the front right channel, the side surround right channel, and the rear right channel in the m-th frame audio signal, respectively; s(t) and These represent the background sound time-domain signal and the background sound frequency-domain signal corresponding to the m-th frame audio signal, respectively.

14. The method according to any one of claims 1 to 7, characterized in that, The calculation of the normalized spectral flux corresponding to the m-th frame audio signal includes: The normalized spectral flux V(m) corresponding to the m-th frame audio signal is calculated using the following equation: Where STFT() represents the short-time Fourier transform function; Where, c(t) and These represent the time-domain signal and frequency-domain signal of the human voice corresponding to the m-th frame of the audio signal, respectively. This represents the human voice frequency domain signal corresponding to the (m-1)th frame of the audio signal. This represents the spectral flux of the human voice frequency domain signal; Where, s(t) and These represent the background sound time-domain signal and the background sound frequency-domain signal corresponding to the m-th frame audio signal, respectively. This represents the background audio frequency domain signal corresponding to the (m-1)th frame audio signal. This represents the spectral flow rate of the background sound frequency domain signal.

15. The method according to claim 14, characterized in that, The method further includes: When the m-th frame audio signal is a two-channel stereo signal, c(t) and s(t) are calculated according to the following equations: Where L and R represent the left channel time-domain signal and the right channel time-domain signal in the m-th frame audio signal, respectively, and β is a constant; When the m-th frame audio signal is a 5.1 channel signal, c(t) and s(t) are calculated according to the following equations: Wherein, C, L, Ls, R, and Rs represent the center channel time domain signal, the front left channel time domain signal, the side surround left channel time domain signal, the front right channel time domain signal, and the side surround right channel time domain signal in the m-th frame 5.1 channel signal, respectively. When the m-th frame audio signal is a 7.1 channel signal, c(t) and s(t) are calculated according to the following equations: Wherein, C, L, Ls, Lb, R, Rs, and Rb represent the center channel time domain signal, the front left channel time domain signal, the side surround left channel time domain signal, the rear left channel time domain signal, the front right channel time domain signal, the side surround right channel time domain signal, and the rear right channel time domain signal in the m-th frame 7.1 channel signal, respectively.

16. The method according to any one of claims 1 to 7, characterized in that, After adjusting the gain of the human voice frequency domain signal according to the human voice gain parameter and the normalized spectral flux to obtain the enhanced human voice frequency domain signal, the method further includes: The enhanced human voice frequency domain signal is downmixed to obtain a stereo frequency domain signal with enhanced human voice.

17. The method according to claim 16, characterized in that, The enhanced human voice frequency domain signal is downmixed to obtain a stereo frequency domain signal with enhanced human voice, including: When the m-th frame audio signal is a two-channel stereo signal, according to the following equation, the enhanced human voice frequency domain signal is downmixed to obtain a human voice enhanced stereo frequency domain signal: in, and The left and right channel frequency domain signals are represented respectively, and the stereo frequency domain signal for enhanced human voice includes... and ; in, This represents the enhanced human voice frequency domain signal. and These represent the frequency domain signals corresponding to the time domain signals of the left and right channels in the m-th frame of the audio signal, respectively.

18. The method according to claim 16, characterized in that, The enhanced human voice frequency domain signal is downmixed to obtain a stereo frequency domain signal with enhanced human voice, including: When the m-th frame audio signal is a 5.1 channel stereo signal, according to the following equation, the enhanced human voice frequency domain signal is downmixed to obtain the human voice enhanced stereo frequency domain signal: in, and The left and right channel frequency domain signals are represented respectively, and the stereo frequency domain signal for enhanced human voice includes... and ; in, This represents the enhanced human voice frequency domain signal. , , , These represent the frequency domain signals corresponding to the time domain signals of the left channel, right channel, left side surround channel, and right side surround channel in the m-th frame 5.1 channel signal, respectively.

19. The method according to claim 16, characterized in that, The step of downmixing the enhanced human voice frequency domain signal to obtain a stereo frequency domain signal with enhanced human voice includes: When the m-th frame audio signal is a 7.1-channel stereo signal, according to the following equation, the enhanced human voice frequency domain signal is downmixed to obtain the human voice enhanced stereo frequency domain signal: in, and The left and right channel frequency domain signals are represented respectively, and the stereo frequency domain signal for enhanced human voice includes... and ; in, This represents the enhanced human voice frequency domain signal. , , , , , These represent the frequency domain signals corresponding to the time domain signals of the left channel, right channel, side surround left channel, side surround right channel, rear left channel, and rear right channel in the m-th frame 7.1 channel signal, respectively.

20. The method according to claim 16, characterized in that, After performing downmixing processing on the enhanced human voice frequency domain signal to obtain a stereo frequency domain signal with enhanced human voice, the method further includes: The stereo frequency domain signal of the enhanced human voice is converted into a stereo time domain signal of the enhanced human voice using the Fourier transform algorithm. The enhanced stereo time-domain signal of the human voice is output through the sound output device; The sound output device is a speaker or headphones.

21. The method according to any one of claims 1 to 7, characterized in that, Before obtaining the human voice frequency domain signal and the background sound frequency domain signal from the m-th frame audio signal, the method further includes: Determine whether the normalized spectral flow is greater than a first preset threshold; When the normalized spectral flow is greater than the first preset threshold, it is determined that the m-th frame audio signal contains speech components.

22. The method according to claim 21, characterized in that, After determining whether the normalized spectral flow is greater than a first preset threshold, the method further includes: When the normalized spectral flow is less than or equal to the first preset threshold, it is determined that the m-th frame audio signal does not contain speech components; When the m-th frame audio signal is a stereo signal, the m-th frame stereo time-domain signal without human voice enhancement is output through the sound output device; When the m-th frame audio signal is a 5.1 / 7.1 channel signal, the m-th frame 5.1 / 7.1 channel signal is downmixed to obtain the m-th frame stereo frequency domain signal without voice enhancement; the m-th frame stereo frequency domain signal without voice enhancement is converted into the m-th frame stereo time domain signal without voice enhancement, and the m-th frame stereo time domain signal without voice enhancement is output through the sound output device. The sound output device is a speaker or headphones.

23. An electronic device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory, the processor executing the computer program to cause the electronic device to perform the method as described in any one of claims 1 to 22.

24. A chip comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, The processor is used to execute the computer program to implement the method as described in any one of claims 1 to 22.

25. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 22.

Images