Signal processing method for enhancing human voice, electronic equipment and storage medium

By dynamically adjusting the voice gain, the problem of unclear voices when playing videos on electronic devices was solved, achieving the effect of clearly hearing voices in different environments and improving the user experience.

CN118942477AActive Publication Date: 2024-11-12HONOR DEVICE CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202310533752.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-11
Publication Date
2024-11-12
Estimated Expiration
2043-05-11

AI Technical Summary

Technical Problem

When playing videos on electronic devices, users often cannot hear the dialogue clearly due to factors such as hardware capabilities, hearing ability, environmental noise, and the content of the video, which affects the user experience.

Method used

By acquiring the m-th frame audio signal of the multi-channel audio signal, calculating the normalized spectral flow, extracting the human voice and background sound frequency domain signals, dynamically adjusting the human voice gain based on the environmental noise and background sound frequency domain signals, and using an improved Wiener filtering algorithm to adjust the human voice frequency domain signal gain, the human voice signal is enhanced.

Benefits of technology

The system dynamically adjusts the voice gain in different environments to avoid sudden changes in volume, ensuring that users can hear voices clearly without disturbing others in quiet environments, and also hear voices clearly in noisy environments, thus improving the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942477A_ABST
    Figure CN118942477A_ABST
Patent Text Reader

Abstract

The invention provides a signal processing method for enhancing human voice, electronic equipment and a storage medium, and relates to the technical field of signal processing. In an audio and video playing scene of the electronic equipment, a human voice gain parameter can be determined according to environment noise and a background audio frequency domain signal and a human voice frequency domain signal corresponding to a multichannel audio stream, and then gain adjustment is performed on the human voice frequency domain signal according to the human voice gain parameter and a normalized spectrum flow to obtain an enhanced human voice frequency domain signal. Under the condition of different environmental noises, the gain of the human voice signal can be dynamically adjusted according to the input multi-channel audio signal, so that the effect of enhancing the human voice is achieved, the phenomenon that the hearing feeling is unstable is avoided, and the user experience is improved. Therefore, according to the scheme of the invention, the effect that the user can hear the voice of the video clearly without disturbing others when playing the video in a quiet environment or the effect that the user can hear the voice of the video clearly when playing the video in a noisy environment can be achieved, and the user experience can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of signal processing technology, and in particular to a signal processing method, electronic device and storage medium for enhancing human voice. Background Art

[0002] Currently, users often play movies, TV series and other videos on electronic devices with speakers, such as mobile phones, tablets, laptops, and smart screens. However, when playing videos on electronic devices, due to factors such as the hardware capabilities of the electronic devices, the user's own listening ability, the listening environment, and the content of the material, the user may not be able to hear the human voice dialogue in the video clearly, which reduces the user experience. Summary of the invention

[0003] The present application provides a signal processing method and electronic device for enhancing human voice, which can dynamically adjust the gain of the human voice signal for the input multi-channel audio signal according to the ambient noise conditions, so as to achieve the effect of enhancing the human voice and keep the output human voice stable, thereby avoiding the phenomenon of fluctuating in hearing.

[0004] In a first aspect, the present application provides a signal processing method for enhancing human voice, the method comprising: obtaining an m-th frame audio signal in a multi-channel audio stream; calculating a normalized spectral flow corresponding to the m-th frame audio signal; obtaining a human voice frequency domain signal and a background sound frequency domain signal from the m-th frame audio signal; determining a human voice gain parameter based on ambient noise, background sound frequency domain signals and human voice frequency domain signals; and adjusting the gain of the human voice frequency domain signal based on the human voice gain parameter and the normalized spectral flow to obtain an enhanced human voice frequency domain signal.

[0005] Through the signal processing method for enhancing human voice provided by the present application, in the scenario where the electronic device plays audio and video, the human voice gain parameter can be determined according to the environmental noise and the background sound frequency domain signal and the human voice frequency domain signal corresponding to the multi-channel audio stream, and then the gain of the human voice frequency domain signal can be adjusted according to the human voice gain parameter and the normalized spectrum flow rate to obtain the enhanced human voice frequency domain signal. Under different environmental noise conditions, the gain of the human voice signal can be dynamically adjusted for the input multi-channel audio signal to achieve the effect of enhancing the human voice, and the output human voice can be kept stable, thereby avoiding the phenomenon of fluctuating in hearing. Therefore, the present application solution can achieve the effect of playing the video in a quiet environment without disturbing others, or the effect of the user being able to hear the video voice clearly when playing the video in a noisy environment, which can improve the user experience.

[0006] It should be noted that the traditional Wiener filtering algorithm significantly improves the gain of signals with large signal spectrum intensity (called large signals), but the gain is not obvious for signals with small signal spectrum intensity (called small signals). It can be understood that in this case, the spectrum range of the human voice signal is relatively large. Among them, the spectrum range of the human voice signal refers to the spectrum range jointly composed of small signals and large signals. In other words, through the traditional Wiener filtering algorithm, the gain of large signals is obvious, and the gain of small signals is too small, which will cause the final hearing perception to fluctuate. In order to solve the problem of fluctuating hearing perception, the embodiment of the present application proposes an improved method for dynamically adjusting the gain of human voice.

[0007] In some possible implementations, determining the human voice gain parameter according to the ambient noise, the background sound frequency domain signal, and the human voice frequency domain signal includes: determining a first gain adjustment coefficient and a second gain adjustment coefficient according to the average amplitude of the ambient noise; and calculating the human voice gain parameter according to the first gain adjustment coefficient, the second gain adjustment coefficient, the background sound frequency domain signal, and the human voice frequency domain signal using the following equation:

[0008] G′(m,k)=p(1-G(m,k))+qG(m,k)

[0009]

[0010] SNR(m,k)=C(m,k) / S(m,k)

[0011] Among them, G′(m,k) represents the human voice gain parameter, p represents the first gain adjustment coefficient, q represents the second gain adjustment coefficient, C(m,k) represents the human voice frequency domain signal, and S(m,k) represents the background sound frequency domain signal.

[0012] It can be understood that the human voice gain estimation formula adopted in the present application is an improved formula based on the traditional Wiener filter formula. Through the human voice gain estimation formula adopted in the present application, the gains of human voice signals of different intensities within the human voice signal spectrum can be adjusted.

[0013] Through the embodiment scheme of the present application, the values ​​of the two coefficients p and q can be dynamically adjusted according to the current environmental noise level, so that the gain of the large signal can be adjusted, the gain of the small signal can be adjusted, or the gain of the large signal and the gain of the small signal can be adjusted at the same time, which can avoid the phenomenon of the hearing feeling being louder and softer. By dynamically adjusting the vocal gain, the effect of playing the video in a quiet environment without disturbing others and being able to hear the human voice in the video can be achieved, and the effect of the user being able to hear the human voice in the video when playing the video in a noisy environment can be achieved.

[0014] Through the solution of this application, in a quiet environment, the values ​​of the two coefficients p and q are dynamically adjusted, and the gain of the large signal is appropriately increased while the gain of the small signal is appropriately increased. Even if the volume is low when the electronic device plays a video, it can ensure that the user can clearly hear the strong voice dialogue in the video and the weak voice dialogue in the video. Thus, when playing a video in a quiet environment, it is achieved that the user can hear the human voice in the video clearly without disturbing others.

[0015] Through the application scheme, in a noisy environment, the values ​​of the two coefficients p and q are dynamically adjusted. While appropriately increasing the gain of large signals, the gain of small signals is also appropriately increased. This can ensure that the user can clearly hear both the strong voice dialogues in the video and the weak voice dialogues in the video, thereby achieving the effect that the user can clearly hear the human voice in the video when playing the video in a noisy environment.

[0016] Compared with the related art of only increasing the gain of large signals, the present application achieves the effect of compressing the dynamic range of the human voice signal by appropriately increasing the gain of small signals while increasing the gain of large signals, thereby avoiding the phenomenon of the sound being louder or smaller.

[0017] The following describes possible specific values ​​of the first gain adjustment coefficient p and the second gain adjustment coefficient q.

[0018] In some possible implementations, determining the first gain adjustment coefficient and the second gain adjustment coefficient according to the average amplitude of the ambient noise includes:

[0019] When N∈[a,b), p=0.02×N-0.8, q=1;

[0020] When N∈[b,c), p=0.15×N-9.25, q=-0.04×N+3.6;

[0021] Wherein, N represents the average amplitude of the ambient noise in decibels (dB), a is less than b, and b is less than c.

[0022] In some possible implementations, a is taken as 40 dB, b is taken as 65 dB, and c is taken as 75 dB.

[0023] Through the solution of the present application, the values ​​of p and q can be determined according to the average amplitude N of the ambient noise based on the linear mapping relationship between p, q and the average amplitude of the ambient noise.

[0024] It should be noted that the above-mentioned linear mapping relationship between p, q and the average amplitude of ambient noise is an exemplary description. The present application does not limit the linear mapping relationship between p, q and the average amplitude of ambient noise. It can be understood that in actual implementation, the correspondence between p, q and the average amplitude of ambient noise can be set according to actual usage requirements.

[0025] In some possible implementations, the method provided by the present application is applied to an electronic device, which includes a microphone. The above method also includes: determining an ambient noise signal according to an audio signal collected by the microphone; and determining an average amplitude of the ambient noise according to the ambient noise signal, sampling duration, sampling frequency, and minimum sound pressure intensity.

[0026] Through the solution of this application, the environmental noise estimation module will periodically call the microphone recording of the electronic device (such as a mobile phone) to collect environmental noise signals, and then estimate the noise level of the environment in which the electronic device is currently located by analyzing the environmental noise signals. Based on power consumption considerations, the microphone is called every x minutes to collect y seconds of data. For example, the microphone is called every 30 minutes to collect 10 seconds of data.

[0027] In some possible implementations, the electronic device also includes a speaker, and the above-mentioned determining the ambient noise signal based on the audio signal collected by the microphone includes: when the speaker is working, determining the ambient noise signal based on the audio signal collected by the microphone and the audio signal played by the speaker; when the speaker is not working, determining the audio signal collected by the microphone as the ambient noise signal.

[0028] It should be noted that when the microphone collects environmental noise signals, if the electronic device is playing a video and the video sound is played through the speaker, the video sound will be transmitted through the environment and received and recorded by the microphone. In other words, the microphone recording result includes both environmental noise and the sound played by the speaker. Therefore, it is necessary to eliminate the signal played by the speaker and then estimate the noise level of the current environment to improve the accuracy of calculating environmental noise.

[0029] In some possible implementations, determining the ambient noise signal according to the audio signal collected by the microphone and the audio signal played by the speaker includes:

[0030] The following equation is used to calculate the ambient noise signal based on the audio signal collected by the microphone and the audio signal played by the speaker: s(t) = d(t) - x(t) × S spk ×S mic . Among them, s(t) represents the environmental noise signal, d(t) represents the audio signal collected by the microphone, x(t) represents the audio signal played by the speaker, S spk Indicates the sensitivity of the speaker, Smic Indicates the sensitivity of the microphone.

[0031] Through the solution of the present application, if the speaker is not working (for example, no audio or video is played) when the microphone collects the ambient noise signal, the average amplitude of the collected data within y seconds can be directly calculated as the result of the ambient noise estimation. If the speaker is working when the microphone collects the ambient noise signal, the current ambient noise situation can be estimated based on the sensitivity of the speaker and the sensitivity of the microphone, which can improve the accuracy of calculating the ambient noise.

[0032] In some possible implementations, determining the average amplitude of the ambient noise according to the ambient noise signal, the sampling duration, the sampling frequency, and the minimum sound pressure intensity includes:

[0033] The following equation is used to calculate the average amplitude of the ambient noise based on the ambient noise signal, sampling duration, sampling frequency, and minimum sound pressure intensity:

[0034]

[0035] Wherein, N represents the average amplitude of ambient noise, s(t) represents the ambient noise signal, T represents the acquisition time, F represents the sampling frequency, and P0 represents the minimum sound pressure intensity.

[0036] In some possible implementations, the gain of the human voice frequency domain signal is adjusted according to the human voice gain parameter and the normalized spectrum flow to obtain an enhanced human voice frequency domain signal, including:

[0037] The enhanced human voice signal spectrum is calculated using the following equation based on the human voice gain parameter and the normalized spectrum flow:

[0038] C′(m,k)=C(m,k)+C(m,k)*G′(m,k)*V(m)

[0039] Among them, C′(m,k) represents the enhanced human voice frequency domain signal, C(m,k) represents the human voice frequency domain signal, G′(m,k) represents the human voice gain parameter, and V(m) represents the normalized spectrum flow corresponding to the m-th frame audio signal.

[0040] Through the application scheme, the gain of the human voice frequency domain signal is adjusted according to the human voice gain parameter and the normalized spectrum flow, and the enhanced human voice frequency domain signal is obtained. By dynamically adjusting the human voice gain, the effect of playing the video in a quiet environment without disturbing others and the effect of the user being able to hear the video voice clearly when playing the video in a noisy environment can be achieved, and the phenomenon of the hearing feeling being loud and soft can be avoided, which can improve the user experience.

[0041] In some possible implementations, the step of obtaining a human voice frequency domain signal from the m-th frame audio signal includes:

[0042] When the m-th frame audio signal is a two-channel stereo signal, the following equation is used to obtain the human voice frequency domain signal from the m-th frame audio signal:

[0043] C(m,k)=ψ(m,k)×[F L (m,k)+F R (m,k)] / 2

[0044]

[0045]

[0046]

[0047]

[0048] F L (m,k)=STFT(L)

[0049] F R (m,k)=STFT(R)

[0050] Among them, C(m,k) represents the frequency domain signal of human voice, L represents the time domain signal of the left channel, and F L (m,k) represents the frequency domain signal of the left channel, represents the conjugate function of the left channel frequency domain signal, R represents the right channel time domain signal, F R (m,k) represents the frequency domain signal of the right channel, represents the conjugate function of the right channel frequency domain signal; where STFT() represents the short-time Fourier transform function. LR (m, k) represents the cross-correlation function between the left channel frequency domain signal and the right channel frequency domain signal, φ LL (m,k) represents the autocorrelation function of the left channel frequency domain signal, φ RR (m, k) represents the autocorrelation function of the right channel frequency domain signal; wherein, ψ(m, k) represents the similarity function between the left channel frequency domain signal and the right channel frequency domain signal of the mth frame.

[0051] In some possible implementations, the method provided in the present application further includes: using the following equation to obtain an optimized ψ(m,k):

[0052] ψ(m,k)=(1-α)ψ(m-1,k)+αψ(m,k)

[0053] Among them, α represents the inter-frame smoothing coefficient, α∈(0,1); among them, ψ(m,k) represents the similarity function between the left channel frequency domain signal and the right channel frequency domain signal of the m-th frame, and ψ(m-1,k) represents the similarity function between the left channel frequency domain signal and the right channel frequency domain signal of the m-1-th frame.

[0054] In some possible implementations, the step of obtaining the background audio frequency domain signal from the m-th frame audio signal includes:

[0055] When the m-th frame audio signal is a two-channel stereo signal, the following equation is used to obtain the background audio frequency domain signal from the m-th frame audio signal:

[0056] s(t)=LR

[0057] S(m,k)=STFT[s(t)]

[0058] Wherein, L and R represent the left channel time domain signal and the right channel time domain signal in the m-th frame audio signal respectively, and s(t) and S(m,k) represent the background sound time domain signal and the background sound frequency domain signal corresponding to the m-th frame audio signal respectively.

[0059] In some possible implementations, the above-mentioned obtaining of the human voice frequency domain signal from the m-th frame audio signal includes: when the m-th frame audio signal is a 5.1-channel or 7.1-channel stereo signal, performing a short-time Fourier transform on the center channel time domain signal in the m-th frame audio signal to obtain the human voice frequency domain signal.

[0060] In some possible implementations, the obtaining of the background audio frequency domain signal from the m-th frame audio signal includes: when the m-th frame audio signal is a 5.1-channel stereo signal, obtaining the background audio frequency domain signal from the m-th frame audio signal using the following equation:

[0061] s(t)=(L+Ls)-(R+Rs)

[0062] S(m,k)=STFT[s(t)]

[0063] Among them, L, Ls, R, and Rs respectively represent the front left channel time domain signal, side surround left channel time domain signal, front right channel time domain signal, and side surround right channel time domain signal in the m-th frame audio signal; s(t) and S(m,k) respectively represent the background sound time domain signal and background sound frequency domain signal corresponding to the m-th frame audio signal.

[0064] When the m-th frame audio signal is a 7.1-channel stereo signal, the following equation is used to obtain the background audio frequency domain signal from the m-th frame audio signal:

[0065] s(t)=(L+Ls+Lb)-(R+Rs+Rb)

[0066] S(m,k)=STFT[s(t)]

[0067] Among them, L, Ls, Lb, R, Rs, and Rb respectively represent the front left channel time domain signal, side surround left channel time domain signal, rear left channel time domain signal, front right channel time domain signal, side surround right channel time domain signal, and rear right channel time domain signal in the m-th frame audio signal; s(t) and S(m,k) respectively represent the background sound time domain signal and background sound frequency domain signal corresponding to the m-th frame audio signal.

[0068] In some possible implementations, the calculating of the normalized spectrum flow corresponding to the m-th frame of the audio signal includes:

[0069] The normalized spectral flow V(m) corresponding to the m-th frame audio signal is calculated using the following equation:

[0070]

[0071]

[0072]

[0073] C(m,k)=STFT[c(t)]

[0074] S(m,k)=STFT[s(t)]

[0075] Where STFT() represents the short-time Fourier transform function. Where c(t) and C(m,k) represent the human voice time domain signal and human voice frequency domain signal corresponding to the m-th frame audio signal, respectively, C(m-1,k) represents the human voice frequency domain signal corresponding to the m-1-th frame audio signal, δ C (m,k) represents the spectrum flow of the human voice frequency domain signal. Among them, s(t) and S(m,k) represent the background sound time domain signal and background sound frequency domain signal corresponding to the m-th frame audio signal, respectively, S(m-1,k) represents the background sound frequency domain signal corresponding to the m-1-th frame audio signal, δ S (m,k) represents the spectrum flow of the background audio frequency domain signal.

[0076] In some possible implementations, the method provided by the present application further includes: when the m-th frame audio signal is a two-channel stereo signal, calculating c(t) and s(t) according to the following equation:

[0077]

[0078] s(t)=LR

[0079] Wherein, L and R represent the left channel time domain signal and the right channel time domain signal in the m-th frame audio signal respectively, and β is a constant.

[0080] When the m-th frame audio signal is a 5.1-channel signal, c(t) and s(t) are calculated according to the following equations:

[0081] c(t)=C

[0082] s(t)=(L+Ls)-(R+Rs)

[0083] Wherein, C, L, Ls, R, and Rs represent the center channel time domain signal, the front left channel time domain signal, the side surround left channel time domain signal, the front right channel time domain signal, and the side surround right channel time domain signal in the m-th frame 5.1 channel signal respectively;

[0084] When the m-th frame audio signal is a 7.1-channel signal, c(t) and s(t) are calculated according to the following equations:

[0085] c(t)=C

[0086] s(t)=(L+Ls+Lb)-(R+Rs+Rb)

[0087] Among them, C, L, Ls, Lb, R, Rs, and Rb respectively represent the center channel time domain signal, the front left channel time domain signal, the side surround left channel time domain signal, the rear left channel time domain signal, the front right channel time domain signal, the side surround right channel time domain signal, and the rear right channel time domain signal in the 7.1 channel signal of the mth frame.

[0088] In some possible implementations, after the gain of the human voice frequency domain signal is adjusted according to the human voice gain parameter and the normalized spectral flow to obtain an enhanced human voice frequency domain signal, the method further includes: performing downmixing processing according to the enhanced human voice frequency domain signal to obtain a stereo frequency domain signal with human voice enhancement.

[0089] The enhanced human voice frequency domain signal is down-mixed with the multi-channel signal to obtain a human voice enhanced stereo frequency domain signal, and then the human voice enhanced stereo frequency domain signal is further converted into a human voice enhanced stereo time domain signal, thus converted into a signal that can be output by a speaker or headphone.

[0090] In some possible implementations, the downmixing process is performed on the enhanced human voice frequency domain signal to obtain a human voice enhanced stereo frequency domain signal, including:

[0091] When the m-th frame audio signal is a two-channel stereo signal, downmixing is performed on the enhanced human voice frequency domain signal according to the following equation to obtain a human voice enhanced stereo frequency domain signal:

[0092] F′ L =F L +C′(m,k)

[0093] F′ R =F R +C′(m,k)

[0094] Among them, F′ L and F′ R They represent the stereo frequency domain signals of human voice enhancement respectively; among them, C′(m,k) represents the enhanced human voice frequency domain signal, F L and F R Respectively represent the frequency domain signals corresponding to the left / right channel time domain signals in the m-th frame audio signal.

[0095] In some possible implementations, the downmixing process is performed on the enhanced human voice frequency domain signal to obtain a human voice enhanced stereo frequency domain signal, including:

[0096] When the m-th frame audio signal is a 5.1-channel stereo signal, downmixing is performed on the enhanced human voice frequency domain signal according to the following equation to obtain a human voice enhanced stereo frequency domain signal:

[0097] F′ L =0.4143*(F L +0.707C′(m,k)+0.707F Ls )

[0098] F′ R =0.4143*(F R +0.707C′(m,k)+0.707F Rs )

[0099] Among them, F′ L and F′ R They represent the stereo frequency domain signals of human voice enhancement respectively; among them, C′(m,k) represents the enhanced human voice frequency domain signal, F L 、F R 、F Ls 、F Rs They respectively represent the frequency domain signals corresponding to the time domain signals of the left channel, right channel, side surround left channel, and side surround right channel in the mth frame 5.1 channel signal.

[0100] In some possible implementations, downmixing is performed according to the enhanced human voice frequency domain signal to obtain a human voice enhanced stereo frequency domain signal, including:

[0101] When the m-th frame audio signal is a 7.1-channel stereo signal, downmixing is performed on the enhanced human voice frequency domain signal according to the following equation to obtain a human voice enhanced stereo frequency domain signal:

[0102] F′ L=0.3204*(F L +0.707C′(m,k)+0.707F Ls +0.707F Lb )

[0103] F′ R =0.3204*(F R +0.707C′(m,k)+0.707F Rs +0.707F Rb )

[0104] Among them, F′ L and F′ R They represent the stereo frequency domain signals of human voice enhancement respectively; among them, C′(m,k) represents the enhanced human voice frequency domain signal, F L 、F R 、F Ls 、F Rs 、F Lb 、F Rb They respectively represent the frequency domain signals corresponding to the time domain signals of the left channel, right channel, side surround left channel, side surround right channel, rear left channel, and rear right channel in the 7.1 channel signal of the mth frame.

[0105] In some possible implementations, after the above-mentioned downmixing processing is performed according to the enhanced human voice frequency domain signal to obtain a stereo frequency domain signal with enhanced human voice, the method provided in the present application also includes: converting the stereo frequency domain signal with enhanced human voice into a stereo time domain signal with enhanced human voice through a Fourier transform algorithm; outputting the stereo time domain signal with enhanced human voice through a sound output device; wherein the sound output device is a speaker or a headset.

[0106] The present application scheme is compatible with processing multi-channel audio signals such as two-channel stereo, 5.1 channels and 7.1 channels. By dynamically adjusting the vocal gain, it finally outputs an audio signal with enhanced vocals. This can achieve the effect of playing the video in a quiet environment without disturbing others while allowing the user to clearly hear the vocals in the video, and can also achieve the effect of allowing the user to clearly hear the vocals in the video when playing the video in a noisy environment.

[0107] In some possible implementations, before obtaining the human voice frequency domain signal and the background sound frequency domain signal from the m-th frame audio signal, the method provided in the present application also includes: determining whether the normalized spectrum flow is greater than a first preset threshold; when the normalized spectrum flow is greater than the first preset threshold, determining that the m-th frame audio signal contains a speech component.

[0108] It should be noted that the m-th frame audio signal can be determined to contain speech components by judging whether the normalized spectral flow rate is greater than a first preset threshold value, and then, based on whether the m-th frame audio signal contains speech components, it is determined whether it is necessary to perform signal processing for enhancing the human voice on the m-th frame multi-channel audio signal.

[0109] On the one hand, if the m-th frame audio signal contains speech components, then the multi-channel audio signal of the m-th frame needs to be processed to enhance the human voice. The signal processing to enhance the human voice includes: obtaining the human voice frequency domain signal and the background sound frequency domain signal from the m-th frame audio signal; determining the human voice gain parameter according to the environmental noise, the background sound frequency domain signal and the human voice frequency domain signal; and adjusting the gain of the human voice frequency domain signal according to the human voice gain parameter and the normalized spectrum flow to obtain the enhanced human voice frequency domain signal.

[0110] On the other hand, if the m-th frame audio signal does not contain a speech component, then there is no need to perform signal processing for enhancing the human voice on the m-th frame multi-channel audio signal.

[0111] Specifically, after the above-mentioned determination of whether the normalized spectral flow rate is greater than the first preset threshold, the method provided in the present application also includes: when the normalized spectral flow rate is less than or equal to the first preset threshold, determining that the m-th frame audio signal does not contain a speech component, in which case there is no need to perform signal processing for enhancing the human voice on the multi-channel audio signal of the m-th frame, but instead perform normal downmixing processing and time-to-frequency conversion processing / frequency-to-time conversion processing.

[0112] For example, when the m-th frame audio signal is a stereo signal, the m-th frame stereo time domain signal without human voice enhancement is output through a sound output device; when the m-th frame audio signal is a 5.1 / 7.1 channel signal, the m-th frame 5.1 / 7.1 channel signal is downmixed to obtain the m-th frame stereo frequency domain signal without human voice enhancement; the stereo frequency domain signal without human voice enhancement is converted into the m-th frame stereo time domain signal without human voice enhancement, and the m-th frame stereo time domain signal without human voice enhancement is output through a sound output device; wherein the sound output device is a speaker or a headset.

[0113] In a second aspect, the present application provides a signal processing device for enhancing human voice, the device comprising a unit for executing the method in the first aspect. The device may correspond to executing the method described in the first aspect, and the relevant description of the unit in the device can refer to the description of the first aspect, which will not be repeated here for the sake of brevity.

[0114] The method described in the first aspect above can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a processing module or unit, a display module or unit, etc.

[0115] In a third aspect, the present application provides an electronic device, comprising a processor, a computer program or instructions stored in the processor and a memory, wherein the processor is configured to execute the computer program or instructions so that the method in the first aspect is executed.

[0116] In a fourth aspect, the present application provides a computer-readable storage medium on which a computer program (also referred to as instructions or codes) for implementing the method in the first aspect is stored. For example, when the computer program is executed by a computer, the computer can perform the method in the first aspect.

[0117] In a fifth aspect, the present application provides a chip, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof. Optionally, the chip also includes a memory, and the memory is connected to the processor via a circuit or wire.

[0118] In a sixth aspect, the present application provides a chip system, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof. Optionally, the chip system also includes a memory, and the memory is connected to the processor via a circuit or wire.

[0119] In a seventh aspect, the present application provides a computer program product, which includes a computer program (also referred to as instructions or codes). When the computer program is executed by an electronic device, the electronic device implements the method in the first aspect.

[0120] It can be understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] Figure 1 A schematic diagram of an application scenario of the signal processing method for enhancing human voice provided in an embodiment of the present application;

[0122] Figure 2 A schematic diagram of an application scenario analysis of a signal processing method for enhancing human voice provided in an embodiment of the present application;

[0123] Figure 3 It is a schematic diagram of the processing flow of various channel audio signals in the related technology;

[0124] Figure 4 A schematic diagram of a flow chart of an electronic device processing an audio signal in a related art;

[0125] Figure 5A schematic diagram of a signal processing flow when an electronic device plays a video scene provided by an embodiment of the present application;

[0126] Figure 6 A schematic diagram of the processing flow of various channel audio signals provided in an embodiment of the present application;

[0127] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0128] Figure 8 A schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application;

[0129] Fig. 9 A schematic diagram of a flow chart of a signal processing method for enhancing human voice provided in an embodiment of the present application;

[0130] Fig.10 A schematic diagram of the module interaction timing of the signal processing method for enhancing human voice provided in an embodiment of the present application;

[0131] Fig.11 Another schematic diagram of a flow chart of a signal processing method for enhancing human voice provided in an embodiment of the present application;

[0132] Fig.12 A schematic diagram of waveforms when detecting and extracting a human voice signal from a two-channel stereo signal in a signal processing method for enhancing human voice provided in an embodiment of the present application;

[0133] Fig.13 A schematic diagram of a microphone collecting ambient noise and a speaker playing sound in a signal processing method for enhancing human voice provided in an embodiment of the present application;

[0134] Fig.14 A schematic diagram of a waveform of a sound played by a speaker in the signal processing method for enhancing human voice provided in an embodiment of the present application;

[0135] Fig.15 A schematic diagram of a waveform of a signal collected by a microphone in a signal processing method for enhancing human voice provided in an embodiment of the present application;

[0136] Fig.16 A schematic diagram of a curve mapping relationship between coefficients p and q and the ambient noise level in the signal processing method for enhancing human voice provided in an embodiment of the present application;

[0137] Fig.17 A schematic diagram of the structure of a signal processing device for enhancing human voice provided in an embodiment of the present application. DETAILED DESCRIPTION

[0138] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0139] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0140] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of objects. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two, for example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0141] Currently, users often play video resources (such as movies and TV series) through electronic devices with speakers such as mobile phones, tablets, laptops, and smart screens. Figure 1 As shown in (a) in Figure 1, in a noisy environment, the user watches a video on a mobile phone. Figure 1 As shown in (b), in a quiet environment, the user watches a video on a smart TV.

[0142] However, when playing videos through electronic devices, various factors may affect the video in different environments, drowning out the human voices and dialogues in the video, causing users to be unable to hear the human voices and dialogues in the video. Figure 2 As shown in (a) in Figure 1, in a noisy environment, when playing a video, the human voice in the video may be masked by the external ambient sound, causing the user to be unable to hear the human voice in the video, even if the volume is increased. Figure 2 As shown in (b) in FIG. 1 , in a quiet environment, the volume is lowered when playing a video to avoid disturbing others. At this time, the human voice dialogue is masked by other sounds in the video, causing the user to not be able to hear the human voice dialogue in the video clearly.

[0143] In response to the above problems, an embodiment of the present application provides a signal processing method for enhancing human voice based on digital signal processing. The method is applied to electronic devices. By processing the human voice enhancement algorithm provided by the present application scheme, the input multi-channel audio signal can be improved to improve the clarity of the human voice, which can solve the above problems to a certain extent and improve the user experience.

[0144] To facilitate understanding of the embodiments of the present application, some terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0145] Channel: Channels usually refer to independent audio signal channels collected or played back at different spatial locations in audio production and playback. The number of channels may be different for different audio / video sources. For example, the currently commonly used multi-channel audio signals use dual channels, or 5.1 channels, or 7.1 channels. Among them, in the embodiments of the present application, audio / video sources are collectively referred to as audio sources, and the audio source is in the form of an audio stream, which includes multiple frames of continuous multi-channel audio signals.

[0146] Stereo: In audio technology, stereo is the stereo surround effect of sound obtained through recording, transmission and playback systems. Among them, for two-channel stereo, 5.1-channel, 7.1-channel audio sources, when played through the speakers or headphones of electronic devices, a stereo effect can be produced.

[0147] Downmixing of multi-channel signals: Using a "down conversion" transfer function to process multi-channel signals so that the number of output channels is less than the number of input channels is called downmixing. Among them, binaural rendering from two-channel to binaural audio is also called downmixing.

[0148] Dual channel: Figure 3 As shown in (a) in FIG. 1 , the two-channel includes a left channel and a right channel. In the related art, when an electronic device receives a two-channel audio source, the electronic device outputs a two-channel stereo signal through a speaker or earphone.

[0149] 5.1 channel: Figure 3 As shown in (b) in FIG. 5 , the 5.1 channel includes five channels, namely, the center channel C, the front left channel L, the front right channel R, the side surround left channel Ls, and the right surround channel Rs. In the related art, when an electronic device receives a 5.1 channel audio source, the electronic device downmixes the five channel signals to obtain a two-channel stereo signal, and then plays the two-channel stereo signal through a speaker or earphone.

[0150] 7.1 channel: Figure 3As shown in (c) in the figure, the 7.1 channel includes seven channels, namely, the center channel C, the front left channel L, the front right channel R, the side surround left channel Ls, the right surround channel Rs, the rear left channel Lb, and the rear right channel Rb. In the related art, when an electronic device receives a 7.1 channel audio source, the electronic device downmixes the seven channel signals to obtain a two-channel stereo signal, and then plays the two-channel stereo signal through a speaker or headphones.

[0151] It should be noted that the signal processing method for enhancing human voice provided in the embodiment of the present application can be applied to process the above-mentioned two-channel audio source, and can also process a 5.1-channel audio source or a 7.1-channel audio source. It can be understood that in actual implementation, the signal processing method for enhancing human voice provided in the embodiment of the present application can also be applied to process other multi-channel audio sources. For ease of explanation, the following embodiments will take the processing of a two-channel audio source, a 5.1-channel audio source, and a 7.1-channel audio source as examples to exemplarily illustrate the signal processing method for enhancing human voice provided in the embodiment of the present application.

[0152] In order to facilitate the description of the improvements of the embodiments of the present application relative to the related art, the following first describes the audio processing strategy of the electronic device in the related art when playing a video. Figure 4 The audio output strategy of the electronic device when playing a video in the related art is shown. Exemplarily, the functional modules used in the related art include software modules of the electronic device such as audio and video applications and downmixing modules, and hardware modules of the electronic device such as speakers / headphones.

[0153] S101: The audio and video application receives an operation triggered by a user to play a video.

[0154] S102: In response to a user operation, the audio and video application obtains an audio stream.

[0155] S103: The audio and video application determines whether the acquired audio stream is a two-channel audio stream.

[0156] S104: When the acquired audio stream is a two-channel audio stream, the audio and video application sends the two-channel audio stream to a speaker / headphone.

[0157] S105: The speaker / headphone outputs a two-channel audio stream frame by frame.

[0158] S106: When the audio stream is not a two-channel audio stream, the audio and video application sends the acquired audio stream to the downmixing module.

[0159] S107, the downmixing module downmixes the multi-channel audio stream to obtain a dual-channel audio stream.

[0160] For the sake of convenience, an audio stream with a channel number greater than 2 is referred to as a multi-channel audio stream. Exemplarily, the multi-channel audio stream may be a 5.1-channel audio stream or a 7.1-channel audio stream.

[0161] S108: The downmixing module transmits the two-channel audio stream obtained by downmixing to a speaker / headphone.

[0162] S109: The speaker / headphone outputs a two-channel audio stream frame by frame.

[0163] pass Figure 4 It can be seen that in the related art, when an electronic device plays a dual-channel or multi-channel audio stream, the human voice / speech in the audio stream is not enhanced. Therefore, when the electronic device is in a noisy environment, even if the volume is increased, there is a situation where the human voice in the audio stream is masked by the surrounding noisy noise, or when the electronic device is in a quiet environment and the media volume of the electronic device is low, the human voice in the audio stream is masked by the background sound in the audio stream, causing the user to be unable to hear the human voice in the audio stream clearly.

[0164] In order to better understand the embodiments of the present application, the embodiments of the present application are briefly described below:

[0165] The present application is intended to enable users to clearly hear the human voice dialogues in film and television dramas in a quiet environment without disturbing others; and to resist external environmental noise in a noisy environment so that users can clearly hear the human voice dialogues in film and television dramas.

[0166] The signal processing method for enhancing human voice provided in the embodiment of the present application can be applied to audio output scenarios where the input signal is a multi-channel audio signal, such as a video playback scenario. In a video playback scenario (such as playing a film or TV series), the audio includes human voice dialogues between actors and background music, and the human voice dialogues between actors in the audio are collected to obtain a human voice signal; then, the human voice signal appearing in the video can be enhanced through the solution of the present application, so that the user can clearly hear the human voice dialogue when playing the video.

[0167] It should be noted that, in actual implementation, the signal processing method for enhancing human voice provided by the embodiment of the present application can also be applied to other more possible scenarios, for example, the embodiment of the present application can also be applied to the scenario of playing multi-channel music through an audio application, and the embodiment of the present application does not limit this. For the sake of convenience, the signal processing scenario of enhancing human voice applied in the present application is described here by taking the electronic device playing a video as an example.

[0168] For example, if Figure 5As shown, when the user triggers the electronic device to play a video, the electronic device first obtains the video file and puts the video file into the memory buffer. These video files are encapsulated format data, including audio streams and video streams. The encapsulation format can be AVI, MOV or MP4 and other formats, and the video file can be figuratively understood as a container for various different encoded data. After obtaining the video file, the electronic device performs decapsulation format processing on the video file, separates the video stream and the audio stream, and obtains audio compressed data and video compressed data. Furthermore, the electronic device decodes the audio compressed data to obtain audio sampling data, and decodes the video compressed data to obtain video pixel data.

[0169] After obtaining the audio sampling data, the common signal processing method in the related art is: the electronic device synchronously processes the audio sampling data and the video pixel data to obtain the video data to be played. Finally, based on the video data to be played, the video screen is displayed on the display and the video sound is output through the speaker / headphone. In this case, when playing the video, the human voice dialogue in the video will be covered by the surrounding environment sound or the background sound in the video, causing the user to not be able to hear the human voice dialogue in the video clearly.

[0170] Different from the related art, the present application adds a signal processing step for enhancing the human voice. Specifically, the electronic device performs signal processing on the audio sampling data through a preset human voice enhancement algorithm to enhance the human voice. The processed audio sampling data is then synchronously processed with the video pixel data to obtain the video data to be played. Finally, based on the video data to be played, the video screen is displayed on the display and the video sound is output through the speaker / headphone. In this case, since the signal enhancement processing is performed on the human voice in the video, when playing the video, even if there is the influence of noise such as the surrounding environment sound or the background sound in the video, the user can clearly hear the human voice dialogue in the video.

[0171] The above describes the signal processing process of adding enhanced human voice in the video playback scenario. The following describes the possible implementation methods of signal processing for audio signals of different channels according to the embodiments of the present application in conjunction with the accompanying drawings.

[0172] Figure 6 A schematic diagram showing signal processing for audio signals of different channels in an embodiment of the present application is shown.

[0173] like Figure 6As shown in (a) in the figure, the audio stream is a two-channel stereo signal, and the two channels include a left channel (denoted as L) and a right channel (denoted as R). The two-channel stereo signal is subjected to vocal extraction processing to obtain a vocal signal, which is then enhanced, and the enhanced vocal signal is down-mixed with the background sound signal, and the down-mixed two-channel stereo signal is then output through a speaker or earphone.

[0174] like Figure 6 As shown in (b), the audio stream is a 5.1-channel audio signal, and the 5.1 channels include a center channel (denoted as C), a left channel L, a right channel R, a side surround left channel Ls, and a side surround right channel Rs. Since the center channel C in the 5.1-channel audio signal is used to carry the vocal signal, the center channel C is directly used as the vocal signal, and then the vocal signal is enhanced, and then the enhanced vocal signal is down-mixed with the background sound signal (i.e., the 4-channel signal), and then the down-mixed two-channel stereo signal is output through a speaker or earphone.

[0175] like Figure 6 As shown in (c), the audio stream is a 7.1-channel audio signal, and the 7.1 channels include a center channel C, a left channel L, a right channel R, a side surround left channel Ls, a side surround right channel Rs, a rear left channel (denoted as Lb), and a rear right channel (denoted as Rb). Since the center channel C in the 7.1-channel audio signal is used to carry the vocal signal, the center channel C is directly used as the vocal signal, and then the vocal signal is enhanced, and then the enhanced vocal signal is down-mixed with the background sound signal (i.e., the 6-channel signal), and then the down-mixed two-channel stereo signal is output through a speaker or earphone.

[0176] This application Figure 6 Related technologies Figure 3 By comparison, it can be seen that the signal processing process of extracting the human voice signal and enhancing the human voice signal is added in the embodiment of the present application. The specific implementation process of extracting the human voice signal and enhancing the human voice signal will be described in detail below.

[0177] The signal processing method for enhancing human voice provided in the embodiment of the present application can be applied to electronic devices with speaker or headphone playback function. For example, the electronic device can be a mobile phone, tablet computer, desktop, laptop, notebook computer, ultra-mobile personal computer (Ultra-mobile Personal Computer, UMPC), handheld computer, netbook, personal digital assistant (Personal Digital Assistant, PDA), wearable electronic device, smart watch and other devices. The present application does not impose any special restrictions on the specific form of the electronic device.

[0178] Figure 7 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown.

[0179] like Figure 7 As shown, the electronic device includes a processor, a memory, a display screen, an audio module, a speaker, a receiver, a microphone, and a headphone interface.

[0180] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0181] The memory can be used to store computer executable program codes, which include instructions. The processor calls the instructions stored in the memory to execute various functional applications and data processing of the electronic device.

[0182] The processor may include one or more processing units. For example, in the present application, the processor executes instructions in the memory to implement the signal processing method for enhancing human voice provided in the embodiment of the present application.

[0183] The display screen is used to display images, videos, etc. The electronic device may include 1 or N display screens, where N is an integer greater than 1.

[0184] The electronic device can realize audio / video functions through an audio module, a speaker, a receiver, a microphone, an earphone interface, and an application processor, etc. For example, music playback, video playback, recording, etc. The electronic device may include 2 or M speakers, where M is a positive integer greater than 2.

[0185] For example, when a user uses a speaker or headphones to listen to music, watch a video, play a game, etc., stereo playback can be achieved through at least two speakers.

[0186] When a person listens to a sound, the ear processes the sound information obtained from the left and right sides respectively, so as to distinguish the main direction from which the sound comes. Different sounds are transmitted through at least two speakers, so that the user's two ears read different sound information, thus achieving stereo playback.

[0187] In addition, an operating system is running on the above components, such as Hongmeng operating system, iOS operating system, Android open source operating system, Windows operating system, etc. Applications can be installed and run on the operating system.

[0188] The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device.

[0189] Figure 8 It is a software structure block diagram of the electronic device according to an embodiment of the present application.

[0190] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, Figure 8 As shown, the Android system can be divided into four layers, from top to bottom, namely the application layer, application framework layer, system library layer, and kernel layer.

[0191] The application layer may include a series of application packages. In the embodiment of the present application, the application layer may include audio and video applications, which refer to applications that can provide audio signals, such as music, video, games, browsers, instant messaging, etc.

[0192] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer, and can also be called the service layer. The application framework layer includes some predefined functions.

[0193] In the embodiment of the present application, the application framework layer includes an audio service module, wherein the audio service module includes a human voice detection module, a human voice extraction module and a channel number detection module.

[0194] The human voice detection module is used to detect whether the audio source contains a human voice signal, that is, to detect whether the audio source contains a speech component.

[0195] The vocal extraction module is used to extract the vocal signal from the audio source.

[0196] The channel number detection module is used to detect the number of channels contained in the audio source. For example, the channel number detection module is used to determine whether the audio signal in the audio source is a two-channel stereo signal, or a 5.1-channel or 7.1-channel audio signal.

[0197] For example, for a two-channel stereo signal, a human voice signal may be extracted from the two-channel stereo signal according to a preset human voice extraction algorithm.

[0198] For another example, for a 5.1-channel or 7.1-channel audio signal, since the center channel is used to transmit a human voice signal, the signal transmitted by the center channel can be directly obtained.

[0199] The system library can include multiple functional modules, such as media library, 3D graphics processing library, 2D graphics engine, etc.

[0200] In the embodiment of the present application, the system library includes signal processing modules such as an environmental noise estimation module, a speech gain estimation module and a downmixing module.

[0201] The environmental noise estimation module is used to detect the noise level of the environment in which the current electronic device is located, also known as the environmental noise level. The environmental noise level can be set according to actual use requirements, and the specific setting method will be described below.

[0202] The human voice gain estimation module is used to adjust the gain of the human voice signal extracted by the human voice extraction module according to the environmental noise level, so as to enhance the human voice signal.

[0203] The downmixing module is used to downmix the enhanced human voice signal with the background sound signal to obtain a two-channel stereo signal.

[0204] The kernel layer is a layer between hardware and software. In the embodiment of the present application, the kernel layer includes an audio driver and a display driver. The audio driver is used to trigger the speaker and / or earphone to output the audio signal obtained after mixing.

[0205] For ease of explanation, Figure 8 The hardware layer that interacts with the above software structure is also embodied. In the embodiment of the present application, the hardware layer includes a display, a speaker and an earphone interface. The earphone interface is a physical interface set on an electronic device for connecting an earphone.

[0206] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0207] The execution subject of the signal processing method for enhancing human voice provided in the embodiment of the present application can be the above-mentioned electronic device, or it can be a functional module and / or functional entity in the electronic device that can implement the signal processing method for enhancing human voice, and the present application scheme can be implemented by hardware and / or software, which can be determined according to actual use requirements, and the embodiment of the present application is not limited. Below, taking an electronic device as an example, the signal processing method for enhancing human voice provided in the embodiment of the present application is exemplarily described in combination with the accompanying drawings.

[0208] The following describes the signal processing method for enhancing human voice provided in the embodiments of the present application in conjunction with specific embodiments.

[0209] Return to the above reference Figure 8As shown, the functional modules adopted by the present application solution include audio and video application, human voice detection module, human voice extraction module, channel number detection module, environmental noise estimation module, human voice gain estimation module and downmixing module. Compared with the functional modules adopted by the related art, the present application solution adds human voice detection module, human voice extraction module, channel number detection module, environmental noise estimation module and human voice gain estimation module.

[0210] It should be noted that in the embodiments of the present application, human voice can also be called speech, for example, human voice detection can also be called speech detection, human voice extraction can also be called speech extraction, and human voice gain can also be called speech gain.

[0211] exist Figure 8 On the basis of Fig. 9 A schematic diagram showing the interaction of various functional modules in the signal processing method for enhancing human voice provided in an embodiment of the present application is shown.

[0212] like Fig. 9 As shown, the audio and video application obtains an audio stream, which includes multiple consecutive frames of multi-channel audio signals. It is assumed that the current frame signal obtained by the audio and video application is the m-th frame of the multi-channel audio signal, wherein the m-th frame of the multi-channel audio signal includes but is not limited to dual-channel stereo signals, 5.1-channel and 7.1-channel audio signals.

[0213] The audio and video application inputs the acquired m-th frame of multi-channel audio signal into the human voice detection module.

[0214] The human voice detection module detects whether the m-th frame multi-channel audio signal (current frame signal) contains a speech component.

[0215] On the one hand, when the detection result is that the m-th frame multi-channel audio signal (current frame signal) contains a speech component, the m-th frame multi-channel audio signal is subjected to a human voice enhancement process.

[0216] First, the channel number detection module determines whether the current frame signal is two-channel stereo or 5.1 / 7.1 channels.

[0217] Then, the human voice extraction module extracts the human voice signal in the m-th frame multi-channel audio signal and sends it to the human voice gain estimation module.

[0218] If the current frame signal is two-channel stereo, the human voice is extracted from the two-channel stereo by using a human voice extraction algorithm, and the extracted human voice signal is sent to the human voice gain estimation module.

[0219] If the current frame signal is a 5.1 / 7.1 channel, the channel number detection module directly uses the center channel in the 5.1 / 7.1 channel as a vocal signal and sends it to the vocal gain estimation module.

[0220] Then, the vocal gain estimation module receives the vocal signal sent by the vocal extraction module and the ambient noise level sent by the ambient noise estimation module. The vocal gain estimation module adjusts the gain of the vocal signal according to the ambient noise level to enhance the vocal. The vocal gain estimation module sends the enhanced vocal signal to the downmix module.

[0221] Then, the enhanced vocal signal and the multi-channel audio signal are down-mixed by the down-mixing module, the multi-channel audio signal is converted into a two-channel stereo signal, and then the two-channel stereo signal is output through a speaker or earphone.

[0222] On the other hand, when the detection result is that the m-th frame multi-channel audio signal (current frame signal) does not contain a voice component, the human voice enhancement process is not performed on the m-th frame multi-channel audio signal.

[0223] The specific processing method is: inputting the m-th frame multi-channel audio signal into a channel number detection module, and the channel number detection module detects the number of channels of the m-th frame multi-channel audio signal.

[0224] When it is detected that the multi-channel audio signal of the mth frame is a 5.1-channel and 7.1-channel audio signal, the downmixing module downmixes the 5.1-channel and 7.1-channel audio signals to convert the multi-channel audio signals into two-channel stereo signals, and then outputs the two-channel stereo signals through speakers or headphones.

[0225] When it is detected that the multi-channel audio signal of the mth frame is a two-channel stereo signal, the two-channel stereo signal is output through a speaker or a headphone.

[0226] Combination of the above Fig. 9 A flowchart briefly describes the interaction between the functional modules in the signal processing method for enhancing human voice provided in the embodiment of the present application. Fig.10 The timing diagram of the interaction of each functional module is shown in detail to illustrate the signal processing method for enhancing human voice provided in the embodiment of the present application.

[0227] First of all, it should be noted that the signal processing method for enhancing human voice provided in the embodiment of the present application can be applied to electronic devices such as mobile phones. The human voice enhancement function can be preset accordingly in electronic devices such as mobile phones. Electronic devices such as mobile phones start the human voice enhancement function and run the human voice enhancement function when conditions are met (such as when playing multi-channel audio), which can achieve the effect of enhancing human voice. The human voice enhancement function can be turned on by system default, or it can support user-defined settings to turn it on or off. For example, a user setting entry for the human voice enhancement function is added to electronic devices such as mobile phones, so that users can turn on or off the human voice enhancement function according to their needs.

[0228] like Fig.10As shown, the signal processing method for enhancing human voice provided in the embodiment of the present application includes steps S201-S225.

[0229] S201: The audio and video application receives an operation triggered by a user to play a video.

[0230] For example, the user triggering the video playback operation may be the user clicking a video playback button, or the user clicking a video application icon. The video playback may refer to on-demand video or live video, and the video format may be a regular video or a short video. A short video refers to a video whose playback duration is less than or equal to a preset duration (e.g., 15 seconds).

[0231] S202: In response to a user operation, the audio and video application obtains an audio stream.

[0232] For example, in response to a user clicking a video play button or a video application icon, the audio and video application obtains an audio stream in the video source. It is assumed here that the audio stream is an audio signal of two or more channels (such as 5.1 / 7.1 channels).

[0233] It should be noted that the input multi-channel audio signal is a time domain signal, which needs to be Fourier transformed to obtain a frequency domain signal. Then, through frequency domain signal processing, human voice detection, human voice extraction, and human voice enhancement can be achieved.

[0234] S203, the audio and video application notifies the human voice detection module, the human voice extraction module, the channel number detection module, the human voice gain estimation module, the ambient noise estimation module, the downmixing module and the post-processing module to start a human voice enhancement algorithm.

[0235] Accordingly, the human voice detection module, the human voice extraction module, the number of channels detection module, the human voice gain estimation module, the environmental noise estimation module, and the downmixing module initialize and start the human voice enhancement algorithm.

[0236] S204, the audio and video application sends the nth frame audio signal (referred to as the current frame signal) in the audio stream to the human voice detection module.

[0237] It should be noted that the audio and video application sends the audio stream frame by frame to the human voice detection module.

[0238] S205, the human voice detection module detects whether the current frame signal contains a speech component according to a preset human voice detection algorithm.

[0239] Among them, the preset human voice detection algorithm is an algorithm provided in the embodiment of the present application for detecting whether an audio signal contains a human voice signal, and the specific algorithm process will be described in detail below.

[0240] It should be noted that, during the detection process, the multi-channel audio signal in the time domain needs to be converted into a multi-channel audio signal in the frequency domain through a time-frequency conversion algorithm, and then human voice detection is performed in the frequency domain. This will be described in detail below.

[0241] S206, when the human voice detection module detects that the current frame signal does not contain a speech component, the human voice detection module sends the current frame signal to the channel number detection module, and the channel number detection module detects whether the current frame signal is a two-channel stereo signal.

[0242] On the one hand, if the current frame signal is a two-channel stereo signal, the following S207-S208 are executed. On the other hand, if the current frame signal is not a two-channel stereo signal, the following S209-S212 are executed.

[0243] S207, when the channel number detection module detects that the current frame signal is a two-channel stereo signal, the channel number detection module sends the two-channel stereo signal to the post-processing module, and the post-processing module performs frequency-time conversion on the two-channel stereo signal to obtain a two-channel stereo signal in the time domain, and then sends the two-channel stereo signal in the time domain to the speaker / headphone.

[0244] S208: The speaker / headphone outputs the two-channel stereo signal in the time domain.

[0245] S209: When the channel number detection module detects that the current frame signal is not a two-channel stereo signal, the channel number detection module sends the 5.1 / 7.1 channel audio signal to the downmixing module.

[0246] S210: The downmixing module downmixes the 5.1 / 7.1 channel audio signal into a two-channel stereo signal.

[0247] S211, the two-channel stereo signal is transmitted to the speaker / headphone after frequency-time conversion.

[0248] Since the two-channel stereo signal obtained after the down-mixing process is a frequency domain signal, it is necessary to perform conversion from the frequency domain to the time domain (frequency-time conversion).

[0249] Specifically, the downmixing module sends the two-channel stereo signal to the post-processing module, which performs frequency-time transformation on the two-channel stereo signal to obtain a two-channel stereo signal in the time domain, and then the post-processing module sends the two-channel stereo signal in the time domain to the speaker / headphone.

[0250] S212, the speaker / headphone outputs a two-channel stereo signal in the time domain.

[0251] The above describes the audio signal processing flow when it is detected that the current frame signal does not contain a voice component. The following describes the audio signal processing flow when it is detected that the current frame signal contains a voice component.

[0252] S213: When the human voice detection module detects that the current frame signal contains speech components, the human voice detection module sends the current frame signal to the channel number detection module.

[0253] S214: The channel number detection module detects whether the current frame signal is two-channel stereo.

[0254] S215 , when the channel number detection module detects that the current frame signal is 5.1 / 7.1 channels, the channel number detection module sends the 5.1 / 7.1 channels to the human voice gain estimation module.

[0255] Among them, the center channel in the 5.1 / 7.1 channel is the human voice frequency domain signal; the other channels except the center channel are the background sound frequency domain signals.

[0256] S216: When the channel number detection module detects that the current frame signal is two-channel stereo, the channel number detection module sends the two-channel stereo to the human voice extraction module.

[0257] S217, the human voice extraction module extracts the human voice frequency domain signal and the background sound frequency domain signal from the current frame signal according to a preset human voice extraction algorithm.

[0258] It should be noted that, for a two-channel stereo signal, a human voice frequency domain signal and a background sound frequency domain signal can be extracted from the audio signal according to a preset human voice extraction algorithm. The preset human voice extraction algorithm is an algorithm for extracting a human voice signal from an audio signal provided in an embodiment of the present application, and the specific algorithm process will be described in detail below.

[0259] S218, the human voice extraction module sends the human voice frequency domain signal and the background sound frequency domain signal to the human voice gain estimation module.

[0260] At the same time, the environmental noise estimation module collects the noise of the environment in which the electronic device is currently located.

[0261] S219, the environmental noise estimation module collects environmental noise, and determines the noise level of the current environment according to the environmental noise.

[0262] For example, when the average amplitude of the ambient noise is in the range of [40dB, 45dB], the ambient noise level is determined to be low. When the average amplitude of the ambient noise is in the range of (45dB, 55dB], the ambient noise level is determined to be medium. When the average amplitude of the ambient noise is in the range of (55dB, 65dB], the ambient noise level is determined to be slightly high. When the average amplitude of the ambient noise is in the range of (65dB, 75dB], the ambient noise level is determined to be high.

[0263] S220: The environmental noise estimation module sends the noise level information to the human voice gain estimation module.

[0264] That is, in addition to receiving the human voice frequency domain signal and the background sound frequency domain signal, the human voice gain estimation module also receives the noise level information sent by the environmental noise estimation module.

[0265] S221, the human voice gain estimation module adjusts the gain of the human voice signal according to the noise level, the human voice frequency domain signal and the background sound frequency domain signal to obtain an enhanced human voice signal.

[0266] S222, the vocal gain estimation module sends the background sound signal and the enhanced vocal signal to the downmix module.

[0267] S223, the down-mixing module down-mixes the enhanced human voice signal and the background sound signal into a two-channel stereo signal.

[0268] S224, the two-channel stereo signal is transmitted to the speaker / headphone after frequency-time conversion.

[0269] Since the two-channel stereo signal obtained after the downmixing process is a frequency domain signal, it is necessary to perform a frequency domain to time domain conversion (frequency-time conversion). Specifically, the downmixing module sends the two-channel stereo signal to the post-processing module, and the post-processing module performs a frequency-time conversion on the two-channel stereo signal to obtain a two-channel stereo signal in the time domain, and then the post-processing module sends the two-channel stereo signal in the time domain to the speaker / headphone.

[0270] S225, the speaker / headphone outputs the two-channel stereo signal.

[0271] The signal processing method for enhancing human voice provided in the embodiment of the present application is compatible with processing multi-channel audio signals such as two-channel stereo, 5.1 channels and 7.1 channels. By dynamically adjusting the human voice gain, an audio signal with enhanced human voice is finally output. This can achieve the effect of playing the video in a quiet environment without disturbing others and being able to hear the human voice in the video, and can achieve the effect of allowing the user to hear the human voice in the video when playing the video in a noisy environment.

[0272] The above describes the overall process of the signal processing method for enhancing human voice provided in an embodiment of the present application. The following describes in stages the steps of detecting human voice signals, extracting human voice signals, detecting noise levels, performing gain adjustment on human voice signals to enhance human voices, and mixing down multi-channels into stereo in the signal processing method for enhancing human voice provided in an embodiment of the present application.

[0273] Stage 1: Detecting human voice signals

[0274] It should be noted that a frame of audio signal in the audio stream may contain only a human voice signal, may contain only a background sound signal, or may contain both a human voice signal and a background sound signal. In order to enhance the human voice signal in the audio stream, the embodiment of the present application detects whether each frame of audio signal contains a human voice signal through a human voice detection module.

[0275] In an embodiment of the present application, the human voice detection module can detect the input audio stream to determine whether the current frame in the input audio stream contains a human voice signal.

[0276] Fig.11 FIG. 1 is a flowchart showing how to detect a human voice signal in a signal processing method for enhancing human voice provided in an embodiment of the present application. Fig.11 As shown, the method includes S301-S308.

[0277] S301, obtaining an m-th frame of a multi-channel audio signal.

[0278] Here, m is a positive integer. That is, first obtain the first frame in the audio stream, then obtain the second frame in the audio stream, and so on, obtain each frame of audio signal in the audio stream one by one, and detect whether each frame of audio signal contains a human voice signal in turn.

[0279] The m-th frame multi-channel audio signal may be any one of a two-channel stereo signal, a 5.1-channel signal, and a 7.1-channel signal.

[0280] It should be noted that if the input signal is a 5.1 channel or 7.1 channel signal, the m-th frame multi-channel audio signal includes a center channel time domain signal and a side channel time domain signal, wherein the center channel time domain signal is used to carry human voice, and the side channel time domain signal is used to carry background sound. For 5.1 channels, the side channel time domain signals include the front left channel and the front right channel, the side surround left channel and the side surround right channel. For 7.1 channels, the side channel time domain signals include the front left channel and the front right channel, the side surround left channel and the side surround right channel, the rear left channel and the rear right channel.

[0281] If the input signal is a two-channel stereo signal, the m-th frame of the multi-channel audio signal includes a left-channel time domain signal and a right-channel time domain signal.

[0282] The following S302A and S302B are executed one by one.

[0283] S302A: If the input signal is a two-channel stereo signal, a human voice time domain signal and a background sound time domain signal are calculated according to the left channel time domain signal and the right channel time domain signal.

[0284] According to the following equation 1-1, the human voice time domain signal c(t) can be calculated based on the left channel time domain signal and the right channel time domain signal. According to the following equation 1-2, the background sound time domain signal s(t) can be calculated:

[0285]

[0286] s(t) = LR (Equation 1-2)

[0287] Wherein, L represents the left channel time domain signal in the two-channel stereo signal, and R represents the right channel time domain signal in the two-channel stereo signal. Wherein, β is a constant.

[0288] S302B: If the input signal is a 5.1 channel signal or a 7.1 channel signal, a human voice time domain signal is obtained according to the center channel time domain signal, and a background sound time domain signal is calculated according to the side channel time domain signal.

[0289] It should be noted that, unlike the processing method of two-channel stereo signals, for 5.1 channel signals or 7.1 channel signals, the center channel signal can be used as a human voice signal. The human voice time domain signal is subjected to short-time Fourier transform, i.e., time-frequency conversion, to obtain a human voice frequency domain signal.

[0290] For a 5.1 channel audio signal, according to the following equation 2-1, the human voice time domain signal c(t) of the 5.1 channel audio signal is equal to the center channel time domain signal C. According to the following equation 2-2, the background sound time domain signal s(t) in the 5.1 channel audio signal can be calculated:

[0291] c(t)=C (Equation 2-1)

[0292] s(t)=(L+Ls)-(R+Rs) (Equation 2-2)

[0293] Among them, C represents the center channel time domain signal, L represents the front left channel time domain signal, Ls represents the side surround left channel time domain signal, R represents the front right channel time domain signal, and Rs represents the side surround right channel time domain signal.

[0294] For a 7.1 channel audio signal, according to the following equation 3-1, the human voice time domain signal c(t) of the 5.1 channel audio signal is equal to the center channel time domain signal C. According to the following equation 3-2, the background sound time domain signal s(t) in the 5.1 channel audio signal can be obtained:

[0295] c(t)=C (Equation 3-1)

[0296] s(t)=(L+Ls+Lb)-(R+Rs+Rb) (Equation 3-2)

[0297] Among them, C represents the center channel time domain signal, L represents the front left channel time domain signal, Ls represents the side surround left channel time domain signal, Lb represents the rear left channel time domain signal, R represents the front right channel time domain signal, Rs represents the side surround right channel time domain signal, and Rb represents the rear right channel time domain signal.

[0298] S303, converting the human voice time domain signal into a human voice frequency domain signal, and converting the background sound time domain signal into a background sound frequency domain signal.

[0299] For two-channel stereo, it includes a left channel time domain signal and a right channel time domain signal. First, the human voice time domain signal and the background sound time domain signal are calculated according to the left channel time domain signal and the right channel time domain signal. Then, the human voice time domain signal is transformed into a human voice frequency domain signal through a time-frequency conversion algorithm, and the background sound time domain signal is transformed into a background sound frequency domain signal.

[0300] According to the following equation 4-1 time-frequency conversion algorithm, the human voice time domain signal is changed into a human voice frequency domain signal:

[0301] C(m,k)=STFT[c(t)](Equation 4-1)

[0302] Wherein, STFT() represents the short-time Fourier transform function. Thus, by performing short-time Fourier transform on the human voice time domain signal c(t), time-frequency conversion is performed to obtain the human voice frequency domain signal C(m,k).

[0303] At the same time, according to the time-frequency conversion algorithm of equation 4-2 below, the background sound time domain signal is changed into the background sound frequency domain signal:

[0304] S(m,k)=STFT[s(t)](Equation 4-2)

[0305] Therefore, by performing short-time Fourier transform on the background sound time domain signal s(t), time-frequency conversion is performed to obtain the background sound frequency domain signal S(m,k).

[0306] In addition, the subsequent downmixing process needs to use multi-signal frequency domain signals for downmixing. Therefore, for two-channel stereo, the left channel time domain signal and the right channel time domain signal need to be converted into the left channel frequency domain signal and the right channel frequency domain signal respectively. For 5.1 channel signals or 7.1 channel signals, the front left channel time domain signal, the front right channel time domain signal, the side surround left channel time domain signal and the side surround right channel time domain signal need to be short-time Fourier transformed to obtain the front left channel frequency domain signal, the front right channel frequency domain signal, the side surround left channel frequency domain signal and the side surround right channel frequency domain signal.

[0307] S304, calculating the spectrum flow rate of the human voice frequency domain signal (denoted as δc) and the spectrum flow rate of the background sound frequency domain signal (denoted as δs).

[0308] The specific process of calculating spectrum flow will be described in detail below.

[0309] S305 , calculating a normalized spectrum flow (denoted as V(m)) according to the spectrum flow δc of the human voice frequency domain signal and the spectrum flow δs of the background sound frequency domain signal.

[0310] In the embodiment of the present application, the spectrum flow of the human voice frequency domain signal and the spectrum flow of the background sound frequency domain signal are calculated according to the spectrum flow algorithm, and then the spectrum flow normalization algorithm is used to calculate the normalized spectrum flow value according to the spectrum flow of the human voice frequency domain signal and the spectrum flow of the background sound frequency domain signal. The specific process of calculating the normalized spectrum flow value will be described in detail below.

[0311] Then, the normalized spectrum flow value V(m) is compared with a preset human voice detection threshold (denoted as V), and whether the current frame audio signal contains a human voice signal is determined according to the comparison result.

[0312] S306, determine whether V(m) is greater than the human voice detection threshold V.

[0313] S307: When V(m) is greater than the human voice detection threshold V, determine that the m-th frame of the multi-channel audio signal contains a human voice signal.

[0314] S308: When V(m) is less than or equal to the human voice detection threshold V, determine that the m-th frame of the multi-channel audio signal does not contain a human voice signal.

[0315] The above method flow chart illustrates the process of detecting a human voice signal. The following describes the calculation process in the above process of detecting a human voice signal in conjunction with a preset human voice detection algorithm provided in an embodiment of the present application.

[0316] In the embodiment of the present application, whether the current frame audio signal contains a human voice signal may be detected based on spectral flux (delta spectrum magnitude).

[0317] Among them, the spectrum flow is defined as:

[0318]

[0319] Among them, δ X (m,k) represents the change of the signal spectrum over time, m represents the current frame audio signal is the mth frame in the audio stream, and k represents the frequency point of the current frame audio signal in the frequency domain is k. X(m,k) represents the frequency domain signal corresponding to the mth frame audio signal.

[0320] For example, m=20, k=35, and X(m, k) represents the spectrum of the 35th frequency point of the 20th frame frequency domain signal.

[0321] In order to facilitate calculation, the spectrum flow of the human voice frequency domain signal and the spectrum flow of the background sound frequency domain signal are normalized to obtain the normalized spectrum flow:

[0322]

[0323]

[0324]

[0325] Among them, δ C (m,k) is the spectrum flow of the human voice frequency domain signal, which indicates the change of the spectrum of the human voice signal over time. C(m,k) represents the spectrum of the human voice signal of the mth frame and the kth frequency point, and C(m-1,k) represents the spectrum of the human voice signal of the previous frame of the mth frame and the kth frequency point.

[0326] Among them, δ S (m,k) is the spectrum flow of the background sound frequency domain signal, which indicates the change of the spectrum of the background sound frequency domain signal over time. S(m,k) represents the spectrum of the background sound signal of the mth frame, and S(m-1,k) represents the spectrum of the background sound signal of the previous frame of the mth frame and the kth frequency point.

[0327] It should be noted that since the spectrum flow rate represents the change of the signal spectrum over time, and there are alternating changes between consonants and vowels in human voice, the spectrum flow rate δ of the human voice signal is C (m,k) is greater than the spectrum flow δ of the background sound signal S (m,k).

[0328] For a 5.1 / 7.1 channel audio signal, the human voice signal may correspond to a center channel signal, and the background sound signal may correspond to a side channel signal except the center channel signal.

[0329] The normalized spectrum flow V(m) represents the proportion of human voice signals in the audio signal. V(m) can be used as a probability value to determine whether the audio signal contains human voice signals. V(m)∈[0,1].

[0330] In the embodiment of the present application, a human voice detection threshold V can be preset. Exemplarily, the human voice detection threshold V is set to 0.5. The specific value of the human voice detection threshold V can be set according to actual use requirements, and the embodiment of the present application does not limit it.

[0331] In the embodiment of the present application, V(m) can be compared with the human voice detection threshold V, and whether the audio signal contains a human voice signal can be determined based on the comparison result.

[0332] Exemplarily, when V(m)>V, it can be considered that the current frame audio signal contains a human voice signal. When V(m)≤V, it can be considered that the current frame audio signal does not contain a human voice signal.

[0333] The above describes how to calculate the normalized spectrum flow for an input audio stream, where the input audio stream may be a two-channel stereo signal or a multi-channel signal such as 5.1 / 7.1. The following describes how to detect human voice according to the above equation when the input audio stream is a two-channel stereo signal, a 5.1 channel or a 7.1 channel audio signal.

[0334] Scenario 1: Detecting human voice in a two-channel stereo signal

[0335] For a two-channel stereo signal, each frame of the audio signal includes a left channel time domain signal and a right channel time domain signal. Specifically, the human voice frequency domain signal C(m,k) in the two-channel stereo signal is substituted into the above equation 7 to obtain the spectrum flow rate δ of the human voice signal C (m, k), substitute the background sound frequency domain signal S(m, k) in the two-channel stereo signal into the above equation 8 to obtain the spectrum flow rate δ of the background sound signal S (m,k), and then the spectrum flow δ of the human voice signal C (m,k) and the spectrum flow of the background sound signal δ S Substituting (m, k) into the above equation 6, we get the normalized spectrum flow V(m).

[0336] When V(m)>V, it can be considered that the current frame audio signal contains a human voice signal. When V(m)≤V, it can be considered that the current frame audio signal does not contain a human voice signal.

[0337] Scenario 2: Detecting human voice in 5.1 multi-channel audio signals

[0338] First of all, it should be noted that for the sake of convenience, the channels other than the center channel are called side channels, and the side channels include the front left channel, the front right channel, the side surround left channel, and the side surround right channel.

[0339] Specifically, the human voice frequency domain signal C(m,k) in the 5.1-channel signal is substituted into the above equation 7 to obtain the spectrum flow rate δ of the human voice signal: C (m, k), substitute the background sound frequency domain signal S(m, k) in the 5.1 channel signal into the above equation 8 to obtain the spectrum flow rate δ of the background sound signal S (m,k), and then the spectrum flow δ of the human voice signal C (m,k) and the spectrum flow of the background sound signal δ S Substituting (m, k) into the above equation 6, we get the normalized spectrum flow V(m).

[0340] When V(m)>V, it can be considered that the current frame audio signal contains a human voice signal. When V(m)≤V, it can be considered that the current frame audio signal does not contain a human voice signal.

[0341] Scenario 3: Detecting human voice in 7.1 multi-channel audio signals

[0342] First of all, it should be noted that for the sake of convenience, the channels other than the center channel are called side channels, and the side channels include the front left channel, the front right channel, the side surround left channel, the side surround right channel, the rear left channel, and the rear right channel.

[0343] Specifically, the human voice frequency domain signal C(m,k) in the 7.1-channel signal is substituted into the above equation 7 to obtain the spectrum flow rate δ of the human voice signal: C (m, k), substitute the background sound frequency domain signal S(m, k) in the 7.1 channel signal into the above equation 8 to obtain the spectrum flow rate δ of the background sound signal S (m,k), and then the spectrum flow δ of the human voice signal C (m,k) and the spectrum flow of the background sound signal δ S Substituting (m, k) into the above equation 6, we get the normalized spectrum flow V(m).

[0344] When V(m)>V, it can be considered that the current frame audio signal contains a human voice signal. When V(m)≤V, it can be considered that the current frame audio signal does not contain a human voice signal.

[0345] According to the following equation 9, the time-frequency parameter C(m, k) of the human voice signal of the 7.1-channel audio signal is equal to the center channel frequency domain signal Fc. According to the following equation 10, the time-frequency parameter S(m, k) of the background sound signal in the 7.1-channel audio signal can be obtained:

[0346] C(m,k)=F C (Equation 9)

[0347] S(m,k)=(F L +F Ls +F Lb )-(F R +F Rs +F Rb ) (Equation 10)

[0348] Among them, F L Represents the frequency domain signal of the front left channel, F Ls Represents the frequency domain signal of the side surround left channel, F Lb Represents the frequency domain signal of the rear left channel, F R Represents the frequency domain signal of the front right channel, F Rs Represents the frequency domain signal of the side surround right channel, F Rb Represents the frequency domain signal of the rear right channel.

[0349] Stage 2: Extracting vocal signals

[0350] In the embodiment of the present application, the human voice extraction module can extract the human voice signal based on the similarity between different channel signals (specifically, the left channel signal and the right channel signal).

[0351] First of all, it should be noted that a cross-correlation function can be used to represent the measure of similarity between two channel signals. The cross-correlation function and the similarity function are respectively expressed as:

[0352]

[0353]

[0354] In the embodiment of the present application, the above equation may be applied to calculate the similarity between signals of different channels.

[0355] Specifically, for a two-channel stereo signal, the similarity between the left channel signal and the right channel signal can be calculated using the following equation:

[0356]

[0357]

[0358]

[0359] Among them, F L (m,k) represents the frequency domain signal of the left channel, Represents the conjugate function of the left channel frequency domain signal, F R (m,k) represents the frequency domain signal of the right channel, Represents the conjugate function of the right channel frequency domain signal;

[0360] Among them, φ LR (m,k) is the frequency domain signal F of the left channel L (m,k) and the right channel frequency domain signal F R The cross-correlation function between (m,k), φ LL (m,k) is the frequency domain signal F of the left channel L The autocorrelation function of (m,k), φ RR (m,k) is the frequency domain signal F of the right channel R Autocorrelation function of (m,k);

[0361] Among them, ψ(m,k) is the similarity function between the frequency domain signal of the left channel and the frequency domain signal of the right channel.

[0362] The similarity ψ(m, k) of the two-channel stereo signal can be calculated through the above equations 11, 12 and 13.

[0363] In some implementations, the present application embodiment increases the stability of the extracted human voice frequency domain signal by adding an inter-frame smoothing coefficient (denoted as α) based on the above equation 11. According to the above equation 11, the following equation 14 can be obtained:

[0364] ψ(m,k)=(1-α)ψ(m-1,k)+αψ(m,k) (Equation 14)

[0365] Among them, the inter-frame smoothing coefficient α∈(0,1), the larger the α, the worse the smoothness, but it can reach the target value quickly.

[0366] Furthermore, after calculating the similarity ψ(m,k) of the two-channel stereo signal, the human voice frequency domain signal can be calculated by the following equation:

[0367] C(m,k)=ψ(m,k)×[F L (m,k)+F R (m,k)] / 2 (Equation 15)

[0368] Wherein, C(m,k) represents the frequency spectrum of the human voice signal extracted from the multi-channel audio signal.

[0369] That is to say, the present application scheme uses the similarity between channels to extract the human voice signal, and the signal with similarity between different channels is the human voice signal. It can be understood that the signal with zero similarity between different channels is the background sound signal.

[0370] The present application scheme utilizes the similarity between channels to extract the human voice signal, wherein the signal with the greater similarity between different channels is usually the human voice signal.

[0371] For example, Fig.12 Schematic diagram of voice detection and voice extraction is shown. Fig.12 As shown, the input audio source includes a left channel signal and a right channel signal. On the one hand, a human voice detection process is performed based on the left channel signal and the right channel signal to obtain a human voice detection result, and the human voice detection result judges the probability of the existence of the human voice by the level, that is, a high level represents the human voice, and a low level represents the background sound. On the other hand, a human voice extraction process is performed based on the left channel signal and the right channel signal to obtain a human voice extraction result, that is, a human voice signal, and the signal other than the human voice signal is a background sound signal.

[0372] Phase 3: Estimation of ambient noise levels

[0373] In an embodiment of the present application, the environmental noise estimation module periodically calls the microphone recording of the electronic device (such as a mobile phone) to collect the environmental noise signal, and then estimates the noise level of the environment in which the electronic device is currently located by analyzing the environmental noise signal.

[0374] In the embodiment of the present application, based on power consumption considerations, the microphone is called every x minutes to collect y seconds of data. Exemplarily, the microphone is called every 30 minutes to collect 10 seconds of data.

[0375] If the speaker is not working (for example, no audio or video is playing) when the microphone collects the environmental noise signal, the average amplitude of the collected data within y seconds can be directly calculated as the result of the environmental noise estimation.

[0376] If the speaker is working when the microphone collects the environmental noise signal, the noise level of the current environment is estimated according to the sensitivity of the speaker and the sensitivity of the microphone.

[0377] Here, the scenario of the speaker working when the microphone collects the environmental noise signal is described with reference to the accompanying drawings. Fig.13 The figure shows the process in which the audio signal output by the electronic device is first played by the speaker, then propagated through the environment, and then received by the microphone. Fig.13As shown in the figure, when the microphone collects the environmental noise signal, the electronic device is playing a video and playing the video sound through the speaker. The video sound will propagate through the environment and be received and recorded by the microphone. In other words, the microphone recording result includes both the environmental noise and the sound played by the speaker. Therefore, it is necessary to eliminate the signal played by the speaker and then estimate the noise level of the current environment.

[0378] The following is a detailed description of the calculation process of eliminating the collected signal played by the loudspeaker and then estimating the noise level of the current environment in the embodiment of the present application.

[0379] In some embodiments, the signal collected by the microphone includes both environmental noise and microphone recorded signal. In the embodiment of the present application, the signal collected by the microphone can be calculated by the following equation:

[0380] d(t)=y(t)+s(t) (Equation 16)

[0381] Among them, d(t) represents the signal collected by the microphone, s(t) represents the ambient noise signal, and y(t) represents the microphone recorded signal.

[0382] Fig.14 FIG. 1 is a schematic diagram showing a two-channel stereo signal x(t) played by an electronic device through a loudspeaker.

[0383] Fig.15 A schematic diagram showing a signal d(t) collected by an electronic device through a microphone.

[0384] After the two-channel stereo signal x(t) played by the electronic device is played by the speaker and recorded by the microphone, the microphone recorded signal y(t) is obtained. In the embodiment of the present application, the microphone recorded signal y(t) can be calculated by the following equation:

[0385] y(t)=x(t)*w(t) (Equation 17)

[0386] Wherein, x(t) is the audio signal played by the electronic device through the speaker, and w(t) represents the system frequency response function of the electronic device.

[0387] It should be noted that the above system frequency response function can be determined according to the speaker sensitivity and the microphone sensitivity. In the embodiment of the present application, the system frequency response function w(t) can be calculated by the following equation:

[0388] w(t)=S spk ×S mic ×H (Equation 18)

[0389] S spk Indicates speaker sensitivity, Smic It represents the microphone sensitivity, and H is the adjustment coefficient. Since the microphone and the speaker are close to each other, the adjustment coefficient H is approximately equal to 1.

[0390] Through the above equations 16, 17 and 18, the ambient noise signal s(t) can be derived as:

[0391] s(t)=d(t)-x(t)×S spk ×S mic (Equation 19)

[0392] Then the average amplitude of the ambient noise can be calculated using the following equation 20:

[0393]

[0394] Where N represents the average amplitude of the ambient noise, in dB SPL, or dB for short. T is the acquisition duration, F is the sampling frequency, and P0 is the minimum sound pressure intensity, which is 20 μPa.

[0395] In the embodiment of the present application, the environmental noise level may be determined according to the average environmental noise amplitude N. The specific determination process will be described in detail below.

[0396] Exemplarily, Table 1 shows the corresponding relationship between the average amplitude of ambient noise and the ambient noise level.

[0397] Table 1

[0398]

[0399]

[0400] When the average amplitude N of the ambient noise is within the range of [40 dB, 45 dB], it is determined that the ambient noise level is low.

[0401] When the average amplitude N of the ambient noise is within the range of (45dB, 55dB], it is determined that the ambient noise level is medium.

[0402] When the average amplitude N of the ambient noise is in the range of (55dB, 65dB], it is determined that the ambient noise level is slightly high.

[0403] When the average amplitude N of the ambient noise is within the range of (65dB, 75dB], it is determined that the ambient noise level is very high.

[0404] It should be noted that the above Table 1 is an exemplary illustration. It can be understood that in actual implementation, the correspondence between the average amplitude of ambient noise and the ambient noise level is not limited to this. Of course, the correspondence between the average amplitude of ambient noise and the ambient noise level can also be set according to actual usage requirements. The embodiment of the present application does not limit this.

[0405] Stage 4: Enhance the human voice signal

[0406] In an embodiment of the present application, the vocal gain estimation module performs gain adjustment on the vocal signal C(m, k) obtained by the vocal extraction algorithm in the second stage according to the ambient noise level determined in the third stage, thereby achieving the effect of dynamically enhancing the vocals.

[0407] First of all, it should be noted that the traditional human voice enhancement algorithm uses the Wiener filter algorithm to perform human voice gain. The calculation equation for adjusting the human voice gain through the Wiener filter algorithm can be expressed as:

[0408]

[0409] SNR(m,k)=C(m,k) / S(m,k) (Equation 22)

[0410] Among them, G(m,k) represents the adjusted vocal gain, SNR(m,k) represents the signal-to-noise ratio, C(m,k) represents the spectrum of the m-th frame vocal signal, and S(m,k) represents the spectrum of the m-th frame background sound signal.

[0411] For the sake of convenience, some vocal signals with smaller signal spectrum intensity in the multi-channel audio signal are referred to as small signals. Compared with small signals, some vocal signals with larger signal spectrum intensity in the multi-channel audio signal are referred to as large signals.

[0412] It should be noted that the traditional Wiener filtering algorithm significantly improves the signal gain of a signal with a large signal spectrum intensity, but the signal gain is not obvious for a signal with a small signal spectrum intensity. It can be understood that in this case, the spectrum range of the human voice signal is relatively large; wherein the spectrum range of the human voice signal refers to the spectrum range composed of small signals and large signals.

[0413] In other words, through the traditional Wiener filtering algorithm, the large signal gain is obvious, and the small signal gain is too small, which will cause the final listening experience to fluctuate.

[0414] In order to solve the problem of the fluctuating sound perception, an improved method is proposed in the embodiment of the present application to dynamically adjust the vocal gain through the following equation.

[0415] G′(m,k)=p(1-G(m,k))+qG(m,k) (Equation 23)

[0416] Among them, G(m,k) represents the vocal gain after preliminary adjustment by the Wiener filter algorithm, and G′(m,k) represents the vocal gain after further adjustment based on the preliminary adjusted vocal gain. p represents the weight coefficient of the small signal, and q represents the weight coefficient of the large signal.

[0417] It can be understood that the human voice gain estimation formula (Equation 23) adopted in the present application is an improved formula based on the Wiener filter formula (Equation 21). Through the human voice gain estimation formula adopted in the present application, the gains of human voice signals of different intensities within the human voice signal spectrum can be adjusted.

[0418] Through the embodiment scheme of the present application, the values ​​of the two coefficients p and q can be dynamically adjusted according to the current environmental noise level, so that the gain of the large signal can be adjusted, the gain of the small signal can be adjusted, or the gain of the large signal and the gain of the small signal can be adjusted at the same time, which can avoid the phenomenon of the hearing feeling being louder and softer. By dynamically adjusting the vocal gain, the effect of playing the video in a quiet environment without disturbing others and being able to hear the human voice in the video can be achieved, and the effect of the user being able to hear the human voice in the video when playing the video in a noisy environment can be achieved.

[0419] For example, through the solution of the present application, in a quiet environment, the values ​​of the two coefficients p and q are dynamically adjusted, and the gain of the large signal is appropriately increased while the gain of the small signal is appropriately increased. Even if the volume is low when the electronic device plays a video, it can ensure that the user can clearly hear both the strong voice dialogue in the video and the weak voice dialogue in the video. Thus, when playing a video in a quiet environment, the user can hear the human voice in the video clearly without disturbing others.

[0420] As another example, through the solution of the present application, in a noisy environment, the values ​​of the two coefficients p and q are dynamically adjusted. While appropriately increasing the gain of large signals, the gain of small signals is also appropriately increased, which can ensure that the user can clearly hear both the louder vocal dialogues in the video and the weaker vocal dialogues in the video, thereby achieving the effect that the user can clearly hear the human voice in the video when playing the video in a noisy environment.

[0421] Compared with the related art of only increasing the gain of large signals, the present application achieves the effect of compressing the dynamic range of the human voice signal by appropriately increasing the gain of small signals while increasing the gain of large signals, thereby avoiding the phenomenon of the sound being louder or smaller.

[0422] The following describes in detail the possible specific values ​​of p and q.

[0423] In some embodiments, the values ​​of p and q may be determined according to the average amplitude N of the ambient noise based on a linear mapping relationship between p, q and the average amplitude of the ambient noise.

[0424] For example, Fig.16 A schematic diagram showing the linear mapping relationship between p, q and the average amplitude of the ambient noise is shown. Fig.16 As shown, the horizontal axis is the average amplitude of the noise level N, in dB SPL, and the vertical axis is the value of p / q. The thick solid line represents p, and the thin solid line represents q. From the linear mapping relationship in the figure, it can be seen that within the range of the average amplitude of the noise level [40, 75), the values ​​of p and q.

[0425] (1) The value of p in the range of the average amplitude of the noise level [40, 75].

[0426] When 40≤N<65, p=0.02×N-0.8.

[0427] When 65≤N<75, p=0.15×N-9.25.

[0428] (2) The value selection rule of q within the range of the average amplitude of the noise level [40, 75).

[0429] When 40≤N<65, q=1.

[0430] When 65≤N<75, q=-0.04×N+3.6.

[0431] In the embodiment of the present application, the specific values ​​of p and q can be determined according to actual use requirements. For example, the value range of p can be set to [0, 2], and the value range of q can be set to [0.25, 1]. Among them, the specific values ​​of p and q can be determined according to actual use requirements, and the embodiment of the present application does not limit them.

[0432] The following example illustrates possible implementations of how to dynamically adjust the two coefficients p and q at different environmental noise levels.

[0433] For example, Table 2 shows the values ​​of p and q and the strategy for adjusting the gain of the human voice signal at different environmental noise levels.

[0434] Table 2

[0435]

[0436]

[0437] With reference to the above Table 2, it can be seen that the solution of the present application takes into account the influence of environmental noise and can adjust the gain control in different environments.

[0438] Scenario 1: When the ambient noise level is low (e.g., 40dB-45dB), the parameters p=0, q=1 are used. At this time, the above equation 23 changes to:

[0439] G′(m,k)=G(m,k)

[0440] Correspondingly, the strategy for adjusting the gain of the vocal signal is to appropriately gain the large signal and retain the dynamic range of the vocal signal.

[0441] Scenario 2: When the ambient noise level is medium (e.g., 45dB-55dB), the parameters p=0.25 and q=1 are used. At this time, the above equation 23 changes to:

[0442] G'(m,k)=0.25(1-G(m,k))+G(m,k)

[0443] Correspondingly, the strategy for adjusting the gain of the vocal signal is: increase the small signal gain and slightly compress the signal dynamic range.

[0444] Scenario 3: When the ambient noise level is slightly higher (e.g., 55dB-65dB), the parameters p=0.5 and q=1 are used. At this time, the above equation 23 changes to:

[0445] G'(m,k)=0.5(1-G(m,k))+G(m,k)

[0446] Correspondingly, the strategy for adjusting the gain of the vocal signal is to further increase the small signal gain and compress the dynamic range.

[0447] Scenario 4: When the ambient noise is very high (e.g., 65dB-75dB), the parameters p=2, q=0.6 are used. At this time, the above equation 23 changes to:

[0448] G'(m,k)=2(1-G(m,k))+0.6G(m,k)

[0449] Correspondingly, the strategy for adjusting the gain of the vocal signal is: the signal gain is greatly increased while the signal dynamic range is greatly compressed.

[0450] The present application scheme takes environmental noise factors into consideration, detects the environmental noise level, and automatically adjusts the vocal gain range according to the environmental noise level, thereby avoiding the phenomenon of the sound being louder or softer.

[0451] It should be noted that the above Table 2 is an exemplary illustration. It can be understood that in actual implementation, the values ​​of p and q and the strategy for adjusting the gain of the human voice signal at different ambient noise levels are not limited to this. Of course, the values ​​of p and q and the strategy for adjusting the gain of the human voice signal at different ambient noise levels can also be set according to actual usage requirements. The embodiments of the present application are not limited to this.

[0452] In the embodiment of the present application, the vocal gain estimation module can perform vocal enhancement on the vocal signal according to the vocal enhancement algorithm. Specifically, the vocal enhancement can be performed on the vocal signal using the following equation:

[0453] C′(m,k)=C(m,k)+C(m,k)*G′(m,k)*V(m) (Equation 24)

[0454] Among them, C′(m,k) represents the enhanced human voice signal spectrum, V(m) is the normalized spectrum flow value obtained by the human voice detection algorithm in the first stage, C(m,k) is the human voice signal spectrum obtained by the human voice extraction algorithm in the second stage, and G′(m,k) is the human voice gain obtained by the human voice gain estimation algorithm in the fourth stage.

[0455] Stage 5: Downmixing of multi-channel signals

[0456] In the embodiment of the present application, the gained vocal signal C′(m, k) and the multi-channel signal may be down-mixed into a two-channel stereo signal through a down-mixing module.

[0457] For example, when the input signal is a two-channel stereo signal, the downmixing module downmixes the enhanced vocal signal with the left channel signal to obtain a left channel frequency domain signal; and downmixes the enhanced vocal signal with the right channel signal to obtain a right channel frequency domain signal.

[0458] For another example, when the input signal is a 5.1-channel / 7.1-channel stereo signal, the downmixing module downmixes the enhanced vocal signal with other channel signals except the center channel to obtain a left channel frequency domain signal and a right channel frequency domain signal.

[0459] After the downmixing process is completed, the left channel frequency domain signal and the right channel frequency domain signal are frequency-time converted through the post-processing module to obtain the left channel time domain signal and the right channel time domain signal, and then the left channel time domain signal and the right channel time domain signal are output through speakers or headphones.

[0460] The down-mixing process and the frequency-time conversion process are described in detail below for different input signals.

[0461] (1) The input signal is a two-channel stereo signal.

[0462] Through the algorithm provided in the above embodiment of the present application, after the input two-channel stereo signal is processed with human voice enhancement, the enhanced human voice signal is down-mixed with the left channel signal and the right channel signal to obtain the left channel frequency domain signal and the right channel frequency domain signal:

[0463] F′ L =FL +C′(m,k) (Equation 25)

[0464] F′ R =F R +C′(m,k) (Equation 26)

[0465] Among them, F′ L Represents the output left channel frequency domain signal, F′ R Represents the output right channel frequency domain signal.

[0466] Among them, F L Represents the input left channel frequency domain signal, F R Represents the input right channel frequency domain signal.

[0467] Wherein, C′(m,k) represents the enhanced human voice signal spectrum obtained by the human voice enhancement algorithm in the fourth stage.

[0468] After the downmixing process is completed, the left channel frequency domain signal F′ is converted into L Perform frequency-time conversion to obtain the left channel time domain signal; similarly, the right channel frequency domain signal F′ is converted through the post-processing module R Perform frequency-time conversion to obtain the right channel time domain signal.

[0469] Then, the post-processing module transmits the converted left channel time domain signal and right channel time domain signal to the speaker or earphone, and the speaker or earphone outputs the left channel time domain signal and the right channel time domain signal, that is, outputs a two-channel stereo signal.

[0470] (2) The input signal is a 5.1-channel audio signal.

[0471] Through the algorithm provided in the above embodiment of the present application, after the input 5.1 channel audio signal is subjected to vocal enhancement processing, the enhanced vocal signal is down-mixed with other channel signals except the center channel to obtain a left channel frequency domain signal and a right channel frequency domain signal:

[0472] F′ L =0.4143*(F L +0.707C′(m,k)+0.707F Ls ) (Equation 27)

[0473] F′ R =0.4143*(F R +0.707C′(m,k)+0.707F Rs ) (Equation 28)

[0474] Among them, F′ L Represents the output left channel frequency domain signal, F′R Represents the output right channel frequency domain signal.

[0475] Among them, F L Represents the input front left channel frequency domain signal, F R Represents the input front right channel frequency domain signal, F Ls Represents the input side surround left channel frequency domain signal, F Rs Represents the input side surround right channel frequency domain signal.

[0476] Wherein, C′(m,k) represents the enhanced human voice signal spectrum obtained by the human voice enhancement algorithm in the fourth stage.

[0477] After the downmixing process is completed, the left channel frequency domain signal F′ is converted into L Perform frequency-time conversion to obtain the left channel time domain signal; similarly, the right channel frequency domain signal F′ is converted through the post-processing module R Perform frequency-time conversion to obtain the right channel time domain signal.

[0478] Then, the post-processing module transmits the converted left channel time domain signal and right channel time domain signal to the speaker or earphone, and the speaker or earphone outputs the signal.

[0479] (3) The input signal is a 7.1-channel audio signal.

[0480] Through the algorithm provided in the above embodiment of the present application, after the input 7.1-channel audio signal is processed for human voice enhancement, the following two-channel stereo signal is output through the speaker or earphone:

[0481] F′ L =0.3204*(F L +0.707C′(m,k)+0.707F Ls +0.707F Lb ) (Equation 29)

[0482] F′ R =0.3204*(F R +0.707C′(m,k)+0.707F Rs +0.707F Rb ) (Equation 30)

[0483] Among them, F′ L Represents the output left channel frequency domain signal, F′ R Represents the output right channel frequency domain signal.

[0484] Among them, F L Represents the input front left channel frequency domain signal, F R Represents the input front right channel frequency domain signal, FLs Represents the input side surround left channel frequency domain signal, F Rs Represents the input side surround right channel frequency domain signal, F Lb Represents the input rear left channel frequency domain signal, F Rb Represents the input rear right channel frequency domain signal.

[0485] Wherein, C′(m,k) represents the enhanced human voice signal spectrum obtained by the human voice enhancement algorithm in the fourth stage.

[0486] Similarly, after the downmixing process is completed, the left channel frequency domain signal F′ is converted into L , F′ R Perform frequency-time conversion to obtain left channel time domain signals and right channel time domain signals.

[0487] Then, the post-processing module transmits the converted left channel time domain signal and right channel time domain signal to the speaker or earphone, and the speaker or earphone outputs the signal.

[0488] The signal processing method for enhancing human voice provided in the embodiment of the present application can be applied to audio output scenarios where the input signal is a multi-channel audio signal. It can adjust the human voice gain for signals of different intensities within the dynamic range of the human voice signal according to the ambient noise level. By adjusting the human voice gain, the effect of enhancing the human voice is achieved, and the output sound is kept stable, thereby avoiding the phenomenon of fluctuating in hearing.

[0489] In a noisy environment, the present application solution can adopt different gain methods for human voice according to the environmental noise level. In a noisy environment, the algorithm automatically selects the human voice gain parameter, which can resist noise interference, so that users can hear the human voice dialogue in movies and TV series clearly.

[0490] Similarly, in a quiet environment, when a user is watching a TV program at night and does not want to disturb someone who has fallen asleep, by using the present application scheme, it can be ensured that the user can clearly hear the human voice dialogue in the TV series without significantly increasing the volume.

[0491] The signal processing method for enhancing human voice provided in the embodiment of the present application can be applied to the scenario where the electronic device outputs audio signals through a speaker, and can also be applied to the scenario where the electronic device outputs audio signals through headphones. For example, in a noisy environment, when a user wears headphones and watches a video played by a mobile phone, after the input multi-channel audio signal is processed for human voice enhancement by the algorithm provided in the present application, the electronic device outputs the audio signal with human voice enhancement through the headphones. Even in a noisy environment, the user can clearly hear the strong and weak human voice dialogues in the video through the headphones, thereby achieving the effect that the user can clearly hear the human voice in the video when the video is played in a noisy environment.

[0492] It should be noted that, in the embodiments of the present application, "greater than" can be replaced by "greater than or equal to", "less than or equal to" can be replaced by "less than", or "greater than or equal to" can be replaced by "greater than", and "less than" can be replaced by "less than or equal to".

[0493] The various embodiments described herein may be independent solutions or may be combined according to internal logic, and all of these solutions fall within the protection scope of this application.

[0494] The above mainly describes the solution provided by the embodiment of the present application from the perspective of method steps. It is understandable that, in order to achieve the above functions, the electronic device implementing the method includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should be aware that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of protection of the present application.

[0495] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other feasible division methods in actual implementation. The following is an example of dividing each functional module corresponding to each function.

[0496] Fig.17 The present invention is a schematic block diagram of a signal processing device 400 for enhancing human voice provided in an embodiment of the present application. The device 400 can be used to perform the actions performed by the electronic device in the above method embodiment. The device 400 includes an acquisition unit 410, a processing unit 420 and an output unit 430.

[0497] The receiving unit 410 is configured to obtain the mth frame audio signal in the multi-channel audio stream. For example, the receiving unit 410 is a microphone.

[0498] The processing unit 420 is used to calculate the normalized spectrum flow corresponding to the m-th frame audio signal; obtain the human voice frequency domain signal and the background sound frequency domain signal from the m-th frame audio signal; determine the human voice gain parameter according to the environmental noise, the background sound frequency domain signal and the human voice frequency domain signal; adjust the gain of the human voice frequency domain signal according to the human voice gain parameter and the normalized spectrum flow to obtain an enhanced human voice frequency domain signal; and then convert the enhanced human voice frequency domain signal into an enhanced human voice time domain signal.

[0499] The output unit 430 is used to output the enhanced human voice time domain signal. For example, the output unit 430 is a speaker or earphone.

[0500] In some possible implementations, the processing unit 420 is specifically configured to:

[0501] Determining a first gain adjustment coefficient and a second gain adjustment coefficient according to an average amplitude of the ambient noise;

[0502] The following equation is used to calculate the vocal gain parameter according to the first gain adjustment coefficient, the second gain adjustment coefficient, the background sound frequency domain signal and the vocal frequency domain signal:

[0503] G'(m,k)=p(1-G(m,k))+qG(m,k)

[0504]

[0505] SNR(m,k)=C(m,k) / S(m,k)

[0506] Among them, G′(m,k) represents the human voice gain parameter, p represents the first gain adjustment coefficient, q represents the second gain adjustment coefficient, C(m,k) represents the human voice frequency domain signal, and S(m,k) represents the background sound frequency domain signal.

[0507] In some possible implementations, the processing unit 420 is specifically configured to:

[0508] The enhanced human voice signal spectrum is calculated using the following equation based on the human voice gain parameter and the normalized spectrum flow:

[0509] C′(m,k)=C(m,k)+C(m,k)*G′(m,k)*V(m)

[0510] Among them, C′(m,k) represents the enhanced human voice frequency domain signal, C(m,k) represents the human voice frequency domain signal, G′(m,k) represents the human voice gain parameter, and V(m) represents the normalized spectrum flow corresponding to the m-th frame audio signal.

[0511] Through the signal processing device for enhancing human voice provided by the present application, in the scenario where the electronic device plays audio and video, the human voice gain parameter can be determined according to the environmental noise and the background sound frequency domain signal and the human voice frequency domain signal corresponding to the multi-channel audio stream, and then the gain of the human voice frequency domain signal is adjusted according to the human voice gain parameter and the normalized spectrum flow rate to obtain the enhanced human voice frequency domain signal. Since the gain of the human voice signal can be dynamically adjusted for the input multi-channel audio signal under different environmental noise conditions to achieve the effect of enhancing the human voice and avoid the phenomenon of the sound being louder and softer, the present application scheme can achieve the effect of playing the video in a quiet environment without disturbing others and being able to hear the human voice in the video, or the effect of the user being able to hear the human voice in the video when playing the video in a noisy environment, which can improve the user experience.

[0512] According to the embodiment of the present application, the device 400 may correspond to executing the method described in the embodiment of the present application, and the above and other operations and / or functions of the units in the device 400 are respectively for implementing the corresponding processes of the method, which will not be repeated here for the sake of brevity.

[0513] The present application also provides a chip, which is coupled to a memory and is used to read and execute computer programs or instructions stored in the memory to execute the methods in the above embodiments.

[0514] The present application also provides an electronic device, which includes a chip, and the chip is used to read and execute a computer program or instruction stored in a memory, so that the method in each embodiment is executed.

[0515] This embodiment also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the signal processing method for enhancing human voice in the above-mentioned embodiment.

[0516] This embodiment further provides a computer program product, wherein the computer-readable storage medium stores program code. When the computer program product is run on a computer, the computer executes the above-mentioned related steps to implement the signal processing method for enhancing human voice in the above-mentioned embodiment.

[0517] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, a component or a module, and the device may include a connected processor and a memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the signal processing method for enhancing human voice in the above-mentioned method embodiments.

[0518] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0519] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0520] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A signal processing method for enhancing human voice, characterized in that: include: Get the mth frame audio signal in the multi-channel audio stream; Calculating the normalized spectrum flow corresponding to the m-th frame of audio signal; Acquire a human voice frequency domain signal and a background sound frequency domain signal from the mth frame audio signal; Determining a human voice gain parameter according to the environmental noise, the background sound frequency domain signal and the human voice frequency domain signal; The gain of the human voice frequency domain signal is adjusted according to the human voice gain parameter and the normalized spectrum flow to obtain an enhanced human voice frequency domain signal.

2. The method according to claim 1, characterized in that The step of determining a human voice gain parameter according to the environmental noise, the background sound frequency domain signal and the human voice frequency domain signal comprises: Determining a first gain adjustment coefficient and a second gain adjustment coefficient according to the average amplitude of the environmental noise; The human voice gain parameter is calculated according to the first gain adjustment coefficient, the second gain adjustment coefficient, the background sound frequency domain signal, and the human voice frequency domain signal using the following equation: G'(m,k)=p(1-G(m,k))+qG(m,k) SNR(m,k)=C(m,k) / S(m,k) Among them, G′(m,k) represents the human voice gain parameter, p represents the first gain adjustment coefficient, q represents the second gain adjustment coefficient, C(m,k) represents the human voice frequency domain signal, and S(m,k) represents the background sound frequency domain signal.

3. The method according to claim 2, characterized in that The determining the first gain adjustment coefficient and the second gain adjustment coefficient according to the average amplitude of the environmental noise includes: When N∈[a,b), p=0.02×N-0.8, q=1; When N∈[b,c), p=0.15×N-9.25, q=-0.04×N+3.6; Wherein, N represents the average amplitude of the environmental noise, in decibels (dB), a is smaller than b, and b is smaller than c.

4. The method according to claim 3, characterized in that a is 40dB, b is 65dB, and c is 75dB.

5. The method according to any one of claims 1 to 4, characterized in that The method is applied to an electronic device, wherein the electronic device includes a microphone; the method further includes: Determining an environmental noise signal according to the audio signal collected by the microphone; The average amplitude of the environmental noise is determined according to the environmental noise signal, sampling duration, sampling frequency and minimum sound pressure intensity.

6. The method according to claim 5, characterized in that The electronic device further includes a speaker, and determining the ambient noise signal according to the audio signal collected by the microphone includes: When the speaker is working, determining the environmental noise signal according to the audio signal collected by the microphone and the audio signal played by the speaker; When the speaker is not working, the audio signal collected by the microphone is determined as the environmental noise signal.

7. The method according to claim 6, characterized in that The step of determining the ambient noise signal according to the audio signal collected by the microphone and the audio signal played by the speaker includes: The ambient noise signal is calculated according to the audio signal collected by the microphone and the audio signal played by the speaker using the following equation: s(t)=d(t)-x(t)×S spk ×S mic Wherein, s(t) represents the environmental noise signal, d(t) represents the audio signal collected by the microphone, x(t) represents the audio signal played by the speaker, S spk represents the sensitivity of the speaker, S mic Indicates the sensitivity of the microphone.

8. The method according to any one of claims 5 to 7, characterized in that The determining the average amplitude of the ambient noise according to the ambient noise signal, the sampling duration, the sampling frequency, and the minimum sound pressure intensity includes: The following equation is used to calculate the average amplitude of the ambient noise according to the ambient noise signal, sampling duration, sampling frequency and minimum sound pressure intensity: Wherein, N represents the average amplitude of the ambient noise, s(t) represents the ambient noise signal, T represents the acquisition time, F represents the sampling frequency, and P0 represents the minimum sound pressure intensity.

9. The method according to any one of claims 1 to 8, characterized in that The step of performing gain adjustment on the human voice frequency domain signal according to the human voice gain parameter and the normalized spectrum flow to obtain an enhanced human voice frequency domain signal includes: The enhanced human voice signal spectrum is calculated according to the human voice gain parameter and the normalized spectrum flow using the following equation: C′(m,k)=C(m,k)+C(m,k)*G′(m,k)*V(m) Among them, C′(m,k) represents the enhanced human voice frequency domain signal, C(m,k) represents the human voice frequency domain signal, G′(m,k) represents the human voice gain parameter, and V(m) represents the normalized spectrum flow corresponding to the m-th frame audio signal.

10. The method according to any one of claims 1 to 9, characterized in that The acquiring of a human voice frequency domain signal from the m-th frame audio signal comprises: When the m-th frame audio signal is a two-channel stereo signal, the human voice frequency domain signal is obtained from the m-th frame audio signal using the following equation: C(m,k)=ψ(m,k)×[F L (m,k)+F R (m,k)] / 2 F L (m,k)=STFT(L) F R (m,k)=STFT(R) Wherein, C(m,k) represents the human voice frequency domain signal, L represents the left channel time domain signal, and F L (m,k) represents the frequency domain signal of the left channel, represents the conjugate function of the left channel frequency domain signal, R represents the right channel time domain signal, F R (m,k) represents the frequency domain signal of the right channel, represents the conjugate function of the right channel frequency domain signal; wherein STFT() represents the short time Fourier transform function; Among them, φ LR (m, k) represents the cross-correlation function between the left channel frequency domain signal and the right channel frequency domain signal, φ LL (m, k) represents the autocorrelation function of the left channel frequency domain signal, φ RR (m, k) represents the autocorrelation function of the right channel frequency domain signal; Wherein, ψ(m,k) represents the similarity function between the frequency domain signal of the left channel and the frequency domain signal of the right channel of the mth frame.

11. The method according to claim 10, characterized in that The method further comprises: The optimized ψ(m,k) is obtained using the following equation: ψ(m,k)=(1-α)ψ(m-1,k)+αψ(m,k) Among them, α represents the inter-frame smoothing coefficient, α∈(0,1); Among them, ψ(m,k) represents the similarity function between the left channel frequency domain signal and the right channel frequency domain signal of the m-th frame, and ψ(m-1,k) represents the similarity function between the left channel frequency domain signal and the right channel frequency domain signal of the m-1-th frame.

12. The method according to any one of claims 1 to 9, characterized in that The step of obtaining a background audio frequency domain signal from the m-th frame audio signal comprises: When the m-th frame audio signal is a two-channel stereo signal, the background audio frequency domain signal is obtained from the m-th frame audio signal using the following equation: s(t)=LR S(m,k)=STFT[s(t)] Wherein, L and R represent the left channel time domain signal and the right channel time domain signal in the m-th frame audio signal respectively, and s(t) and S(m,k) represent the background sound time domain signal and the background sound frequency domain signal corresponding to the m-th frame audio signal respectively.

13. The method according to any one of claims 1 to 9, characterized in that The acquiring of a human voice frequency domain signal from the m-th frame audio signal comprises: When the m-th frame audio signal is a 5.1-channel or 7.1-channel stereo signal, a short-time Fourier transform is performed on a center channel time domain signal in the m-th frame audio signal to obtain the human voice frequency domain signal.

14. The method according to any one of claims 1 to 9, characterized in that The step of obtaining a background audio frequency domain signal from the m-th frame audio signal comprises: When the m-th frame audio signal is a 5.1-channel stereo signal, the background audio frequency domain signal is obtained from the m-th frame audio signal using the following equation: s(t)=(L+Ls)-(R+Rs) S(m,k)=STFT[s(t)] Wherein, L, Ls, R, and Rs respectively represent the front left channel time domain signal, the side surround left channel time domain signal, the front right channel time domain signal, and the side surround right channel time domain signal in the m-th frame audio signal; s(t) and S(m,k) respectively represent the background sound time domain signal and the background sound frequency domain signal corresponding to the m-th frame audio signal; When the m-th frame audio signal is a 7.1-channel stereo signal, the background audio frequency domain signal is obtained from the m-th frame audio signal using the following equation: s(t)=(L+Ls+Lb)-(R+Rs+Rb) S(m,k)=STFT[s(t)] Among them, L, Ls, Lb, R, Rs, and Rb respectively represent the front left channel time domain signal, side surround left channel time domain signal, rear left channel time domain signal, front right channel time domain signal, side surround right channel time domain signal, and rear right channel time domain signal in the m-th frame audio signal; s(t) and S(m,k) respectively represent the background sound time domain signal and background sound frequency domain signal corresponding to the m-th frame audio signal.

15. The method according to any one of claims 1 to 14, characterized in that The calculating the normalized spectrum flow corresponding to the m-th frame audio signal includes: The normalized spectrum flow V(m) corresponding to the m-th frame audio signal is calculated using the following equation: C(m,k)=STFT[c(t)] S(m,k)=STFT[s(t)] Wherein, STFT() represents the short-time Fourier transform function; Wherein, c(t) and C(m,k) represent the human voice time domain signal and human voice frequency domain signal corresponding to the m-th frame audio signal, respectively, C(m-1,k) represents the human voice frequency domain signal corresponding to the m-1-th frame audio signal, δ C (m, k) represents the spectrum flow of the human voice frequency domain signal; Wherein, s(t) and S(m,k) represent the background sound time domain signal and background sound frequency domain signal corresponding to the m-th frame audio signal, respectively, S(m-1,k) represents the background sound frequency domain signal corresponding to the m-1-th frame audio signal, δ S (m, k) represents the spectrum flow of the background audio frequency domain signal.

16. The method according to claim 15, characterized in that The method further comprises: When the m-th frame audio signal is a two-channel stereo signal, c(t) and s(t) are calculated according to the following equations: s(t)=LR Wherein, L and R represent the left channel time domain signal and the right channel time domain signal in the m-th frame audio signal respectively, and β is a constant; When the m-th frame audio signal is a 5.1-channel signal, c(t) and s(t) are calculated according to the following equations: s(t)=(L+Ls)-(R+Rs) Wherein, C, L, Ls, R, and Rs respectively represent the center channel time domain signal, the front left channel time domain signal, the side surround left channel time domain signal, the front right channel time domain signal, and the side surround right channel time domain signal in the m-th frame 5.1 channel signal; When the m-th frame audio signal is a 7.1-channel signal, c(t) and s(t) are calculated according to the following equations: c(t)=C s(t)=(L+Ls+Lb)-(R+Rs+Rb) Among them, C, L, Ls, Lb, R, Rs, and Rb respectively represent the center channel time domain signal, the front left channel time domain signal, the side surround left channel time domain signal, the rear left channel time domain signal, the front right channel time domain signal, the side surround right channel time domain signal, and the rear right channel time domain signal in the 7.1 channel signal of the mth frame.

17. The method according to any one of claims 1 to 16, characterized in that After the gain of the human voice frequency domain signal is adjusted according to the human voice gain parameter and the normalized spectrum flow to obtain an enhanced human voice frequency domain signal, the method further includes: Downmixing is performed on the enhanced human voice frequency domain signal to obtain a human voice enhanced stereo frequency domain signal.

18. The method according to claim 17, characterized in that Downmixing is performed according to the enhanced human voice frequency domain signal to obtain a human voice enhanced stereo frequency domain signal, including: When the m-th frame audio signal is a two-channel stereo signal, downmixing processing is performed on the enhanced human voice frequency domain signal according to the following equation to obtain a human voice enhanced stereo frequency domain signal: F′ L =F L +C′(m,k) F′ R =F R +C′(m,k) Among them, F′ L and F′ R Respectively represent the stereo frequency domain signals of the human voice enhancement; Among them, C′(m,k) represents the enhanced human voice frequency domain signal, F L and F R Respectively represent the frequency domain signals corresponding to the left / right channel time domain signals in the m-th frame audio signal.

19. The method according to claim 17, characterized in that Downmixing is performed according to the enhanced human voice frequency domain signal to obtain a human voice enhanced stereo frequency domain signal, including: When the m-th frame audio signal is a 5.1-channel stereo signal, downmixing is performed on the enhanced human voice frequency domain signal according to the following equation to obtain a human voice enhanced stereo frequency domain signal: F′ L =0.4143*(F L +0.707C′(m,k)+0.707F Ls ) F′ R =0.4143*(F R +0.707C′(m,k)+0.707F Rs ) Among them, F′ L and F′ R Respectively represent the stereo frequency domain signals of the human voice enhancement; Among them, C′(m,k) represents the enhanced human voice frequency domain signal, F L 、F R 、F Ls 、F Rs They respectively represent the frequency domain signals corresponding to the time domain signals of the left channel, the right channel, the side surround left channel and the side surround right channel in the m-th frame 5.1 channel signal.

20. The method according to claim 17, characterized in that The downmixing process is performed according to the enhanced human voice frequency domain signal to obtain a human voice enhanced stereo frequency domain signal, including: When the m-th frame audio signal is a 7.1-channel stereo signal, downmixing is performed on the enhanced human voice frequency domain signal according to the following equation to obtain a human voice enhanced stereo frequency domain signal: F′ L =0.3204*(F L +0.707C′(m,k)+0.707F Ls +0.707F Lb ) F′ R =0.3204*(F R +0.707C′(m,k)+0.707F Rs +0.707F Rb ) Among them, F′ L and F′ R Respectively represent the stereo frequency domain signals of the human voice enhancement; Among them, C′(m,k) represents the enhanced human voice frequency domain signal, F L 、F R 、F Ls 、F Rs 、F Lb 、F Rb They respectively represent the frequency domain signals corresponding to the time domain signals of the left channel, right channel, side surround left channel, side surround right channel, rear left channel and rear right channel in the mth frame 7.1 channel signal.

21. The method according to any one of claims 17 to 20, characterized in that After performing downmixing processing according to the enhanced human voice frequency domain signal to obtain a human voice enhanced stereo frequency domain signal, the method further includes: Converting the human voice enhanced stereo frequency domain signal into a human voice enhanced stereo time domain signal by Fourier transform algorithm; Outputting the human voice enhanced stereo time domain signal through a sound output device; Wherein, the sound output device is a speaker or earphone.

22. The method according to any one of claims 1 to 21, characterized in that Before acquiring the human voice frequency domain signal and the background sound frequency domain signal from the m-th frame audio signal, the method further includes: Determining whether the normalized spectrum flow is greater than a first preset threshold; When the normalized spectrum flow rate is greater than the first preset threshold, it is determined that the m-th frame audio signal contains a speech component.

23. The method according to claim 22, characterized in that After determining whether the normalized spectrum flow is greater than a first preset threshold, the method further includes: When the normalized spectrum flow is less than or equal to the first preset threshold, determining that the m-th frame audio signal does not contain a speech component; When the m-th frame audio signal is a stereo signal, outputting the m-th frame stereo time domain signal without human voice enhancement through a sound output device; In the case where the m-th frame audio signal is a 5.1 / 7.1 channel signal, downmixing the m-th frame 5.1 / 7.1 channel signal to obtain an m-th frame stereo frequency domain signal without human voice enhancement; converting the stereo frequency domain signal without human voice enhancement into an m-th frame stereo time domain signal without human voice enhancement, and outputting the m-th frame stereo time domain signal without human voice enhancement through a sound output device; Wherein, the sound output device is a speaker or earphone.

24. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory, wherein the processor is configured to execute the computer program so that the electronic device implements the method according to any one of claims 1 to 23.

25. A chip comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that: The processor is configured to execute the computer program to implement the method according to any one of claims 1 to 23.

26. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 23.

Citation Information

Patent Citations

  • Speech processing device and speech processing method

    CN112133320A

  • Audio signal processing method and device

    CN114203163A

  • Multi-sound-source sound signal processing method and device, storage medium and electronic equipment

    CN114627889A

  • Speech enhancement method and device, computer equipment and storage medium

    CN114783453A

  • Automatic volume and dynamic range adjustment for mobile audio devices

    US20080269926A1