A digital audio mixing method for multi-person voice conference scenarios

Abstract

The application discloses a digital audio mixing method for a multi-person voice conference scene, comprising the following steps: S1, in a multi-person voice conference scene, assigning a participant weight M[k], a voice coefficient Vp and a decay factor f[k] and other parameters to the audio of each participant; S2, converting the digital audio from a time domain signal to a frequency domain signal, and calculating the voice energy Pv of the audio; S3, calculating the real-time voice coefficient Vp corresponding to each audio; S4, updating the participant weight M[k] of each audio according to the current voice coefficient Vp of each audio in the conference; S5, updating the current decay factor f[k]; and S6, finally calculating the mixing data Mix through the participant weight M[k], the voice coefficient Vp, the time domain signal and the decay factor f[k]. The audio mixing scheme can avoid overflow during mixing, reasonably allocate the weight according to the state of the speaker, avoid the audio of the main speaker from being attenuated, and ensure that the audio of the main speaker is not affected as much as possible in the conference scene.

Description

Technical Field

[0001] This invention relates to the field of communication data processing technology, and in particular to a digital audio mixing method for multi-person voice conferencing scenarios. Background Technology

[0002] Audio and video data processing is an essential part of modern wireless communication. With the development of communication technology, audio and video data processing techniques and solutions are constantly being improved and optimized. For digital audio signals, existing mixing processing methods mainly include the following:

[0003] Option 1: Directly add the multiple audio streams together.

[0004] Option 2: Average the multiple audio streams.

[0005] Option 3: When multiple audio streams are directly added together and overflow occurs, an attenuation coefficient is introduced to attenuate the mixed audio stream, and then the attenuation factor is gradually eliminated.

[0006] Option 4: Newlc algorithm mixing. This mixing scheme uses empirical formulas for mixing.

[0007] The above-mentioned digital audio mixing methods have some obvious drawbacks, as follows:

[0008] Adding multiple audio streams directly will cause overflow, resulting in a significant decrease in audio quality.

[0009] Using a multi-channel audio averaging method can eliminate the overflow problem, but it will significantly reduce the volume of each audio channel.

[0010] The adaptive attenuation coefficient scheme can improve the volume reduction problem when only one audio stream is speaking, compared to the average scheme. However, the volume reduction problem still occurs when multiple audio streams are present at the same time.

[0011] The NewLC algorithm for mixing works well for two-channel mixing, but the quality drops significantly for multi-channel mixing.

[0012] It can be seen that existing audio mixing methods suffer from problems such as audio overflow and volume attenuation, which lead to a decrease in audio quality after mixing. They are also less effective at handling multi-channel audio mixing. Therefore, it is necessary to research and design new digital audio mixing methods to improve the mixing effect of multi-channel audio and enhance the application effect in multi-person voice conferencing scenarios. Summary of the Invention

[0013] To address the aforementioned problems, this invention provides a digital audio mixing method for multi-person voice conferencing scenarios, comprising the following steps:

[0014] S1, In a multi-person voice conference scenario, several initial value parameters are assigned to the audio of each participant. The initial value parameters include the participant weight M[k], the voice coefficient Vp, and the attenuation factor f[k].

[0015] S2, which converts digital audio from a time-domain signal to a frequency-domain signal, and calculates the speech energy Pv of the audio within a specified frequency range using the frequency-domain signal;

[0016] S3, calculates the real-time speech coefficient Vp for each audio stream based on the speech energy sequence of each audio stream over a certain period of time;

[0017] S4, based on the current speech coefficient Vp of each audio stream in the meeting room, update the participation weight M[k] of each audio stream;

[0018] S5, based on the latest participation weight M[k] and speech coefficient Vp, as well as the current time domain signal, update the current attenuation factor f[k];

[0019] S6, the final mixed output Mix is ​​calculated using the participation weight M[k], speech coefficient Vp, time domain signal and attenuation factor f[k].

[0020] As a further explanation of the present invention, the speech coefficient Vp is used to characterize the speaking intensity of the speaker in the audio stream over a recent period of time, and its calculation formula is as follows:

[0021] Vp = SUM(Pv[i] * (1 / (1+i / 5)))

[0022] Let the audio frame interval be T, let Pv[0] be the latest speech energy, Pv[1] be the speech energy of the previous frame, Pv[2] be the speech energy of the previous two frames, Pv[i] be the speech energy of the time T*i away from the current time, and i be any value that makes T*i < 1 second true.

[0023] Furthermore, the speech energy Pv is used to characterize the instantaneous intensity of a human speaking voice in a frame of audio signal in an audio stream, and its calculation formula is as follows:

[0024] Pv = SUM(X[i]^2)

[0025] The audio PCM time domain data x[i] is used to perform FFT (Fast Fourier Transform) to obtain X[i], where x[i] is the time domain amplitude sequence and X[i] is the spectrum sequence of each frequency point, where i takes values ​​in the closed interval [a,b].

[0026] Furthermore, when i is in the range [a,b], X[i] is the spectral data in the frequency range of 60Hz–400Hz.

[0027] Furthermore, the participation weight M[k] is used to characterize the speech weight of one audio stream in the meeting up to the present, and its calculation formula is as follows:

[0028] M[k][i] = 0.8 * M[k][i-1] + Am[k]

[0029] M[k][i] is the participation weight at the current frame time, M[k][i-1] is the participation weight at the previous frame time, and Am[k] is the participation gain at the current time.

[0030] Let there be N audio streams in the venue at the current moment, with the speech coefficient of the k-th audio stream being Vp[k].

[0031] For the k-th audio channel

[0032] Let the initial participation weight be M[k][0] = 1

[0033] Let Vps be the sum of the voice coefficients of all N audio channels, then:

[0034] Vps = SUM(Vp[k]), where k belongs to [0, N]

[0035] Let A[k] be the participation gain of the k-th audio channel at the current moment, and let:

[0036] Am[k]=(Vp[k] / Vps>30%)? Vp[k]:0.

[0037] Furthermore, for the k-th audio stream, the initial attenuation factor is f[k][0] = 1, the attenuation factor of the previous frame is f[k][i], the attenuation factor of the last frame is f[k][i-1], and the iterative attenuation factor is f[k][i][j].

[0038] f[k][i][j]=f[k][i-1]*0.9^(j*(1-Vp[k] / Vps))

[0039] The coefficient j is the smallest positive integer or 0 that prevents the mix from overflowing, Vp[k] is the current speech coefficient, and Vps is the sum of the speech coefficients of all audio.

[0040] Furthermore, the formula for calculating the final mix output is:

[0041] Mix=SUM(f[k][i]*M[k]*Vp[k]*x[k])*2 / (M*Vps)

[0042] f[k][i] is the attenuation factor at the previous frame time, M[k] is the current participation weight, Vp[k] is the current speech coefficient, x[k] is the amplitude of the time-domain sampling of the current frame, M is the sum of the participation weights of all audio, and Vps is the sum of the speech coefficients of all audio;

[0043] Denote the maximum positive amplitude as MAX, the minimum negative amplitude as MIN, the iteration coefficient as j, and the iterative mixing result as mix.

[0044] Let:

[0045] mix = SUM(f[k][i][j] * M[k] * Vp[k] * x[k]) * 2 / (M * Vps)

[0046] If mix > MAX, then starting from 1, increase the value of j in sequence until mix is not greater than MAX. Denote the value of j at this time as J, and let f[k][i] = f[k][i][J]

[0047] If mix < MIN, then starting from 1, increase the value of j in sequence until mix is not less than MIN. Denote the value of j at this time as J, and let f[k][i] = f[k][i][J];

[0048] If mix > MIN and mix < MAX, let j = 0.

[0049] Furthermore, for each audio channel, if j = 0 and f[k][i - 1] < 1, then update the attenuation factor, and its formula is:

[0050] f[k][i] = f[k][i - 1] + (1 - f[k][i - 1]) / 12.

[0051] Advantages of the present invention:

[0052] The audio mixing scheme of the present invention can avoid overflow during mixing; can reasonably allocate weights according to the state of the speaker; avoid attenuation of the audio of the main speaker; can reasonably allocate speech weights according to the discussion situation of the meeting in the venue; there will be no obvious change in the volume strength of each audio channel caused by mixing. It can effectively eliminate the audio overflow problem while minimizing the volume attenuation problem, and ensure that the audio of the main speaker is minimally affected in the meeting scenario, which is more suitable for the application in the voice conference scenario. Brief Description of the Drawings

[0053] Figure 1 It is a flow chart showing the digital audio mixing method of the present invention for the multi-person voice conference scenario. Detailed Embodiments

[0054] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0055] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", "first", "second", etc., indicate the orientation, position or order relationship based on the orientation, position or order relationship shown in the drawings, and are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0056] A digital audio mixing method for multi-person voice conferencing scenarios, comprising the following steps:

[0057] S1. In a multi-person voice conference scenario, several initial value parameters are assigned to the audio of each participant. The initial value parameters include the participant weight M[k], the voice coefficient Vp, and the attenuation factor f[k].

[0058] S2 involves converting digital audio from a time-domain signal to a frequency-domain signal, and then calculating the speech energy Pv of the audio within a specified frequency range using the frequency-domain signal. In this embodiment, the speech energy Pv is used to characterize the instantaneous intensity of a human voice in a frame of the audio stream. Specifically, X[i] is obtained by performing FFT (Fast Fourier Transform) on the audio PCM time-domain sampling data x[i], where x[i] is the time-domain amplitude sampling sequence and X[i] is the spectral sequence of each frequency point.

[0059] Let X[i] be the spectral data in the frequency range of 60Hz–400Hz when i is in the range [a,b]. Then the speech energy Pv can be calculated by the following formula:

[0060] Pv = SUM(X[i]^2)

[0061] Where i takes values ​​in the closed interval [a, b].

[0062] S3, by calculating the real-time speech coefficient Vp for each audio stream based on the speech energy sequence of each audio stream over a certain period of time. In this embodiment, the speech coefficient Vp is used to characterize the speaking intensity of the speaker in the audio stream over a recent period of time. Specifically, let the audio frame interval be T, and let Pv[i] be the speech energy at a time T*i away from the current time, that is, the latest speech energy is Pv[0], the speech energy of the previous frame is Pv[1], the speech energy of the two previous frames is Pv[2], and so on. Then the current speech coefficient Vp can be calculated by the following formula:

[0063] Vp = SUM(Pv[i] * (1 / (1+i / 5)))

[0064] Where i represents all values ​​that make T*i < 1 second true.

[0065] S4, update the participation weight M[k] of each audio stream based on its current speech coefficient Vp in the meeting room. In this embodiment, the participation weight M[k] represents the speech weight of an audio stream in the meeting up to the present. Specifically, there are N audio streams in the meeting room at the current moment, and the speech coefficient of the k-th audio stream is Vp[k].

[0066] For the k-th audio channel, let the participation weight at the current frame be M[k][i], and the participation weight at the previous frame be M[k][i-1]. Let the initial participation weight be M[k][0] = 1. Let the sum of the speech coefficients of all N audio channels be Vps, then:

[0067] Vps = SUM(Vp[k]), where k belongs to [0, N]

[0068] Let A[k] be the participation gain of the k-th audio channel at the current moment, and let:

[0069] Am[k] = (Vp[k] / Vps>30%)? Vp[k]: 0, that is, if Vp[k] / Vps>30% is true, then Am[k] = Vp[k], otherwise Am[k] = 0.

[0070] The current participation weight M[k][i] is calculated based on the current participation gain Am[k] and the participation weight M[k][i-1] of the previous frame, using the following formula:

[0071] M[k][i]=0.8*M[k][i-1]+Am[k].

[0072] S5 updates the current attenuation factor f[k] based on the latest participation weight M[k] and speech coefficient Vp, as well as the current time domain signal.

[0073] S6, the final mix output Mix is ​​calculated using the participation weight M[k], speech coefficients Vp, time-domain signal, and attenuation factor f[k]. To prevent mix overflow, different attenuation factors are set for each audio channel. The maximum positive amplitude is MAX, the minimum negative amplitude is MIN, the sum of the speech coefficients of all audio channels is Vps, and the sum of the participation weights of all audio channels is M.

[0074] For the k-th audio stream, let the attenuation factor at the current frame be f[k][i], and the attenuation factor at the previous frame be f[k][i-1]. Let:

[0075] The attenuation factor at the initial moment is f[k][0] = 1.

[0076] Denote the time-domain sampling amplitude of the current frame as x[k], the current participation weight as M[k], the current voice coefficient as Vp[k], and the iterative attenuation factor as f[k][i][j]. Let:

[0077] f[k][i][j] = f[k][i - 1] * 0.9^(j * (1 - Vp[k] / Vps))

[0078] Where the coefficient j is the smallest positive integer or 0 that prevents the mixing from overflowing.

[0079] Denote the iterative coefficient as j and the iterative mixing result as mix. Let:

[0080] mix = SUM(f[k][i][j] * M[k] * Vp[k] * x[k]) * 2 / (M * Vps)

[0081] If mix > MAX, then starting from 1, increase the value of j sequentially until mix is no greater than MAX. Denote the value of j at this time as J.

[0082] Let f[k][i] = f[k][i][J];

[0083] If mix < MIN, then starting from 1, increase the value of j sequentially until mix is no less than MIN. Denote the value of j at this time as J.

[0084] Let f[k][i] = f[k][i][J];

[0085] If mix > MIN and mix < MAX, let j = 0.

[0086] Denote the final mixed audio output as Mix, then

[0087] Mix = SUM(f[k][i] * M[k] * Vp[k] * x[k]) * 2 / (M * Vps)

[0088] For each audio channel k, if j = 0 and f[k][i - 1] < 1, update the attenuation factor. The update calculation formula is:

[0089] f[k][i] = f[k][i - 1] + (1 - f[k][i - 1]) / 12.

[0090] The audio mixing scheme of this invention can avoid overflow during mixing; it can reasonably allocate weights according to the speaker's state; it avoids audio attenuation of the main speaker; it can reasonably allocate voice weights according to the discussion in the meeting room; and each audio stream will not produce obvious volume changes caused by mixing. It effectively eliminates audio overflow problems while minimizing volume attenuation problems, and ensures that the audio of the main speaker is as unaffected as possible in a meeting setting, making it more suitable for voice conferencing applications.

[0091] The above description only illustrates preferred embodiments of the present invention and should not be construed as limiting the scope of the claims. The present invention is not limited to the above embodiments, and variations in its specific structure are permitted. In short, all variations made within the scope of the independent claims of the present invention are within the scope of protection of the present invention.

Claims

1. A digital audio mixing method for multi-person voice conferencing scenarios, characterized in that, Includes the following steps: S1, In a multi-person voice conference scenario, several initial value parameters are assigned to the audio of each participant. The initial value parameters include the participant weight M[k], the voice coefficient Vp, and the attenuation factor f[k]. S2 converts the digital audio from a time-domain signal to a frequency-domain signal, and calculates the speech energy Pv of the audio within a specified frequency range using the frequency-domain signal; S3, calculates the real-time speech coefficient Vp for each audio stream based on the speech energy sequence of each audio stream over a certain period of time; S4, based on the current speech coefficient Vp of each audio stream in the meeting room, update the participation weight M[k] of each audio stream; S5, based on the latest participation weight M[k] and speech coefficient Vp, as well as the current time domain signal, update the current attenuation factor f[k]; S6, the final mixed output Mix is ​​calculated using the participation weight M[k], speech coefficient Vp, time-domain signal, and attenuation factor f[k]. The formula for calculating the final mixed output Mix is ​​as follows: Mix=SUM(f[k][i]*M[k]*Vp[k]*x[k])*2 / (M*Vps) f[k][i] is the attenuation factor of the previous frame, M[k] is the current participation weight, Vp[k] is the current speech coefficient, x[k] is the temporal sampling amplitude of the current frame, M is the sum of the participation weights of all audio, and Vps is the sum of the speech coefficients of all audio.

2. The digital audio mixing method for multi-person voice conferencing scenarios according to claim 1, characterized in that: The speech coefficient Vp is used to characterize the speaking intensity of the speaker in the audio stream over a recent period of time, and its calculation formula is as follows: Vp=SUM(Pv[i]*(1 / (1+i / 5))) Let the audio frame interval be T, let Pv[0] be the latest speech energy, Pv[1] be the speech energy of the previous frame, Pv[2] be the speech energy of the previous two frames, Pv[i] be the speech energy of the time T*i away from the current time, and i be any value that makes T*i < 1 second true.

3. The digital audio mixing method for multi-person voice conferencing scenarios according to claim 2, characterized in that: The speech energy Pv is used to characterize the instantaneous intensity of a human speaking voice in a frame of audio signal in an audio stream, and its calculation formula is as follows: Pv=SUM(X[i]^2) The audio PCM time domain is obtained by performing a fast Fourier transform on the data x[i] to obtain X[i], where x[i] is the time domain amplitude sequence and X[i] is the spectrum sequence of each frequency point, where i takes values ​​in the closed interval [a,b].

4. The digital audio mixing method for multi-person voice conferencing scenarios according to claim 3, characterized in that: When i is in the range [a,b], X[i] is the spectral data in the frequency range of 60Hz–400Hz.

5. The digital audio mixing method for multi-person voice conferencing scenarios according to claim 1, characterized in that: The participation weight M[k] is used to characterize the speech weight of one audio stream in the meeting up to the present, and its calculation formula is as follows: M[k][i] = 0.8 * M[k][i-1] + Am[k] M[k][i] is the participation weight at the current frame time, M[k][i-1] is the participation weight at the previous frame time, and Am[k] is the participation gain at the current time. Let there be N audio streams in the venue at the current moment, with the speech coefficient of the k-th audio stream being Vp[k]. For the k-th audio channel Let the initial participation weight be M[k][0]=1 Let Vps be the sum of the voice coefficients of all N audio channels, then: Vps = SUM(Vp[k]), where k belongs to [0, N] Denote the participating gain of the k-th audio at the current moment as A[k], and let: Am[k] = (Vp[k] / Vps > 30%)? Vp[k] :

0.

6. The digital audio mixing method for multi-person voice conferencing scenarios according to claim 1, characterized in that: For the k-th audio, the attenuation factor at the initial moment is f[k][0] = 1, the attenuation factor at the previous frame moment is f[k][i], the attenuation factor at the previous frame moment is f[k][i - 1], and the iterative attenuation factor is f[k][i][j].[[]END]] f[k][i][j] = f[k][i - 1] * 0.9 ^ (j * (1 - Vp[k] / Vps)) The coefficient j is the smallest positive integer or 0 that prevents the mixing from overflowing. Vp[k] is the current speech coefficient, and Vps is the sum of the speech coefficients of all audios.

7. The digital audio mixing method for a multi-person voice conference scenario according to claim 6, wherein: Denote the maximum positive amplitude as MAX, the minimum negative amplitude as MIN, the iterative coefficient as j, and the iterative mixing result as mix, and let: mix = SUM(f[k][i][j] * M[k] * Vp[k] * x[k]) * 2 / (M * Vps) If mix > MAX, then starting from 1, increase the value of j in sequence until mix is not greater than MAX. Denote the value of j at this time as J, and let f[k][i] = f[k][i][J].[[]END]] If mix < MIN, then starting from 1, increase the value of j in sequence until mix is not less than MIN. Denote the value of j at this time as J, and let f[k][i] = f[k][i][J]; If mix > MIN and mix < MAX, let j = 0.

8. The digital audio mixing method for multi-person voice conferencing scenarios according to claim 7, characterized in that: For each audio, if j = 0 and f[k][i - 1] < 1, then update the attenuation factor, and the formula is: f[k][i] = f[k][i - 1] + (1 - f[k][i - 1]) / 12.

Images