Digital audio mixing method for multi-user voice conference scene

Abstract

The invention discloses a digital audio mixing method for a multi-person voice conference scene, which comprises the following steps of: S1, endowing parameters such as a conference participation weight M [k], a voice coefficient Vp and an attenuation factor f [k] to the audio of each participant in the multi-person voice conference scene; s2, converting the digital audio from a time domain signal to a frequency domain signal, and calculating the voice energy Pv of the audio; s3, calculating to obtain a real-time voice coefficient Vp corresponding to each path of audio; s4, updating the conference participation weight M [k] of each path of audio according to the current voice coefficient Vp of each path of audio in the conference place; s5, updating f [k] on the current attenuation factor; and S6, finally calculating through the participation weight M [k], the voice coefficient Vp, the time domain signal and the attenuation factor f [k] to obtain sound mixing data Mix. According to the audio mixing scheme, overflow during audio mixing can be avoided, weights are reasonably distributed according to the states of the speakers, audio attenuation of the main speakers is avoided, and the audios of the main speakers are not affected as much as possible in a conference scene.

Description

Technical Field

[0001] The present invention relates to the technical field of communication data processing, and in particular to a digital audio mixing method for a multi-person voice conference scenario. Background Art

[0002] Audio and video data processing is an essential part of modern wireless communication. With the development of communication technology, the technology and solutions of audio and video data processing are also constantly improving and optimizing. For digital audio signals, the existing mixing processing methods mainly include the following:

[0003] Solution 1: Directly add multiple audio streams.

[0004] Solution 2: Averaging multiple audio streams.

[0005] Solution 3: When multiple audio streams are directly added together and overflow occurs, an attenuation factor is introduced to attenuate the mixed audio stream, and then the attenuation factor is gradually eliminated.

[0006] Solution 4: NewLC algorithm mixing, this mixing solution uses an empirical formula for mixing.

[0007] The above digital audio mixing methods have some obvious defects, as follows:

[0008] Directly adding multiple audio channels will cause overflow, which will significantly reduce the audio quality.

[0009] The solution of averaging multiple audio channels can eliminate the overflow problem, but it will significantly reduce the volume of each audio channel.

[0010] The adaptive attenuation coefficient solution can improve the volume reduction problem when only one audio stream is speaking, but the volume reduction problem will still occur when multiple streams have audio at the same time.

[0011] NewLC algorithm mixing, this solution has a good effect when mixing two channels, but the quality of multi-channel mixing is significantly reduced.

[0012] It can be seen that the existing mixing processing method has problems such as audio overflow, volume attenuation, etc., which cause the audio quality to deteriorate after mixing. The effect of processing multi-channel audio mixing is poor. Therefore, it is necessary to study and design a new digital audio mixing method to improve the mixing effect of multi-channel audio and improve the application effect in multi-person voice conferencing scenarios. Summary of the invention

[0013] In view of the above problems, the present invention provides a digital audio mixing method for a multi-person voice conference scenario, comprising the following steps:

[0014] S1, in a multi-person voice conference scenario, assigning a number of initial value parameters to the audio of each participant, the initial value parameters including the participant weight M[k], the voice coefficient Vp and the attenuation factor f[k];

[0015] S2, converting the digital audio from a time domain signal to a frequency domain signal, and calculating the speech energy Pv of the audio within a specified frequency range through the frequency domain signal;

[0016] S3, calculating a real-time speech coefficient Vp corresponding to each audio channel based on the speech energy sequence of each audio channel within a period of time;

[0017] S4, updating the conference weight M[k] of each audio channel according to the current voice coefficient Vp of each audio channel in the conference venue;

[0018] S5, based on the latest current conference weight M[k] and speech coefficient Vp, as well as the current time domain signal, update the current attenuation factor f[k];

[0019] S6, finally the final mixed output Mix is ​​calculated by the conference weight M[k], the speech coefficient Vp, the time domain signal and the attenuation factor f[k].

[0020] As a further explanation of the present invention, the speech coefficient Vp is used to characterize the speech intensity of the speaker of the audio stream in the recent period of time, and its calculation formula is:

[0021] Vp=SUM(Pv[i]*(1 / (1+i / 5)))

[0022] Let the audio frame interval be T, Pv[0] be the current latest speech energy, Pv[1] be the speech energy of the previous frame, Pv[2] be the speech energy of the previous two frames, Pv[i] be the speech energy at T*i time from the current moment, and i be all values ​​that make T*i<1 second.

[0023] Furthermore, the speech energy Pv is used to characterize the instantaneous intensity of a person's speaking voice in a frame of audio signal of the audio stream, and its calculation formula is:

[0024] Pv=SUM(X[i]^2)

[0025] The audio PCM time domain uses data x[i] to perform FFT (Fast Fourier Transform) to obtain X[i], where x[i] is the time domain amplitude sampling sequence, X[i] is the spectrum sequence of each frequency point, and i takes values ​​in the closed interval [a, b].

[0026] Furthermore, when i is in the range of [a,b], X[i] is the spectrum data in the frequency range of 60 Hz–400 Hz.

[0027] Furthermore, the conference participation weight M[k] is used to characterize the voice weight of one audio channel in the current conference, and its calculation formula is:

[0028] M[k][i]=0.8*M[k][i-1]+Am[k]

[0029] M[k][i] is the participation weight at the current frame, M[k][i-1] is the participation weight at the previous frame, and Am[k] is the participation gain at the current moment;

[0030] At this moment, there are N audio channels in the conference room, and the speech coefficient of the kth audio channel is Vp[k]

[0031] For the kth audio channel

[0032] Let the initial participation weight be M[k][0]=1

[0033] The sum of the voice coefficients of all N audio channels is Vps, then:

[0034] Vps=SUM(Vp[k]), k belongs to [0,N]

[0035] Let the current participation gain of the k-th audio channel be A[k], and let:

[0036] Am[k]=(Vp[k] / Vps>30%)? Vp[k]:0.

[0037] Furthermore, for the k-th audio channel, the attenuation factor at the initial moment is f[k][0]=1, the attenuation factor at the previous frame moment is f[k][i], the attenuation factor at the previous frame moment is f[k][i-1], and the iterative attenuation factor is f[k][i][j].

[0038] f[k][i][j]=f[k][i-1]*0.9^(j*(1-Vp[k] / Vps))

[0039] The coefficient j is the smallest positive integer or 0 that prevents the mixing from overflowing, Vp[k] is the current speech coefficient, and Vps is the sum of the speech coefficients of all audios.

[0040] Furthermore, the calculation formula of the final mixed output Mix is:

[0041] Mix=SUM(f[k][i]*M[k]*Vp[k]*x[k])*2 / (M*Vps)

[0042] f[k][i] is the attenuation factor at the previous frame time, M[k] is the current participation weight, Vp[k] is the current speech coefficient, x[k] is the amplitude of the time-domain sampling of the current frame, M is the sum of the participation weights of all audio, and Vps is the sum of the speech coefficients of all audio;

[0043] Let the maximum positive amplitude be MAX, the minimum negative amplitude be MIN, the iteration coefficient be j, and the iterative mixing result be mix.

[0044] Let:

[0045] mix = SUM(f[k][i][j] * M[k] * Vp[k] * x[k]) * 2 / (M * Vps)

[0046] If mix > MAX, then starting from 1, increase the value of j in sequence until mix is not greater than MAX. Denote the value of j at this time as J, and let f[k][i] = f[k][i][J]

[0047] If mix < MIN, then starting from 1, increase the value of j in sequence until mix is not less than MIN. Denote the value of j at this time as J, and let f[k][i] = f[k][i][J];

[0048] If mix > MIN and mix < MAX, let j = 0.

[0049] Furthermore, for each audio channel, if j = 0 and f[k][i - 1] < 1, then update the attenuation factor, and its formula is:

[0050] f[k][i] = f[k][i - 1] + (1 - f[k][i - 1]) / 12.

[0051] Advantages of the present invention:

[0052] The audio mixing scheme of the present invention can avoid overflow during mixing; can reasonably allocate weights according to the state of the speaker; avoid attenuation of the audio of the main speaker; can reasonably allocate speech weights according to the discussion situation of the meeting in the venue; each audio channel will not produce obvious volume strength changes caused by mixing. It can effectively eliminate the audio overflow problem while minimizing the volume attenuation problem, and ensure that the audio of the main speaker is minimally affected in the conference scenario, which is more suitable for the application of the voice conference scenario. Brief Description of the Drawings

[0053] Figure 1 It is a flow chart showing the digital audio mixing method for the multi-person voice conference scenario of the present invention. Detailed Embodiment

[0054] The following is a detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0055] In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside", "first", "second", etc., indicating directions or positions or sequential relationships, are based on the directions or positions or sequential relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention.

[0056] A digital audio mixing method for a multi-person voice conference scenario is invented, comprising the following steps:

[0057] S1, in a multi-person voice conference scenario, a number of initial value parameters are assigned to the audio of each participant, and the initial value parameters include the participant weight M[k], the voice coefficient Vp and the attenuation factor f[k].

[0058] S2, converting the digital audio from a time domain signal to a frequency domain signal, and calculating the speech energy Pv of the audio within a specified frequency range through the frequency domain signal. In this embodiment, the speech energy Pv is used to characterize the instantaneous intensity of a person's voice in a frame of an audio signal of an audio stream. Specifically, the audio PCM time domain sampling data x[i] is subjected to FFT (Fast Fourier Transform) to obtain X[i], where x[i] is a time domain amplitude sampling sequence, and X[i] is a frequency spectrum sequence of each frequency point.

[0059] When i is in the range of [a, b], X[i] is the spectrum data in the frequency range of 60Hz–400Hz, then the speech energy Pv can be calculated by the following formula:

[0060] Pv=SUM(X[i]^2)

[0061] Here, i takes values ​​in the closed interval [a,b].

[0062] S3, calculates the real-time speech coefficient Vp corresponding to each audio channel based on the speech energy sequence of each audio channel within a period of time. In this embodiment, the speech coefficient Vp is used to characterize the speech intensity of the speaker of the audio stream in the recent period of time. Specifically, the audio frame interval is recorded as T, and Pv[i] is recorded as the speech energy at T*i time from the current moment, that is, the current latest speech energy is Pv[0], the speech energy of the previous frame is Pv[1], the speech energy of the previous two frames is Pv[2], and so on. The current speech coefficient Vp can be calculated by the following formula:

[0063] Vp=SUM(Pv[i]*(1 / (1+i / 5)))

[0064] Where i is all the values ​​that make T*i<1 second.

[0065] S4, based on the current voice coefficient Vp of each audio channel in the conference room, update the conference weight M[k] of each audio channel. In this embodiment, the conference weight M[k] is the voice weight of a channel in the current conference. Specifically, there are N audio channels in the conference room at the current moment, and the voice coefficient of the kth audio channel is Vp[k].

[0066] For the k-th audio channel, let the participation weight at the current frame be M[k][i], the participation weight at the previous frame be M[k][i-1], let the participation weight at the initial moment be M[k][0]=1, let the sum of the speech coefficients of all N audio channels be Vps, then:

[0067] Vps=SUM(Vp[k]), k belongs to [0,N]

[0068] Let the current participation gain of the k-th audio channel be A[k], and let:

[0069] Am[k]=(Vp[k] / Vps>30%)? Vp[k]:0, that is, if Vp[k] / Vps>30% is true, then Am[k]=Vp[k], otherwise Am[k]=0.

[0070] The current participation weight M[k][i] is calculated based on the current participation gain Am[k] and the previous frame participation weight M[k][i-1], and the formula is:

[0071] M[k][i]=0.8*M[k][i-1]+Am[k].

[0072] S5, based on the latest current conference weight M[k] and speech coefficient Vp, as well as the current time domain signal, the current attenuation factor f[k] is updated.

[0073] S6, finally the final mixed output Mix is ​​calculated by the conference weight M[k], speech coefficient Vp, time domain signal and attenuation factor f[k]. In order to prevent the overflow of the mixing, a different attenuation factor is set for each audio channel, the maximum positive amplitude is MAX, the minimum negative amplitude is MIN, the sum of the speech coefficients of all audio is Vps, and the sum of the conference weights of all audio is M.

[0074] For the k-th audio channel, let the attenuation factor of the current frame be f[k][i], and the attenuation factor of the previous frame be f[k][i-1]. Let:

[0075] The attenuation factor at the initial moment is f[k][0] = 1.

[0076] Denote the time-domain sampling amplitude of the current frame as x[k], the current participation weight as M[k], the current voice coefficient as Vp[k], and the iterative attenuation factor as f[k][i][j]. Let:

[0077] f[k][i][j] = f[k][i - 1] * 0.9^(j * (1 - Vp[k] / Vps))

[0078] Where the coefficient j is the smallest positive integer or 0 that prevents the mix from overflowing.

[0079] Denote the iterative coefficient as j and the iterative mix result as mix. Let:

[0080] mix = SUM(f[k][i][j] * M[k] * Vp[k] * x[k]) * 2 / (M * Vps)

[0081] If mix > MAX, then starting from 1, increase the value of j successively until mix is not greater than MAX. Denote the value of j at this time as J.

[0082] Let f[k][i] = f[k][i][J];

[0083] If mix < MIN, then starting from 1, increase the value of j successively until mix is not less than MIN. Denote the value of j at this time as J.

[0084] Let f[k][i] = f[k][i][J];

[0085] If mix > MIN and mix < MAX, let j = 0.

[0086] Denote the final mix output as Mix, then

[0087] Mix = SUM(f[k][i] * M[k] * Vp[k] * x[k]) * 2 / (M * Vps)

[0088] For each audio channel k, if j = 0 and f[k][i - 1] < 1, update the attenuation factor. The update calculation formula is:

[0089] f[k][i] = f[k][i - 1] + (1 - f[k][i - 1]) / 12.

[0090] The audio mixing solution of the present invention can avoid overflow during mixing; can reasonably allocate weights according to the speaker's status; avoid audio attenuation of the main speaker; can reasonably allocate voice weights according to the discussion of the meeting at the venue; each audio channel will not produce obvious volume changes caused by mixing. While effectively eliminating the audio overflow problem, it can also minimize the volume attenuation problem, and in the conference scene, it can ensure that the audio of the main speaker is not affected as much as possible, which is more suitable for the application of voice conference scenes.

[0091] The above only describes the preferred embodiments of the present invention, but it cannot be understood as limiting the claims. The present invention is not limited to the above embodiments, and its specific structure is allowed to be changed. In short, all changes made within the protection scope of the independent claims of the present invention are within the protection scope of the present invention.

Claims

1. A digital audio mixing method for a multi-person voice conference scenario, characterized in that: The following steps are involved: S1, in a multi-person voice conference scenario, assigning a number of initial value parameters to the audio of each participant, the initial value parameters including the participant weight M[k], the voice coefficient Vp and the attenuation factor f[k]; S2, converting the digital audio from a time domain signal to a frequency domain signal, and calculating the speech energy Pv of the audio within a specified frequency range through the frequency domain signal; S3, calculating a real-time speech coefficient Vp corresponding to each audio channel based on the speech energy sequence of each audio channel within a period of time; S4, updating the conference weight M[k] of each audio channel according to the current voice coefficient Vp of each audio channel in the conference venue; S5, based on the latest current conference weight M[k] and speech coefficient Vp, as well as the current time domain signal, update the current attenuation factor f[k]; S6, finally the final mixed output Mix is ​​calculated by the conference weight M[k], the speech coefficient Vp, the time domain signal and the attenuation factor f[k].

2. The digital audio mixing method for a multi-person voice conference scenario according to claim 1 is characterized in that: The speech coefficient Vp is used to characterize the speech intensity of the speaker in the audio stream in the recent period of time, and its calculation formula is: Vp=SUM(Pv[i]*(1 / (1+i / 5))) Let the audio frame interval be T, Pv[0] be the current latest speech energy, Pv[1] be the speech energy of the previous frame, Pv[2] be the speech energy of the previous two frames, Pv[i] be the speech energy at T*i time from the current moment, and i be all values ​​that make T*i<1 second.

3. The digital audio mixing method for a multi-person voice conference scenario according to claim 2 is characterized in that: The speech energy Pv is used to characterize the instantaneous intensity of a person's speaking voice in a frame of audio signal of the audio stream, and its calculation formula is: Pv=SUM(X[i]^2) The audio PCM time domain uses data x[i] to perform FFT (Fast Fourier Transform) to obtain X[i], where x[i] is the time domain amplitude sampling sequence, X[i] is the spectrum sequence of each frequency point, and i takes values ​​in the closed interval [a, b].

4. The digital audio mixing method for a multi-person voice conference scenario according to claim 3 is characterized in that: When i is in the range [a,b], X[i] is the spectrum data in the frequency range 60 Hz–400 Hz.

5. The digital audio mixing method for a multi-person voice conference scenario according to claim 1 is characterized in that: The conference participation weight M[k] is used to characterize the voice weight of one audio channel in the current conference, and its calculation formula is: M[k][i]=0.8*M[k][i-1]+Am[k] M[k][i] is the participation weight at the current frame, M[k][i-1] is the participation weight at the previous frame, and Am[k] is the participation gain at the current moment; At this moment, there are N audio channels in the conference room, and the speech coefficient of the kth audio channel is Vp[k] For the kth audio channel Let the initial participation weight be M[k][0]=1 The sum of the voice coefficients of all N audio channels is Vps, then: Vps=SUM(Vp[k]), k belongs to [0,N] The current conference gain of the k-th audio channel is A[k], and: Am[k]=(Vp[k] / Vps>30%)? Vp[k]:

0.

6. According to claim 1, the digital audio mixing method for multi-person voice conference scenario is characterized by: For the k-th audio channel, the attenuation factor at the initial moment is f[k][0] = 1, the attenuation factor at the previous frame moment is f[k][i], the attenuation factor at the previous frame moment is f[k][i - 1], and the iterative attenuation factor is f[k][i][j]. f[k][i][j] = f[k][i - 1] * 0.9^(j * (1 - Vp[k] / Vps)) The coefficient j is the smallest positive integer or 0 that prevents the mix from overflowing. Vp[k] is the current voice coefficient, and Vps is the sum of the voice coefficients of all audio channels.

7. The digital audio mixing method for a multi-person voice conference scenario according to claim 6 is characterized in that: The formula for calculating the final mixed output Mix is as follows: Mix = SUM(f[k][i] * M[k] * Vp[k] * x[k]) * 2 / (M * Vps) f[k][i] is the attenuation factor at the previous frame moment, M[k] is the current participant weight, Vp[k] is the current voice coefficient, x[k] is the amplitude of the time-domain sampling of the current frame, M is the sum of the participant weights of all audio channels, and Vps is the sum of the voice coefficients of all audio channels. Denote the maximum positive amplitude as MAX, the minimum negative amplitude as MIN, the iterative coefficient as j, and the iterative mixing result as mix. Let: mix = SUM(f[k][i][j] * M[k] * Vp[k] * x[k]) * 2 / (M * Vps) If mix > MAX, then starting from 1, increase the value of j sequentially until mix is no greater than MAX. Denote the value of j at this time as J, and let f[k][i] = f[k][i][J]. If mix < MIN, then starting from 1, increase the value of j sequentially until mix is no less than MIN. Denote the value of j at this time as J, and let f[k][i] = f[k][i][J]. If mix > MIN and mix < MAX, let j = 0.

8. The digital audio mixing method for a multi-person voice conference scenario according to claim 7 is characterized in that: For each audio channel, if j = 0 and f[k][i - 1] < 1, update the attenuation factor. The formula is: f[k][i] = f[k][i - 1] + (1 - f[k][i - 1]) / 12.

Images