Adaptive array microphone cascade sound mixing method

Abstract

The invention discloses an adaptive array microphone cascade sound mixing method, which is applied to a microphone cascade sound mixing system, and belongs to the technical field of communication. The microphone cascade sound mixing system comprises a synchronization module, a channel gain equalization estimation module and a self-adaptive sound mixing module, the synchronization module is in communication connection with the channel gain equalization estimation module through a plurality of first channels, and the channel gain equalization estimation module is in communication connection with the self-adaptive sound mixing module through a plurality of second channels. The synchronization module is in communication connection with the self-adaptive sound mixing module through a plurality of channels III; the first microphone array device is connected with the synchronization module through a Ch0 channel,..., and the Nth microphone array device is connected with the synchronization module through a Ch (N-1) channel. By arranging the synchronization module, the channel gain equalization estimation module and the self-adaptive sound mixing module and adopting two-stage phase-level synchronization processing and sound mixing synchronization processing, multiple beneficial effects of volume equalization, target voice enhancement, signal quality optimization and the like can be realized.

Description

Technical Field

[0001] The invention belongs to the technical field of communications, and in particular relates to an adaptive array microphone cascade mixing method. Background Art

[0002] In conference scenarios, microphone device cascading solutions are widely used to cover a wider pickup area and ensure that no matter where the participants are in the conference room, their speeches can be clearly captured. Traditional conference microphone cascading systems are usually composed of multiple microphone array devices, which are dispersed in various key locations in the conference room to achieve comprehensive coverage of the entire conference space. As in the prior art, a Chinese patent with publication number CN205666934U discloses a headset and its cascade system, wherein the headset includes a headset speaker and a microphone, and also includes: an audio interface, which is electrically connected to the headset speaker and the microphone for signal interaction with an external audio device; at least one cascade interface, which is electrically connected to the headset speaker, the microphone and the audio interface for cascading with other headsets; wherein the cascade system is a star connection or serial connection structure composed of a plurality of such headsets.

[0003] Although the above-mentioned existing technology can cascade multiple headsets for use by multiple people and can realize the director function, it can be especially applied to a variety of application scenarios such as chorus and online conferencing. However, in actual applications, how to effectively mix the output voices of these microphone array devices has become a technical problem in this field.

[0004] In traditional mixing methods, the output voices of various microphone array devices are often simply added or averaged. This processing method ignores the actual distance differences between different microphone arrays and speakers, as well as the uneven sound pickup quality of each array. Therefore, the final output mixing results often have the following problems: First, the voice output of the microphone array far away from the speaker may occupy too much weight, resulting in inaccurate sound positioning and reduced speech clarity; second, when participants at multiple locations speak at the same time, the sound volume at each location is difficult to balance, and some sounds are prone to being suppressed or over-amplified; third, the output of a single microphone array may have defects, such as noise interference, echo effect, etc. These defects may be amplified during the mixing process, thereby affecting the overall voice quality.

[0005] For example, the most commonly used method in existing solutions is to configure different mixing weights according to the energy of the output signal of each device. The calculation formula of the mixed signal y(n) is:

[0006] y(n)=a0x0(n)+a1x1(n)+a2x2(n)+a3x3(n)…;

[0007] Among them, x l(n) represents the time domain signal of the first channel; a l is the weighting coefficient of the mth channel; l is the microphone array channel index, which is between 0 and the number of microphones in the microphone array; n represents the sequence number of discrete time, and n is a positive integer. Assume that the output energy of the mth array channel is the largest, and the maximum output energy is: p m =x 2 m , then the maximum output energy p m The corresponding weight configuration of the mth array is: a m =α max ; Weight configuration of arrays other than the mth array: a b =α min ; Output energy p corresponding to other arrays b Both are less than the maximum output energy p m The advantage of this solution is that it is simple to implement, but it also has the above-mentioned problems: if there are people speaking at different positions, it will result in the inability to enhance all human voices at the same time. When speaking at the same time, one voice will be enhanced while another will be weakened. It is also impossible to use multiple microphone arrays at the same time to enhance human voices that are far away.

[0008] Therefore, there is an urgent need for an adaptive array microphone cascade mixing method to obtain better cascade voice quality. Summary of the invention

[0009] In view of the problems in the related art, the present invention proposes an adaptive array microphone cascade mixing method to overcome the above technical problems existing in the existing related art. The present invention can intelligently process signals from different microphone arrays and realize adaptive mixing processing through the coordinated work of the synchronization module, the channel gain equalization estimation module and the adaptive mixing module. The method also enables the present invention to achieve multiple beneficial effects such as volume equalization, target speech enhancement, signal quality optimization, adaptive mixing processing, output defect repair and low-quality output elimination.

[0010] The technical solution of the present invention is implemented as follows: an adaptive array microphone cascade mixing method is applied to a microphone cascade mixing system, wherein the microphone cascade mixing system comprises a synchronization module, a channel gain equalization estimation module and an adaptive mixing module, wherein the synchronization module is connected to the channel gain equalization estimation module through a plurality of channel one communications, the channel gain equalization estimation module is connected to the adaptive mixing module through a plurality of channel two communications, and the synchronization module is connected to the adaptive mixing module through a plurality of channel three communications;

[0011] Also includes a first microphone array device, a second microphone array device, ..., an Nth microphone array device, where N is a positive integer; wherein the first microphone array device is connected to the synchronization module via a Ch0 channel, the second microphone array device is connected to the synchronization module via a Ch1 channel, ..., the Nth microphone array device is connected to the synchronization module via a Ch(N-1) channel;

[0012] The following steps are involved:

[0013] Step 1: Each microphone array device transmits data to the synchronization module, and the synchronization module performs relative delay analysis and voice activation analysis on the data of each channel received to obtain analysis results; then, according to the analysis results, the data of each channel is subjected to secondary phase-level synchronization processing to obtain a first signal;

[0014] Step 2: The channel gain equalization estimation module first receives the first signal after the secondary phase level synchronization processing, and selects the first signal that needs to be mixed synchronously processed, and obtains the second signal after the mixing synchronously processed; in the selection process, the channel gain equalization estimation module analyzes several channels 1 that are currently close to the speaker according to the energy of each channel 1, and selects them as candidate channels, while discarding other channels 1; wherein, the first signal corresponding to the candidate channel needs to be mixed synchronously processed;

[0015] Step three: The adaptive mixing module first receives the second signal processed by the channel gain equalization estimation module; the adaptive mixing module then smoothes the gain according to the gain obtained in the channel gain equalization estimation module of the second signal, and applies it to the signal of each aligned channel two, and then performs delayed addition and mixing processing, and finally obtains the mixed signal after the adaptive array microphone cascade mixing.

[0016] Further, the synchronization module includes a coarse delay synchronization unit and a phase synchronization unit that are communicatively connected to each other; each of the microphone array devices is connected to the coarse delay synchronization unit through different channels, that is, the first microphone array device is connected to the coarse delay synchronization unit through the Ch0 channel, ..., the Nth microphone array device is connected to the coarse delay synchronization unit through the Ch(N-1) channel; the phase synchronization unit is communicatively connected to the channel gain equalization estimation module and the adaptive mixing module respectively;

[0017] Furthermore, the phase synchronization unit is communicatively connected to the channel gain equalization estimation module through a plurality of channels two; and the phase synchronization unit is communicatively connected to the adaptive mixing module through a plurality of channels three.

[0018] Furthermore, in step 1, the process of the secondary phase level synchronization processing is as follows:

[0019] Select Ch0 channel as the reference channel for delay analysis;

[0020] The VAD algorithm is used to analyze whether the signal in the Ch0 channel is speech or noise; it should be noted that the VAD algorithm can be any optional existing algorithm and is not specifically limited in the present invention;

[0021] When the VAD algorithm detects that the voice frame of a signal in a channel is 1, a delay analysis is performed:

[0022] Taking Ch0 and Ch1 as examples, assuming that x o And x1 do real Fourier transform, get The frequency domain signals of X0(ω) and X1(ω) at each point, the calculation formula of the frequency domain signal X0(ω) is:

[0023] X0(ω)=rfft(x0(nK / 2),K);

[0024] Among them, 0 represents the channel ID, n represents the voice frame, and n is a positive integer. For example, if a frame of data has 320 samples, n=0 represents all the data at 1:320 points, n=1 represents all the data at 321:640 points, and so on;

[0025] The calculation formula of the frequency domain signal X1(ω) is:

[0026] X1(ω)=rfft(x1(nK / 2),K);

[0027] Among them, x N represents the time domain signal of the Nth channel; K represents the number of points of the fast Fourier transform FFT; ω represents the frequency; rfft() is the real number Fourier transform;

[0028] Perform mutual coherence calculation on the frequency domain signals X0(ω) and X1(ω) to obtain the coherence coefficient G 0,1 , the mutual interference calculation formula is:

[0029]

[0030] Among them, α is the smoothing coefficient, * means taking the conjugate;

[0031] Then the coherence coefficient G 0,1 Do normalized weighting, the calculation formula is:

[0032]

[0033] Then Perform the inverse real Fourier transform of the length of MK to get R, and the calculation formula is:

[0034]

[0035] Among them, irfft() is the inverse real Fourier transform;

[0036] Finally, the calculation formula for the relative delay of the two signals is:

[0037] D 0,1 =argmax(R)-MK / 2;

[0038] Among them, argmax() represents the index value when the maximum value is obtained; M is the upsampling multiple, ranging from 4 to 6; D 0,1 Indicates the number of delayed samples between Ch0 and Ch1.

[0039] Further, after calculating the relative delay of each channel equivalent to the Ch0 channel, secondary synchronization processing is performed on each channel, and the first signal is obtained after the secondary synchronization processing; wherein the secondary synchronization processing includes coarse synchronization processing and phase synchronization processing;

[0040] It should be noted that if only coarse synchronization is used in the present invention, the delay synchronization granularity will be too large and inaccurate; if only phase synchronization is used, the delay between signals will be too large, resulting in spatial aliasing in the high-frequency part and signal distortion; therefore, the present invention adopts a two-level synchronization process, that is, combining coarse synchronization and phase synchronization at the same time;

[0041] The process of rough synchronization is as follows:

[0042] For the signal excluding the Ch0 channel, the calculation formula is:

[0043]

[0044] in, It represents an integer not greater than this value; n represents the sequence number of discrete time, and n is a positive integer.

[0045] Furthermore, the process of phase synchronization is as follows:

[0046] For the signal excluding the Ch0 channel, the time of the Ch0 channel is aligned with a higher accuracy than the coarse synchronization process. The calculation formula is:

[0047]

[0048] Wherein, l represents the lth frame, k represents the kth frequency point of the current frame, and stft() represents short-time Fourier transform;

[0049] The calculation formula for fractional delay is:

[0050]

[0051] Among them, exp(-2jπ·fs·d 0,1 / K) is the complex exponential function, j is the imaginary unit, and fs is the sampling frequency;

[0052] The same synchronized signal The calculation formula is:

[0053]

[0054] Among them, isft() represents short-time inverse Fourier transform;

[0055] It should be noted that according to the above steps, a strictly synchronized signal can be obtained. and The acquisition method and signal of these data The method of obtaining is the same.

[0056] Further, in the step 2, for the first signal that needs to be mixed synchronously processed, the channel gain equalization estimation module sets the corresponding channel gain to gChn=1, and for the first signal that does not need to be mixed synchronously processed, the channel gain corresponding to the first signal is set to gChn=0; the specific steps are as follows:

[0057] Find the first-order smoothed energy p of each channel signal after alignment m , the calculation formula is:

[0058]

[0059] Where n represents the discrete time sequence number and is a positive integer; β is the smoothing coefficient, p m is the smoothed energy of channel m, where m is a positive integer; gChn represents the gain of each channel.

[0060] Furthermore, according to the first-order smoothing energy p m As a result, get the channel idx with the current maximum energy max , the calculation formula is:

[0061] idx max = argmax(p m ), and set vadCh idx_max =1;

[0062] The energy of all other channels p idx_max and the maximum channel energy p m Compare, if |p idx_max -p mIf Thr_p, then set vadCh m = 1, otherwise set vadCh m = 0;

[0063] where idx max is the channel index number with the maximum energy; argmax() represents the index value when obtaining the maximum value; vadCh m is the vad flag of channel m; Thr_p is the energy interpolation threshold.

[0064] Furthermore, to prevent fluctuations, hangover processing is also performed:

[0065] if (vadCh m == 1);

[0066] hangover m = 50;

[0067] else;

[0068] hangover m = hangover m - 1;

[0069] where hangover m represents the hangover frame number of the m-th channel;

[0070] If the hangover m value is greater than 0, then set gChn = 1, otherwise set gChn = 0.

[0071] Furthermore, in the third step, the adaptive mixing module performs gain smoothing processing according to the channel gain calculation result in the channel gain equalization estimation module. The calculation formula is:

[0072] g m = γ * g m + (1 - γ) * gChm;

[0073] where γ is the smoothing constant, and its value range is between 0 and 1; g m refers to the gain of channel m; gChn represents the gain of each channel.

[0074] Furthermore, the calculation formula for delay addition and mixing processing is:

[0075] x mix (n) = g0x0(n) + g1x1(n) + g2x2(n) + g3x3(n)……;

[0076] where n represents the serial number of discrete time, and n is a positive integer; the mixed xmix (n) is the mixed signal after the adaptive array microphone cascade is finally mixed.

[0077] Beneficial effects of the present invention:

[0078] (1) First, the present invention can intelligently process signals from different microphone arrays and realize adaptive mixing processing through the coordinated work of the synchronization module, the channel gain equalization estimation module and the adaptive mixing module; it can also better balance the volume of multiple target speakers speaking at the same time in a cascade array scenario, avoid the situation where a certain speaker is suppressed, and ensure that the voices of speakers at different positions can obtain a more balanced volume output.

[0079] (2) Moreover, when the target speaker is at a similar relative distance from multiple microphone arrays, the present invention further enhances the target speech using a method similar to delay addition. This enhancement effect is particularly evident when the target speaker is relatively far away from all arrays.

[0080] (3) In addition, when the target speaker is one close to and one far from the cascade microphones, the present invention is able to identify this situation and give priority to using the signal from the close-distance microphone (because its quality is much better than that of the long-distance microphone), while discarding the input from the long-distance microphone, thereby ensuring the quality of the target speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 A flowchart of an adaptive array microphone cascade mixing method of the present invention;

[0082] Figure 2 This is a specific implementation flow chart of the synchronization module of the present invention. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0085] like Figure 1-2 As shown, this embodiment provides an adaptive array microphone cascade mixing method, which is applied to a microphone cascade mixing system, wherein the microphone cascade mixing system includes a synchronization module, a channel gain equalization estimation module and an adaptive mixing module, wherein the synchronization module is connected to the channel gain equalization estimation module through a plurality of channel one communications, the channel gain equalization estimation module is connected to the adaptive mixing module through a plurality of channel two communications, and the synchronization module is connected to the adaptive mixing module through a plurality of channel three communications;

[0086] Also includes a first microphone array device, a second microphone array device, ..., an Nth microphone array device, where N is a positive integer; wherein the first microphone array device is connected to the synchronization module via a Ch0 channel, the second microphone array device is connected to the synchronization module via a Ch1 channel, ..., the Nth microphone array device is connected to the synchronization module via a Ch(N-1) channel;

[0087] The following steps are involved:

[0088] Step 1: Each microphone array device transmits data to the synchronization module, and the synchronization module performs relative delay analysis and voice activation analysis on the data of each channel received to obtain analysis results; then, according to the analysis results, the data of each channel is subjected to secondary phase-level synchronization processing to obtain a first signal;

[0089] Step 2: The channel gain equalization estimation module first receives the first signal after the secondary phase level synchronization processing, and selects the first signal that needs to be mixed synchronously processed, and obtains the second signal after the mixing synchronously processed; in the selection process, the channel gain equalization estimation module analyzes several channels 1 that are currently close to the speaker according to the energy of each channel 1, and selects them as candidate channels, while discarding other channels 1; wherein, the first signal corresponding to the candidate channel needs to be mixed synchronously processed;

[0090] Step three: The adaptive mixing module receives the second signal processed by the channel gain equalization estimation module; the adaptive mixing module then smoothes the gain according to the gain obtained in the channel gain equalization estimation module of the second signal, and applies it to the signal of each aligned channel two, and then performs delayed addition and mixing processing, and finally obtains the mixed signal after the adaptive array microphone cascade mixing.

[0091] Firstly, the present embodiment can intelligently process signals from different microphone arrays and implement adaptive mixing processing through the coordinated work of the synchronization module, the channel gain equalization estimation module and the adaptive mixing module.

[0092] Secondly, this embodiment can better balance the volume of multiple target speakers speaking simultaneously in a cascade array scenario, avoid the situation where a certain speaker is suppressed, and ensure that the voices of speakers at different positions can obtain a more balanced volume output.

[0093] Moreover, when the target speaker is at a similar relative distance from multiple microphone arrays, this embodiment further enhances the target speech using a similar delay addition method. This enhancement effect is particularly obvious when the target speaker is relatively far from all arrays.

[0094] In addition, when the target speaker is one close to and one far from the cascade microphones, this embodiment can identify this situation and give priority to using the signal of the close microphone (because its quality is much better than that of the far microphone), while discarding the input of the far microphone, thereby ensuring the quality of the target speech.

[0095] Specifically, the synchronization module includes a coarse delay synchronization unit and a phase synchronization unit that are communicatively connected to each other; each of the microphone array devices is connected to the coarse delay synchronization unit through different channels, that is, the first microphone array device is connected to the coarse delay synchronization unit through the Ch0 channel, ..., the Nth microphone array device is connected to the coarse delay synchronization unit through the Ch(N-1) channel; the phase synchronization unit is communicatively connected to the channel gain equalization estimation module and the adaptive mixing module respectively;

[0096] More specifically, the phase synchronization unit is communicatively connected to the channel gain equalization estimation module via a plurality of channels two; and the phase synchronization unit is communicatively connected to the adaptive mixing module via a plurality of channels three.

[0097] Specifically, in step 1, the process of the secondary phase-level synchronization processing is as follows:

[0098] Select Ch0 channel as the reference channel for delay analysis;

[0099] The VAD algorithm is used to analyze whether the signal in the Ch0 channel is speech or noise; it should be noted that the VAD algorithm can be any optional existing algorithm and is not specifically limited in this embodiment;

[0100] When the VAD algorithm detects that the voice frame of a signal in a channel is 1, a delay analysis is performed:

[0101] Taking Ch0 and Ch1 as examples, assuming that x o And x1 do real Fourier transform, get The frequency domain signals of X0(ω) and X1(ω) at each point, the calculation formula of the frequency domain signal X0(ω) is:

[0102] X0(ω)=rfft(x0(nK / 2),K);

[0103] Among them, 0 represents the channel ID, n represents the voice frame, and n is a positive integer. For example, if a frame of data has 320 samples, n=0 represents all the data at 1:320 points, n=1 represents all the data at 321:640 points, and so on;

[0104] The calculation formula of the frequency domain signal X1(ω) is:

[0105] X1(ω)=rfft(x1(nK / 2),K);

[0106] Among them, x N represents the time domain signal of the Nth channel; K represents the number of points of the fast Fourier transform FFT; ω represents the frequency; rfft() is the real number Fourier transform;

[0107] Perform mutual coherence calculation on the frequency domain signals X0(ω) and X1(ω) to obtain the coherence coefficient G 0,1 , the mutual interference calculation formula is:

[0108]

[0109] Among them, α is the smoothing coefficient, * means taking the conjugate;

[0110] Then the coherence coefficient G 0,1 Do normalized weighting, the calculation formula is:

[0111]

[0112] Then Perform the inverse real Fourier transform of the length of MK to get R, and the calculation formula is:

[0113]

[0114] Among them, irfft() is the inverse real Fourier transform;

[0115] Finally, the calculation formula for the relative delay of the two signals is:

[0116] D 0,1 =argmax(R)-MK / 2;

[0117] Among them, argmax() represents the index value when the maximum value is obtained; M is the upsampling multiple, ranging from 4 to 6; D 0,1 Indicates the number of delayed samples between Ch0 and Ch1.

[0118] Specifically, after calculating the relative delay of each channel equivalent to the Ch0 channel, start performing secondary synchronization processing on each channel, and obtain the first signal after the secondary synchronization processing; wherein the secondary synchronization processing includes coarse synchronization processing and phase synchronization processing;

[0119] It should be noted that if only coarse synchronization is used in this embodiment, the delay synchronization granularity will be too large and inaccurate; if only phase synchronization is used, the delay between signals will be too large, resulting in spatial aliasing in the high-frequency part and signal distortion; therefore, this embodiment adopts a two-level synchronization process, that is, combining coarse synchronization and phase synchronization at the same time;

[0120] The process of rough synchronization is as follows:

[0121] For the signal excluding the Ch0 channel, the calculation formula is:

[0122]

[0123] in, It represents an integer not greater than this value; n represents the sequence number of discrete time, and n is a positive integer.

[0124] Specifically, the phase synchronization process is as follows:

[0125] For the signal excluding the Ch0 channel, the time of the Ch0 channel is aligned with a higher accuracy than the coarse synchronization process. The calculation formula is:

[0126]

[0127] Wherein, l represents the lth frame, k represents the kth frequency point of the current frame, and stft() represents short-time Fourier transform;

[0128] The calculation formula for fractional delay is:

[0129]

[0130] Among them, exp(-2jπ·fs·d0,1 / K) is the complex exponential function, j is the imaginary unit, and fs is the sampling frequency;

[0131] The signal after phase synchronization The calculation formula is:

[0132]

[0133] Among them, istft() represents the inverse short-time Fourier transform;

[0134] It should be noted that according to the above steps, a strictly synchronized signal can be obtained. The acquisition methods of signals such as and are the same as the acquisition method of the signal .

[0135] Specifically, in the second step, for the first signal that needs to be processed for mixing synchronization, the channel gain equalization estimation module sets its corresponding channel gain to gChn = 1, and for the channel gain corresponding to the first signal that does not need to be processed for mixing synchronization, it is set to gChn = 0; the specific steps are as follows:

[0136] Calculate the first-order smoothed energy p of each channel signal after alignment m , and the calculation formula is:

[0137]

[0138] Among them, n represents the serial number of discrete time, and n is a positive integer; β is the smoothing coefficient, p m is the smoothed energy of channel m, m is a positive integer; gChn represents the gain of each channel.

[0139] Specifically, according to the result of the calculated first-order smoothed energy p m , obtain the channel idx with the current maximum energy max , and the calculation formula is:

[0140] idx max = argmax(p m ), and at the same time set vadCh idx_max = 1;

[0141] Compare the energy p of all other channels idx_max with the obtained maximum channel energy p m . If |p idx_max - p m | < Thr_p, then set vadCh m = 1, otherwise set vadCh m = 0;

[0142] Among them, idx max is the channel index with the largest energy; argmax() indicates the index value when the maximum value is obtained; vadCh m is the vad flag of channel m; Thr_p is the energy interpolation threshold.

[0143] Specifically, in order to prevent fluctuations, hangover processing is also performed:

[0144] if(vadCh m = =1);

[0145] hangover m =50;

[0146] else;

[0147] hangover m =hangover m -1;

[0148] Among them, hangover m Indicates the number of hangover frames of the mth channel;

[0149] If hangover m If the value is greater than 0, set gChn=1, otherwise set gChn=0.

[0150] Specifically, in step 3, the adaptive mixing module performs gain smoothing processing according to the channel gain calculation result in the channel gain equalization estimation module, and the calculation formula is:

[0151] g m =γ*g m +(1-γ)*gChm;

[0152] Among them, γ is a smoothing constant, ranging from 0 to 1; g m refers to the gain of channel m; gChn represents the gain of each channel.

[0153] Specifically, the calculation formula for delay addition and mixing processing is:

[0154] x mix (n)=g0x0(n)+g1x1(n)+g2x2(n)+g3x3(n)……;

[0155] Where n represents the sequence number of discrete time, and n is a positive integer; the mixed x mix (n) is the mixed signal after the adaptive array microphone cascade is finally mixed.

[0156] In this embodiment, the above-mentioned adaptive array microphone cascade mixing method enables multiple microphone arrays to aggregate array outputs close to the speaker after cascading, enhance the speaker's voice, and use the outputs of different microphone arrays to repair the output defects of a single array, thereby improving the overall voice quality after mixing. Moreover, while maintaining the above advantages, this embodiment excludes the array outputs at a relatively long distance to prevent the mixing of array outputs with relatively poor quality, thereby avoiding the deterioration of the overall voice quality after mixing.

[0157] In summary, this embodiment achieves multiple beneficial effects in the cascade array microphone scenario, such as volume equalization, target speech enhancement, signal quality optimization, adaptive mixing processing, output defect repair, and low-quality output elimination, which significantly improves the overall speech processing performance and user experience.

[0158] According to the disclosure and teaching of the above description, those skilled in the art to which the present invention belongs may also make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the present invention should also fall within the scope of protection of the claims of the present invention. In addition, although some specific terms are used in this specification, these terms are only for the convenience of description and do not constitute any limitation to the present invention.

Claims

1. An adaptive array microphone cascade mixing method, applied to a microphone cascade mixing system, the microphone cascade mixing system comprising a synchronization module, a channel gain equalization estimation module and an adaptive mixing module, the synchronization module and the channel gain equalization estimation module are connected to each other through a plurality of channels one, the channel gain equalization estimation module and the adaptive mixing module are connected to each other through a plurality of channels two, and the synchronization module and the adaptive mixing module are connected to each other through a plurality of channels three; It also includes a first microphone array device, a second microphone array device, ..., an Nth microphone array device, where N is a positive integer; wherein, The first microphone array device is connected to the synchronization module via the Ch0 channel, the second microphone array device is connected to the synchronization module via the Ch1 channel, ..., the Nth microphone array device is connected to the synchronization module via the Ch(N-1) channel; It is characterized in that The following steps are involved: Step 1: Each microphone array device transmits data to the synchronization module, and the synchronization module performs relative delay analysis and voice activation analysis on the data of each channel received to obtain analysis results; then, according to the analysis results, the data of each channel is subjected to secondary phase-level synchronization processing to obtain a first signal; Step 2: The channel gain equalization estimation module first receives the first signal after the secondary phase level synchronization processing, and selects the first signal that needs to be mixed synchronously processed, and obtains the second signal after the mixing synchronously processed; in the selection process, the channel gain equalization estimation module analyzes several channels 1 that are currently close to the speaker according to the energy in each channel 1, and selects them as candidate channels, while discarding other channels 1; wherein, the first signal corresponding to the candidate channel needs to be mixed synchronously processed; Step three: The adaptive mixing module first receives the second signal processed by the channel gain equalization estimation module; the adaptive mixing module then smoothes the gain according to the gain obtained in the channel gain equalization estimation module of the second signal, and applies it to the signal of each aligned channel two, and then performs delayed addition and mixing processing, and finally obtains the mixed signal after the adaptive array microphone cascade mixing.

2. The adaptive array microphone cascade mixing method according to claim 1, characterized in that: The synchronization module includes a coarse delay synchronization unit and a phase synchronization unit that are communicatively connected to each other; each of the microphone array devices is connected to the coarse delay synchronization unit via a different channel, that is, the first microphone array device is connected to the coarse delay synchronization unit via the Ch0 channel, ..., the Nth microphone array device is connected to the coarse delay synchronization unit via the Ch(N-1) channel.

3. The adaptive array microphone cascade mixing method according to claim 2, characterized in that: In step 1, the process of the secondary phase level synchronization is as follows: Select Ch0 channel as the reference channel for delay analysis; The VAD algorithm is used to analyze whether the signal in the Ch0 channel is speech or noise; When the VAD algorithm detects that the voice frame of a signal in a channel is 1, a delay analysis is performed: Taking Ch0 and Ch1 as examples, assuming that x o And x1 do real Fourier transform, get The frequency domain signals of X0(ω) and X1(ω) at each point, the calculation formula of the frequency domain signal X0(ω) is: X0(ω)=rfft(x0(nK / 2),K); Among them, 0 represents the channel ID, n represents the voice frame, and n is a positive integer; The calculation formula of the frequency domain signal X1(ω) is: X1(ω)=rfft(x1(nK / 2),K); Among them, x N represents the time domain signal of the Nth channel; K represents the number of points of the fast Fourier transform FFT; ω represents the frequency; rfft() is the real number Fourier transform; Perform mutual coherence calculation on the frequency domain signals X0(ω) and X1(ω) to obtain the coherence coefficient G 0,1 , the mutual interference calculation formula is: Among them, α is the smoothing coefficient, * means taking the conjugate; Then the coherence coefficient G 0,1 Do normalized weighting, the calculation formula is: Then Perform the inverse real Fourier transform of the length of MK to get R, and the calculation formula is: Among them, irfft() is the inverse real Fourier transform; Finally, the calculation formula for the relative delay of the two signals is: D 0,1 =argmax(R)-MK / 2; Among them, argmax() represents the index value when the maximum value is obtained; M is the upsampling multiple, ranging from 4 to 6; D 0,1 Indicates the number of delayed samples between Ch0 and Ch1.

4. The adaptive array microphone cascade mixing method according to claim 3, characterized in that: After calculating the relative delay of each channel equivalent to the Ch0 channel, start performing secondary synchronization processing on each channel, and obtain the first signal after the secondary synchronization processing; wherein the secondary synchronization processing includes coarse synchronization processing and phase synchronization processing; The process of rough synchronization is as follows: For the signal excluding the Ch0 channel, the calculation formula is: in, It represents an integer not greater than this value; n represents the sequence number of discrete time, and n is a positive integer.

5. The adaptive array microphone cascade mixing method according to claim 4, characterized in that: The phase synchronization process is as follows: For the signal excluding the Ch0 channel, the time of the Ch0 channel is aligned with a higher accuracy than the coarse synchronization process. The calculation formula is: Wherein, l represents the lth frame, k represents the kth frequency point of the current frame, and stft() represents short-time Fourier transform; The calculation formula for fractional delay is: Among them, exp(-2jπ·fs·d 0,1 / K) is the complex exponential function, j is the imaginary unit, and fs is the sampling frequency; Phase-synchronized signal The calculation formula is: Among them, isft() represents short-time inverse Fourier transform.

6. The adaptive array microphone cascade mixing method according to claim 1, characterized in that: In the step 2, for the first signal that needs to be mixed synchronously processed, the channel gain equalization estimation module sets the corresponding channel gain to gChn=1, and for the first signal that does not need to be mixed synchronously processed, the channel gain corresponding to the first signal is set to gChn=0; the specific steps are as follows: Find the first-order smoothed energy p of each channel signal after alignment m , the calculation formula is: Where n represents the discrete time sequence number and is a positive integer; β is the smoothing coefficient, p m is the smoothed energy of channel m, where m is a positive integer; gChn represents the gain of each channel.

7. The adaptive array microphone cascade mixing method according to claim 6, characterized in that: According to the first-order smoothed energy p m As a result, get the channel idx with the current maximum energy max , the calculation formula is: idx max = argmax(p m ), and set vadCh idx_max =1; The energy p of all other channels idx_max is compared with the obtained maximum channel energy p m such that if |p idx_max - p m | < Thr_p, then set vadCh m = 1, otherwise set vadCh m = 0; Among them, idx max is the channel index with the largest energy; argmax() indicates the index value when the maximum value is obtained; vadCh m is the vad flag of channel m; Thr_p is the energy interpolation threshold.

8. The adaptive array microphone cascade mixing method according to claim 7, characterized in that: In order to prevent fluctuations, hangover processing is also performed: if(vadCh m ==1); hangover m =50; else; hangover m =hangover m -1; Among them, hangover m Indicates the number of hangover frames of the mth channel; If hangover m If the value is greater than 0, set gChn=1, otherwise set gChn=0.

9. The adaptive array microphone cascade mixing method according to claim 1, characterized in that: In the step 3, the adaptive mixing module performs gain smoothing processing according to the channel gain calculation result in the channel gain equalization estimation module, and the calculation formula is: g m =γ*g m +(1-γ)*gChm; Among them, γ is a smoothing constant, ranging from 0 to 1; g m refers to the gain of channel m; gChn represents the gain of each channel.

10. The adaptive array microphone cascade mixing method according to claim 9, characterized in that: The calculation formula for delay addition and mixing processing is: x mix (n)=g0x0(n)+g1x1(n)+g2x2(n)+g3x3(n)……; Where n represents the sequence number of discrete time, and n is a positive integer; the mixed x mix (n) is the mixed signal after the adaptive array microphone cascade is finally mixed.

Images