A microphone array audio processing method and sound pickup badge

Abstract

The present invention discloses a microphone array audio processing method and a sound pickup badge. The threshold value of the current frame is determined according to the threshold value of each frame from the previous frame to the previous e frames of the current frame of the mixed voice signal, and a comparison is made based on the determined threshold value. When it is greater than the threshold value, it is a non-silent frame. Except for the initial e frames, the threshold value of each frame of the audio is dynamically calculated according to the aforementioned steps, and each frame of the audio signal is compared with the threshold value to obtain all non-silent voice signals. The threshold value is set by moving the sliding window to adapt to the scene of the actual microphone collecting audio, and the rationality of the threshold value is ensured to avoid inaccurate detection caused by the unified threshold value; then, the detected non-silent voice signal is subjected to speaker spatial orientation, multi-beam noise reduction and enhancement processing to obtain better audio of different speakers.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic products, and in particular to a microphone array audio processing method and a sound pickup badge. Background Art

[0002] In the actual microphone recording process, there will be some silent segments, such as a few minutes before the teacher's class, a break in the middle, the start, end, and intermission of recording in the conference room, etc. These situations will cause the audio recorded by the microphone to have a certain amount of silence. These silent audios only contain background noise and do not have any actual useful information.

[0003] Based on this, in the process of microphone audio signal processing, the Voice Activity Detection (VAD) algorithm is usually used to record non-human voice segments (silent parts) and human voice segments in the audio. The purpose is to identify and eliminate long silent periods from the sound signal.

[0004] The criteria for determining whether there is human voice in audio can be divided into three categories: one is based on threshold; the second is based on statistical model; and the third is based on deep learning.

[0005] In the threshold-based discrimination criterion, by setting the threshold value, speech greater than the threshold value is considered to be human voice. That is, under the assumption that the speech energy is greater than the background noise energy, when the audio energy is greater than a certain threshold, it is considered that there may be human voice. The determination of the threshold value is related to the actual recording scene. When the noise in the scene (that is, the background noise in the environment) is constantly changing, the threshold value must also change accordingly to ensure the accuracy of the detection. Therefore, it is necessary to change the threshold value of speech detection according to the actual audio acquisition situation.

[0006] Based on this, it is necessary to determine a suitable threshold value when performing VAD processing on the microphone array audio. Only when the VAD cuts the audio accurately can the accuracy of subsequent audio processing be guaranteed. Summary of the invention

[0007] The object of the present invention is to provide a method for processing microphone array audio and a sound pickup badge to ensure the accuracy of audio detection when VAD processing is performed on the microphone audio.

[0008] In order to solve the above technical problems, the present invention provides a method for processing microphone array audio, comprising:

[0009] S1: Obtain the mixed speech signal collected by the microphone array;

[0010] S2: Determine the threshold value of the current frame by weighting the threshold value of each frame in the previous e frames of the current frame of the mixed speech signal;

[0011] S3: performing speech detection on the speech signal of the current frame according to the threshold value of the current frame, and when it is detected that the speech signal of the current frame is greater than the threshold value of the current frame, determining that the current frame is a non-silent frame, and temporarily storing the threshold value of the current frame;

[0012] S4: Move the language frame backward, use the method described in S2 to S3 to predict the threshold value of each frame and perform speech detection on the speech signal of each frame, and regard the speech signals of all frames greater than the threshold value of each frame as non-silent speech signals;

[0013] S5: performing speaker spatial positioning on the non-silent speech signal to obtain steering vectors of different speakers;

[0014] S6: generating multiple beams according to the steering vectors of the different speakers, picking up the speech signal on each beam, and realizing speech signal enhancement of the speaker;

[0015] S7: Perform noise reduction processing on the enhanced speaker speech signal, and output the noise-reduced speech signal on each beam.

[0016] Preferably, the threshold value of the current frame is determined by weighting the threshold value of each frame before the current frame of the mixed speech signal as follows:

[0017] η i =w i-1 η i-1 +w i-2 η i-2 +…+w i-e η i-e

[0018] Where η i is the threshold value of the current frame, that is, the threshold value of the i-th frame audio signal in the mixed speech signal, (η i-1 , η i-2 ,…,η i-e ) is the threshold value of each audio signal from the previous frame to the previous e frames of the current frame, that is, the threshold value of the audio signal of the (i-1, i-2, ..., ie)th frame in the mixed speech signal, (w i-1 , w i-2 ,…,w i-e ) are respectively the weight values ​​of the audio signal threshold values ​​of each frame from the previous frame to the previous e frames of the current frame, that is, the weight values ​​of the audio signal threshold values ​​of the (i-1, i-2, ..., ie)th frames in the mixed speech signal, wherein i≥e+1, e represents the size of the sliding window during speech detection.

[0019] Preferably, the step of determining the size of the sliding window during speech detection includes:

[0020] Using sliding windows of different sizes to maximize the accuracy of speech detection based on the H∞ criterion:

[0021]

[0022] In the formula, A target is the standard audio in the basic test set, A correct is the audio after speech detection, Represents the accuracy of speech detection;

[0023] The corresponding relationship between the speech detection accuracy and the sliding windows of different sizes is obtained, and the minimum value in the sliding window corresponding to the speech detection accuracy meeting the set accuracy threshold is selected as the size of the sliding window during speech detection.

[0024] Preferably, the threshold values ​​of the first frame to the nth frame of the mixed speech signal are respectively the energy mean values ​​of the audio signals of each frame.

[0025] Preferably, performing speech detection on the speech signal of the current frame according to the threshold value of the current frame includes:

[0026] The mixed speech signal collected by the microphone array is:

[0027]

[0028] In the formula, x m (t) is the audio signal collected by the microphone array, is the response function of microphone m and sound source c, D represents the number of sound sources, S c (t) is the output signal of the sound source c to the mth microphone, n m (t) is the noise signal of the mth microphone, and the total number of microphones is M;

[0029] Using threshold value for speech detection:

[0030]

[0031] Where N N 、N S Represent the noise vector matrix and speech vector matrix respectively, X k，i represents the speech signal with the ith frequency point index in the kth frame audio spectrum in the mixed speech signal, L represents the number of frequency points in the kth frame audio spectrum, and η is the threshold value of the kth frame.

[0032] Preferably, the speaker spatial positioning is performed on the non-silent speech signal to obtain the steering vectors of different speakers:

[0033]

[0034] In the formula, is the steering vector of the determined speaker, N M-D is the residual matrix composed of noise feature vectors, the subscript sr represents the number of peak searches, i.e. the number of speakers selected, and θ is the azimuth of the spherical coordinate system. is the elevation angle in spherical coordinates, For The probability of the speaker at .

[0035] Preferably, generating multiple beams according to the steering vectors of the different speakers, picking up the speech signal on each beam, and realizing speech signal enhancement of the speaker includes:

[0036]

[0037] Where, X sr (t, f) is the Fourier transformed signal of the input speaker’s speech signal, Y sr (t, f) is the Fourier transform output signal of the speaker after azimuth enhancement, that is, the enhanced speaker's speech signal, For The noise variance matrix at θ sr is the speaker's azimuth in the spherical coordinate system, is the elevation angle of the speaker in the spherical coordinate system.

[0038] Preferably, the noise reduction processing on the enhanced speaker speech signal comprises: using a convolution non-negative matrix to reduce the noise of the enhanced speech audio:

[0039]

[0040] Where D psa is the constructed loss function, H sr (t, f) is the speaker coefficient matrix, Y sr (t, f) is the enhanced speech audio, is the speaker's denoised speech signal, and F represents the norm;

[0041] Through the gradient search algorithm, the loss function D psa The difference is less than the set difference, and the best speaker coefficient matrix H is obtained. sr_opt (t, f);

[0042] According to the optimal speaker coefficient matrix H sr_opt (t, f) generates the final denoised speech signal:

[0043]

[0044] Preferably, before outputting the speech signal after noise reduction processing on each beam, the method further includes:

[0045] Convert the denoised speaker's speech signal from the frequency domain to the time domain:

[0046]

[0047] In the formula, rSTFT represents the inverse short-time Fourier transform, is the frequency domain signal of the speaker’s speech signal after noise reduction, is the time domain signal of the speaker's speech signal after noise reduction;

[0048] The AGC method is used to enhance the time domain signal:

[0049]

[0050] Where g(n) is the speaker's speech signal after AGC enhancement, A(n) is the dynamic gain coefficient, is the gain input, i.e., the time domain signal of the speaker's speech signal after noise reduction;

[0051] Wherein, the dynamic gain coefficient is determined according to the target gain amplitude:

[0052]

[0053] In the formula, a is the gain adjustment factor, R is the target gain amplitude, and the target gain amplitude is no greater than -3dB.

[0054] The present invention also provides a microphone array sound pickup badge, comprising:

[0055] Signal receiving module: used to pick up audio using the microphone array in the badge;

[0056] Audio processing module: adopting the steps of the microphone array audio processing method as described above to realize the processing of the audio picked up by the microphone array;

[0057] Signal transmission module: used to upload the audio processed by the audio processing module;

[0058] The data storage module is used to store the audio processed by the audio processing module.

[0059] Preferably, it also includes a wake-up module for realizing the sound pickup control of the microphone array sound pickup badge, and when the collected voice contains a preset wake-up word, the microphone array sound pickup badge is controlled to start or end sound pickup.

[0060] The microphone array audio processing method and sound pickup badge provided by the present invention, when performing speech detection on the mixed speech signal acquired by the microphone, determine the threshold value of each frame according to the collected mixed speech signal, and use the threshold value of each frame from the previous frame to the previous e frames of the current frame to determine the threshold value of the current frame. The sliding window is used to move the threshold value to set the threshold value, so as to adapt to the actual microphone audio collection scene, and ensure the rationality of the threshold value, so as to avoid the inaccurate speech detection caused by the unified threshold value, and the calculation of the threshold value is also simple. Speech detection is performed according to the determined threshold value. When the current frame is detected as a non-silent frame, the current frame threshold value is temporarily stored, and the signal of each subsequent frame uses the weighted threshold value of the previous e frames for speech detection to obtain all non-silent speech signals; then, the detected non-silent speech signal is subjected to speaker spatial orientation, multi-beam noise reduction and enhancement processing to obtain better audio of different speakers. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 A flowchart of a method for processing microphone array audio provided by the present invention;

[0063] Figure 2 A flowchart of a first specific embodiment of the microphone array audio processing method provided by the present invention;

[0064] Figure 3 Graph showing the relationship between VAD cutting accuracy and sliding window in an embodiment of the present invention;

[0065] Figure 4 This is an example diagram of multi-channel audio collected by a microphone array in an embodiment of the present invention;

[0066] Figure 5 This is an example diagram of audio containing human voice retained after VAD detection in an embodiment of the present invention;

[0067] Figure 6 is a modeling diagram of spatial spectrum estimation centered on a device for receiving audio in an embodiment of the present invention;

[0068] Figure 7 is a schematic diagram of beam forming in an embodiment of the present invention;

[0069] Figure 8 This is an example diagram of two-channel audio generated by MVDR in an embodiment of the present invention;

[0070] Fig. 9 This is an example diagram of two-channel audio after noise reduction by a neural network in an embodiment of the present invention;

[0071] Fig.10 This is an example diagram of a clear human voice output after inverse Fourier transform processing in an embodiment of the present invention;

[0072] Fig.11 This is a structural diagram of a specific embodiment of the intelligent sound pickup badge proposed by the present invention;

[0073] Fig.12 This is a diagram showing the software and hardware structure of the smart sound pickup badge in an embodiment of the present invention;

[0074] Fig.13 The figure is a flow chart of processing of collected signals by the intelligent sound pickup badge in an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The core of the present invention is to provide a microphone array audio processing method and a sound pickup badge. It is mainly used to pick up the voice audio of teachers and students during teaching in campus churches to achieve audio amplification of teachers and students, meet the interaction between teachers and students in actual teaching, and facilitate teachers to determine students' mastery of teaching content, so as to better achieve teaching results.

[0076] The present invention provides a method for processing microphone array audio, please refer to Figure 1 , Figure 1 A specific flow chart of the microphone array audio processing method provided by the present invention;

[0077] First, VAD (Voice Activity Detection) speech detection is performed on the obtained mixed speech signal. The purpose of VAD detection is to separate pure noise. The present invention performs VAD detection based on threshold values. The threshold values ​​of the 1st frame to the eth frame of the mixed speech signal are respectively the energy mean values ​​of each frame signal. Then, the threshold value of each frame signal is determined according to the threshold values ​​of the previous 1 frame to the previous e frames of the current frame signal, based on which the non-silent speech signal is detected.

[0078] Then the speaker space positioning is performed on the non-silent speech signal to obtain the steering vectors of different speakers;

[0079] Generate multiple beams based on the steering vectors of different speakers, pick up the speech signal on each beam, and enhance the speaker's speech signal;

[0080] The enhanced speaker speech signal is subjected to noise reduction processing, and the noise-reduced speech signal on each beam is output.

[0081] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0082] Please refer to Figure 2 , Figure 2 This is a flowchart of the first specific embodiment of a microphone array audio processing method provided by the present invention; taking a teacher teaching in a large classroom as an example, a microphone array device worn by the teacher is used to pick up voices. Affected by the external environment, the collected mixed voice signals include different human voices and external environmental noises (noise); wherein the layout of the microphone array can be a one-dimensional array, a two-dimensional array, or a three-dimensional array, the arrangement of the microphones can be a uniform array or a spiral array, and the number and distribution rules of the microphones can be determined according to the actual size of the pickup device. Taking the example of picking up two-way audio of the teacher and the student in the classroom as an example, it is specifically explained.

[0083] This embodiment first uses a microphone array to pick up audio; performs VAD preprocessing on the picked up audio; performs speaker localization on the preprocessed audio to determine the speaker's angle; generates a beam based on the speaker vector, picks up the speaker's audio on the beam, and realizes directional sound pickup; performs noise reduction on the picked up audio to obtain pure speech estimation; converts the obtained pure speech into a time domain signal; and finally further enhances the time domain signal to output clear human voices of different speakers. The specific process is as follows:

[0084] S1: The microphone collects the original audio and obtains the audio data picked up by each microphone;

[0085] At this time, the mixed speech signal collected by the microphone array can be expressed as:

[0086]

[0087] In the formula, x m (t) is the audio signal collected by the microphone array, is the response function of microphone m and sound source c, D represents the number of sound sources, S c (t) is the output signal of the sound source c to the mth microphone, n m (t) is the noise signal of the mth microphone.

[0088] S2: Perform VAD speech detection on the mixed speech signal to remove noise and silence;

[0089] Since the originally collected audio is affected by the external environment and is mixed with external noise, this embodiment adopts the VAD audio detection method. The purpose of VAD audio detection is mainly to identify and eliminate long periods of silence from the picked-up voice signal. It can also be understood that this method is to determine whether there is voice (human voice), that is, to separate the noisy voice and the audio containing only noise. VAD is also called voice activity detection. Using VAD to detect effective voice can reduce the amount of data processing and the amount of data stored or transmitted. In this embodiment, the set adaptive threshold is used for judgment, which is as follows:

[0090]

[0091]

[0092] Where N N 、N S Represent the noise vector and speech vector matrix respectively, W S , W N are the speech basis matrix and the noise basis matrix set respectively, T is the number of matrices contained in the basis matrix set; X k，i The signal representing the index of the ith frequency point in the kth frame audio spectrum, L represents the number of frequency points in the kth audio spectrum, and η is an adaptive threshold defined based on the sampled audio;

[0093] The determination of the threshold value is related to the scenario in which the audio is acquired. When the noise in the actual recording environment is constantly changing, the threshold value also needs to change accordingly to ensure the accuracy of the detection. Therefore, the threshold value should be determined based on the actual recorded audio.

[0094] The process of determining the threshold value is as follows:

[0095] η=f(E num , f ps )

[0096] Where η is the threshold value, E num , f ps The audio energy mean is obtained according to the target scene. After the initial frame, a sliding window is used to iterate the threshold value. The sliding window width is e frames. η corresponds to the adaptive threshold of the non-silent frame moving forward by e frames. The threshold value can be updated only when silent audio is detected. The corresponding η values ​​are {η e , η e-1 ,…,η1}, the vector {η e , η e-1 ,…,η1} is recorded as Π, and the initial Π0 takes the energy mean of the previous e frames.

[0097] That is to say, the threshold value of the mixed speech signal from the first frame to the eth frame is the energy average of each frame signal in the first e frames of the mixed speech signal; generally, the first few frames of speech signals are assumed to be silent;

[0098] A sliding window (the width of the sliding window is e) is used to set the threshold value η of each frame signal after the e-th frame of the mixed speech signal. i Determine, calculate the threshold value of the current frame by the threshold value of each frame from the previous frame to the previous e frames of the current frame:

[0099] η i =w i-1 η i-1 +w i-2 η i-2 +…+w i-e η i-e

[0100] Where η i is the threshold value of the current frame, that is, the threshold value of the i-th frame audio signal in the mixed speech signal, (η i-1 , η i-2 ,…,η i-e ) is the threshold value of the audio signal of each frame from the previous frame to the previous e frames of the current frame, that is, the threshold value of the audio signal of the (i-1, i-2, ..., ie)th frame in the mixed speech signal, (w i-1 , w i-2 ,…,w i-e ) are respectively the weight values ​​of the audio signal threshold values ​​of each frame from the previous frame to the previous e frames of the current frame, that is, the weight values ​​of the audio signal threshold values ​​of the (i-1, i-2, ..., ie)th frames in the mixed speech signal, where i≥e+1, and e represents the size of the sliding window.

[0101] Among them, the corresponding weight matrix W = (w i-1 , w i-2 ,…,w i-e ) is pre-set with a training set before being applied in each scenario.

[0102] The width of the sliding window is based on H ∞ Criteria and basic test set determination, H ∞ The objective function of the criterion optimization is the speaker segmentation accuracy. The ∏ and e values ​​are determined in the real number domain through gradient search, and the objective function is maximized to achieve the highest speaker segmentation accuracy.

[0103]

[0104] In the formula, A target The standard audio for the basic test set, A correct Cut audio for VAD. ∞Involving matrix budget, the larger the e, the lower the audio insertion error, which also means an increase in computing power. A small value should be taken within the acceptable range of cutting accuracy. Taking ∏ and e as independent variables, and speaking cutting accuracy (speech detection accuracy) as the target variable, the relationship between cutting accuracy and parameter e in this embodiment is as follows: Figure 3 As shown, in the embodiment of the present invention, e is 5. Due to different recording scenes, the value of e can be determined according to the recording conditions of the actual microphone device. In this embodiment, the corresponding weight matrix W is:

[0105] W=[0.414, 0.350, 0.122, 0.059, 0.055]

[0106] The corresponding value of η is:

[0107] η i =0.414η i-1 +0.350η i-2 +0.122η i-3 +0.059η i-4 +0.055η i-5

[0108] Then, based on the calculated signal-to-noise ratio and the changed threshold value, the human voice is retained to obtain an audio signal with speech, that is, a non-silent speech signal; as another implementation, when performing VAD detection, only one of the audio channels can be selected for detection, which can save computing time and memory. There are several audio channels as many as there are microphones. VAD detection can be performed based on only one audio channel, so that non-silent segments can be extracted, and non-silent segments of each channel can be determined based on the non-silent segments determined by one audio channel; you can refer to Figure 4 and Figure 5 As shown, Figure 4 The multi-channel audio is collected by the microphone array. In the figure, 6 audios are collected, corresponding to 6 microphones. Figure 5 The audio containing human voice is retained after VAD cutting. After VAD processing, long periods of silence and pure noise are removed. Any of the 6 audio channels is used as the preprocessing object for VAD detection. The time period of non-silent audio is determined according to the detection result. The non-silent audio in each mixed voice audio is extracted according to the determined time period.

[0109] S3: Select the speaker's perspective;

[0110] Because noise reduction needs to be performed based on the speaker's angle later, that is, only the audio from the specified direction is picked up and the audio from non-specified directions is ignored;

[0111] In this embodiment, the audio output greater than η in the above step S2 is considered to contain speech. Since there is more than one speaker in the actual application scenario (for example, in a classroom or a large conference room), it is necessary to determine the position of the speaker. In this embodiment, an improved two-dimensional spatial spectrum estimation is used to estimate the direction of arrival of the spatial signal to achieve the spatial orientation of the speaker, as follows:

[0112]

[0113] In the formula, is the steering vector of the determined speaker, is the residual matrix composed of noise feature vectors, the subscript sr represents the peak search number. In this embodiment, sr refers to the number of people to be selected (that is, the number of speakers), θ is the azimuth of the spherical coordinate system, is the elevation angle in spherical coordinates, For According to the number of speakers, the number of speakers in the classroom scene is set to 2, and the calculated The first two largest The speaker steering vector is determined. The number of speakers can be set arbitrarily according to different scenarios.

[0114] Among them, each frame (fps) data has a corresponding Frequent updates of fps can cause sound jitter and distortion, so Smoothing is performed according to the following rules:

[0115]

[0116] Among them, β is the smoothing parameter, which ranges from 0.6 to 0.8.

[0117] Please refer to Figure 6 , Figure 6 The modeling diagram of spatial spectrum estimation centered on the device receiving audio is shown in , where θ is the azimuth angle of the spherical coordinate system, is the elevation angle of the spherical coordinate system, and the speaker orientation is determined according to the preset number of speakers; through the speaker orientation, it is determined whether the speaker is a wearer or a non-wearer at this time. The judgment criteria in this embodiment are:

[0118] (1) Wearer: For the wearer, such as When there are multiple speakers, the speaker with the highest energy is designated as the wearer, and the others are classified as non-wearers. When the microphone picks up the wearer's audio, it picks up the audio above the microphone.

[0119] (2) Non-wearer: All search results are sorted by energy. Except for the wearer pointing vector, all others are non-wearers. All audio except the wearer is considered as non-wearer. By distinguishing the wearer from the non-wearer, the speaker of the audio can be distinguished.

[0120] For example, when the preset number of speakers is 2, the speaker orientation is used to determine whether the speaker is the wearer or the non-wearer. Since the wearer is very close and the non-wearer is relatively far away from the wearer, the wearing position is generally located below the wearer's head. When the wearer speaks, the microphone receives the upper voice, and when the non-wearer speaks, the microphone receives the front voice.

[0121] S4: Qualitative enhancement based on angle, that is, only picking up audio from the specified direction and ignoring audio from non-specified directions;

[0122] Please refer to Figure 7 , Figure 7 The figure is a beamforming diagram of the microphone badge worn by the teacher as the recording device in this embodiment. According to the speaker steering vector determined in step S3, the MVDR beamforming method is used. When it is determined that the speaker is in front of the wearer (not the wearer), a forward azimuth beam is formed to carry out directional sound pickup in the forward direction. When it is determined that the speaker is the wearer, an upward azimuth beam is formed to carry out directional sound pickup of the wearer to obtain enhanced speaker audio:

[0123]

[0124] Where, X sr (t, f) is the SFT transform of the input speech signal, Y sr (t, f) SFT output signal after beam orientation enhancement, exist The noise variance matrix at .

[0125] S5: noise reduction on the enhanced audio;

[0126] The above steps only pick up the audio in the specified direction. The noise is usually a diffuse sound field. The audio picked up in the specified direction is a mixture of noise and human voice. Therefore, secondary noise reduction is required to remove the noise. In this embodiment, two beams are divided into two channels for audio collection (teacher and student), and the two channels of audio collected are noise-reduced respectively. This embodiment adopts the sparse convolution non-negative matrix decomposition (SCNMF) algorithm to remove background noise and mutation noise. When the beam picks up sound, when the speaker is spatially oriented to the wearer (ie, teacher), an upward beam is formed to pick up the teacher's audio in the upward beam. When the speaker is spatially oriented to a non-wearer (ie, student), a forward beam is formed to pick up the student's audio in the forward beam.

[0127] Furthermore, the Kullback-Leibler (KL) loss function is constructed based on the SCNMF algorithm to perform speaker differentiation;

[0128] The enhanced speech audio is denoised using a convolution non-negative matrix: The speech extraction KL function in this embodiment is:

[0129]

[0130] Where D psa is the constructed loss function, H sr (t, f) is the speaker coefficient matrix, Y sr (t, f) is the enhanced speech audio, is the speaker's denoised speech signal, and F represents the norm;

[0131] Through the gradient search algorithm, the loss function D psa The difference is less than the set difference, and the best speaker coefficient matrix H is obtained. sr_opt (t, f);

[0132] For example Figure 8 As shown, Figure 8 A total of two audio channels are displayed, one for the wearer and the other for the conversation. CH1 is the teacher's audio and CH2 is the student's audio. It can be seen that the peak value of CH1 audio is much larger than the peak value of CH2 audio. This is because the teacher is closer to the microphone than the student.

[0133] According to the optimal speaker coefficient matrix H sr_opt (t, f) generates the final pure speech signal after noise reduction

[0134]

[0135] For example Fig. 9 As shown, Fig. 9 The teacher and student speech signals obtained by removing the noise signal are shown for comparison. Figure 8 , it can be seen Fig. 9 The noise of the two audio channels is significantly reduced, and it can be seen that the peak values ​​of the student's audio (CH2) and the teacher's audio (CH1) are similar.

[0136] S6: Convert the spectrum into time domain signal;

[0137] The pure speech audio obtained in step S5 is subjected to an inverse short-time Fourier transform to convert the frequency domain signal of the pure speech into a time domain signal output, and the following is obtained:

[0138]

[0139] In the formula, rSTFT represents the inverse short-time Fourier transform, is the frequency domain signal of the speech signal under each beam, It is the time domain signal of the human voice signal finally extracted.

[0140] S7: adjust the gain of the time domain signal to ensure the consistency of the far-field and near-field speech amplitudes;

[0141] Sound waves will attenuate during transmission, and the amplitude of far-field speech will be smaller than that of near-field speech.

[0142]

[0143] Where g(n) is the audio sequence after gain adjustment, A(n) is the dynamic gain coefficient, is the gain input, i.e. the denoised audio sequence. The gain iteration is as follows:

[0144]

[0145] In the formula, a is the gain adjustment shadow. The larger a is, the faster the convergence speed is, but the jitter of the audio will increase with the mean R. R is the target gain amplitude, and the maximum gain is usually -3dB.

[0146] The above steps generate the final vocal output of two-way audio, such as Fig.10 As shown, Fig.10 Showing the clear vocals of both groups in the final output.

[0147] The treatment effect of the present invention is further illustrated by the following experiments;

[0148] 200 human voices from the TIMIT dataset were randomly selected, mixed with white noise from the Noisex-92 noise test set, and the signal-to-distortion ratio (SDR), signal-to-interference ratio (SIR), and system error ratio (SAR) perceived speech quality (PESQ) were introduced to compare the effects of VAD on separation accuracy, subjective evaluation, and power consumption under different signal-to-noise ratios. The evaluation results on the random test set are shown in Table 1. The introduction of VAD reduces the disturbance of noise to the separation algorithm and reduces audio insertion errors. The separation accuracy-related indicators are improved by more than 10%, and the PESQ subjective evaluation value is improved by more than 50%. At the same time, the overall power consumption is reduced by 27%.

[0149] Table 1:

[0150]

[0151] At the same time, under low signal-to-noise ratio, the audio signal is processed by different methods to obtain the noise reduction and signal separation effects shown in Table 2 below, where MUSIC+CNFM+AGC is the method adopted by the present invention. From Table 2, it can be seen that the MUSIC+CNFM algorithm can provide a relatively ideal noise reduction and signal separation effect on a low-power platform, while the AGC algorithm can significantly improve the subjective PESQ value.

[0152] Table 2:

[0153]

[0154] The present invention also proposes an intelligent sound-pickup badge, which is a sound-pickup badge based on a microphone array. The audio picked up by the sound-pickup badge is processed using the steps of the above-mentioned microphone array audio processing method, and the processed audio is output to meet the needs of teachers in teaching.

[0155] Mainly include:

[0156] A signal receiving module, used to pick up audio using a microphone array in the badge;

[0157] An audio processing module, which uses the steps of the microphone array audio processing method to realize the processing of the audio picked up by the microphone array;

[0158] A signal transmission module, used for transmitting the audio processed by the audio processing module;

[0159] The data storage module is used to store the audio processed by the audio processing module.

[0160] In this embodiment, two beams are used to pick up audio in corresponding directions respectively to collect the voices of teachers and students, and noise reduction and speaker differentiation are performed on the audios of teachers and students respectively to achieve the purpose of good teaching interaction.

[0161] Please refer to Fig.11 , Fig.11 This is a structural diagram of a specific embodiment of the intelligent sound pickup badge of the present invention. The badge includes a switch, an indicator light, a front panel, a rear shell, and a microphone array disposed in the shell.

[0162] In this embodiment, the sound pickup badge includes two parts: hardware and software. Fig.12 , Fig.12 The hardware and software structure diagram of the sound pickup badge is shown, where the hardware part includes MCU, DSP, Bluetooth + WIFI dual-mode transmission unit, data storage unit, badge power supply unit, LED unit, and microphone array. The software part includes VAD module, wake-up module, AoA Bluetooth module, signal processing module, speaker differentiation module, and streaming media module.

[0163] The main control chip MCU realizes the overall business process processing; the coprocessor DSP realizes signal processing; the storage unit realizes signal storage; the badge power supply unit provides the card with more than 8 hours of battery life; the LED unit is used for abnormal prompts, charging prompts, and work prompts; Bluetooth and WIFI are used for digital signal transmission; the microphone array is divided into linear two-microphone, linear four-microphone and rectangular six-microphone, which is suitable for small, medium and large application environments.

[0164] The VAD module is used for voice detection. After the card is turned on, the VAD module is started to identify human voices, remove blank audio and noise, and improve battery life and reduce memory (RAM) usage. In the wake-up module, the audio obtained by VAD extracts feature audio, compares the feature audio with the wake-up word pre-stored in the memory unit, and stores the valid voice after successful comparison. The wake-up module enables the card to realize voice control. The Bluetooth module realizes real-time positioning of the wearer based on AoA technology, and punches in without feeling. Both Bluetooth and WIFI can realize data transmission, and the data transmission mode can be selected through firmware configuration. The signal processing module uses AEC, DOA, BF, and DEC technologies to reduce the noise of the audio storage after VAD cutting and remove environmental noise. Speaker separation uses AGC technology to increase the wearer or non-wearer, selectively zero unnecessary audio, and store the enhanced audio near the card ROM. The audio separation technology can selectively amplify the teacher's audio and student audio in the classroom scene. In the service scene, it can be used to discard the customer's voice to prevent the leakage of customer privacy. The streaming media module is used to control audio transmission. The card uses Bluetooth and WIF dual-mode chips. The streaming media service can choose to use Bluetooth mode or WIFI mode to transmit audio. The Bluetooth module is used by default to save power.

[0165] The following uses an actual campus teaching classroom as an example to explain the audio processing of sound pickup badges and microphone arrays in detail.

[0166] The smart sound pickup badge can be fixed on the chest by means of pins, buttons, metal clips, etc. When the teacher enters the classroom, he / she turns on the badge hardware switch, the card enters the standby state, the Bluetooth module is turned on, the Bluetooth module broadcasts the Bluetooth signal, and the Bluetooth receiving device fixed in the classroom receives the Bluetooth signal. It can be used for teachers to clock in and out of the classroom. At this time, the badge is in the standby state. To save energy, only the VAD module is in the low-power operation mode.

[0167] Teachers can control the badge recording by voice. Voice control is based on wake-up technology, and the wake-up words need to be entered in advance, such as "start class, start recording"; the VAD module detects the human voice. When the recognized human voice contains the wake-up word, the badge MCU starts working, the badge enters normal working state, and the microphone array starts to pick up audio. The badge microphone array selects any audio channel and sends it to VAD for detection. VAD outputs the status bits of the start and end of the audio. Selecting one audio channel can save memory and reduce algorithm delay. The MCU intercepts multiple audio channels through the status bits and sends them to the DSP for signal processing. Among them, multi-channel audio refers to the original audio output by the microphone, and the number of audio channels is the same as the number of microphones.

[0168] Please refer to Fig.13 , Fig.13 The processing flow chart of the collected signal (audio) is shown. The signal processing algorithm first performs echo cancellation on the multi-channel audio, and then performs spatial sound field detection after echo cancellation to determine the speaker's position, that is, the sound source location DOA. After DOA is completed, the speaker's position is determined, and the beamforming algorithm selects the beam based on the position information provided by DOA for directional pickup. DOA is mainly used for directional enhancement and noise reduction in the direction of students. DOA in the direction of teachers is an optional algorithm. If the DOA algorithm is selected in the direction of teachers, strong noise reduction with a small constraint angle is required. DOA can track the position of the teacher's head to avoid the loss of the teacher's audio. The DOA algorithm is updated every 100ms. Frequent updates of the DOA algorithm can improve the energy consumption of the badge.

[0169] After beamforming, multiple audio channels will be output. The number of audio channels is consistent with the number of beams. At this time, there are two audio channels (including student direction and teacher direction), including forward beam and upward beam. The forward beam picks up the student's audio and the upward beam picks up the teacher's audio. The two audio channels are denoised separately. The denoising is usually based on the SCNMF algorithm to further remove background noise and mutation noise. The algorithm after SCNMF denoising passes through the speaker separation module. The speaker separation module determines whether the audio is AGC or zeroed (or ignored) based on the firmware configuration item. If you only want to save the teacher's audio, the teacher's audio after SCNMF denoising is amplified by AGC, the teacher's audio amplitude is increased, and written to RAM. Similarly, if you only want to amplify the student audio, the teacher's channel audio is zeroed (or ignored), and the student's audio after SCNMF is sent to the AGC module to amplify the far-field student's voice. The audio after AGC is written to RAM. The streaming media module reads the audio stored in RAM based on the firmware configuration item, and selects wifi or Bluetooth for transmission. Bluetooth transmission is usually used to save energy.

[0170] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0171] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0172] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0173] The above is a detailed introduction to the microphone array audio processing method and the sound pickup badge provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A method for processing microphone array audio, characterized in that: include: S1: Obtain the mixed speech signal collected by the microphone array; S2: Determine the threshold value of the current frame by weighting the threshold value of each frame in the previous e frames of the current frame of the mixed speech signal; S3: performing speech detection on the speech signal of the current frame according to the threshold value of the current frame, and when it is detected that the speech signal of the current frame is greater than the threshold value of the current frame, determining that the current frame is a non-silent frame, and temporarily storing the threshold value of the current frame; S4: Move the language frame backward, use the method described in S2 to S3 to predict the threshold value of each frame and perform speech detection on the speech signal of each frame, and regard the speech signals of all frames greater than the threshold value of each frame as non-silent speech signals; S5: performing speaker spatial positioning on the non-silent speech signal to obtain steering vectors of different speakers; S6: generating multiple beams according to the steering vectors of the different speakers, picking up the speech signal on each beam, and realizing speech signal enhancement of the speaker; S7: Perform noise reduction processing on the enhanced speaker speech signal, and output the noise-reduced speech signal on each beam.

2. The method for processing microphone array audio according to claim 1, characterized in that: The threshold value of the current frame is determined by weighting the threshold value of each frame in the previous e frames of the current frame of the mixed speech signal: or i =w i-1 or i-1 +w i-2 or i-2 +…+w i-e or i-e Where η i is the threshold value of the current frame, that is, the threshold value of the i-th frame audio signal in the mixed speech signal, (η i-1 , η i-2 , ..., η i-e ) is the threshold value of each audio signal from the previous frame to the previous e frames of the current frame, that is, the threshold value of the audio signal of the (i-1, i-2, ..., ie)th frame in the mixed speech signal, (w i-1 , w i-2 ,…,w i-e ) are respectively the weight values ​​of the audio signal threshold values ​​of each frame from the previous frame to the previous e frames of the current frame, that is, the weight values ​​of the audio signal threshold values ​​of the (i-1, i-2, ..., ie)th frames in the mixed speech signal, wherein i≥e+1, e represents the size of the sliding window during speech detection.

3. The method for processing microphone array audio according to claim 2, characterized in that: The step of determining the size of the sliding window during speech detection includes: Using sliding windows of different sizes based on H ∞ The criteria maximize the accuracy of speech detection: In the formula, A target is the standard audio in the basic test set, A correct is the audio after speech detection, Represents the accuracy of speech detection; The corresponding relationship between the speech detection accuracy and the sliding windows of different sizes is obtained, and the minimum value of the sliding window corresponding to the speech detection accuracy meeting the set accuracy threshold is selected as the size of the sliding window during speech detection.

4. The method for processing microphone array audio according to claim 1, characterized in that: The threshold values ​​of the first frame to the e-th frame of the mixed speech signal are respectively the energy mean values ​​of the audio signals of each frame.

5. The method for processing microphone array audio according to claim 1, characterized in that: The performing speech detection on the speech signal of the current frame according to the threshold value of the current frame comprises: The mixed speech signal collected by the microphone array is: In the formula, x m (t) is the audio signal collected by the microphone array, is the response function of microphone m and sound source c, D represents the number of sound sources, S c (t) is the output signal of the sound source c to the mth microphone, n m (t) is the noise signal of the mth microphone, and the total number of microphones is M; Using threshold value for speech detection: Where N N 、N S Represent the noise vector matrix and speech vector matrix respectively, X k，i represents the speech signal with the ith frequency point index in the kth frame audio spectrum in the mixed speech signal, L represents the number of frequency points in the kth frame audio spectrum, and η is the threshold value of the kth frame.

6. The method for processing microphone array audio according to claim 1, characterized in that: The speaker spatial positioning of the non-silent speech signal is performed to obtain the steering vectors of different speakers: In the formula, is the steering vector of the determined speaker, N M-D is the residual matrix composed of noise feature vectors, the subscript sr represents the number of peak searches, i.e. the number of speakers selected, and θ is the azimuth of the spherical coordinate system. is the elevation angle in spherical coordinates, For The probability of the speaker at .

7. The method for processing microphone array audio according to claim 1, characterized in that: Generating multiple beams according to the steering vectors of different speakers, picking up the speech signal on each beam, and realizing speech signal enhancement of the speaker includes: Where, X sr (t, f) is the Fourier transformed signal of the input speaker’s speech signal, Y sr (t, f) is the Fourier transform output signal of the speaker after azimuth enhancement, that is, the enhanced speaker's speech signal, For The noise variance matrix at θ sr is the speaker's azimuth in the spherical coordinate system, is the elevation angle of the speaker in the spherical coordinate system.

8. The method for processing microphone array audio according to claim 1, characterized in that: The noise reduction processing on the enhanced speaker voice signal includes: using a convolution non-negative matrix to reduce the noise on the enhanced speaker voice signal: Where D psa is the constructed loss function, H sr (t, f) is the speaker coefficient matrix, Y sr (t, f) is the enhanced speaker speech signal, is the speaker's denoised speech signal, and F represents the norm; Through the gradient search algorithm, the loss function D psa The difference is less than the set difference, and the best speaker coefficient matrix H is obtained. sr_opt (t, f); According to the optimal speaker coefficient matrix H sr_opt (t, f) generates the final denoised speech signal 9. The method for processing microphone array audio according to claim 1, characterized in that: Before outputting the voice signal after noise reduction processing on each beam, the method further includes: Convert the denoised speaker's speech signal from the frequency domain to the time domain: In the formula, rSTFT represents the inverse short-time Fourier transform, is the frequency domain signal of the speaker’s speech signal after noise reduction, is the time domain signal of the speaker's speech signal after noise reduction; The AGC method is used to To enhance the signal: Where g(n) is the enhanced speaker speech signal, A(n) is the dynamic gain coefficient, is the gain input, i.e., the time domain signal of the speaker's speech signal after noise reduction; Wherein, the dynamic gain coefficient is determined according to the target gain amplitude: In the formula, a is the gain adjustment factor, R is the target gain amplitude, and the target gain amplitude is no greater than -3dB.

10. A microphone array sound pickup badge, characterized in that: include: Signal receiving module: used to pick up audio using the microphone array in the badge; Audio processing module: adopting the steps of the microphone array audio processing method as described in any one of claims 1 to 9 above to realize the processing of the audio picked up by the microphone array; Signal transmission module: used to upload the audio processed by the audio processing module; The data storage module is used to store the audio processed by the audio processing module.

Images