Multi-channel audio data processing method and device, computer device, and storage medium

Abstract

The application relates to the technical field of audio data processing, and discloses a multi-channel audio data processing method, device, computer equipment and storage medium, which comprises the following steps: acquiring sampled audio data collected by a multi-channel microphone array; calculating a first covariance matrix of the sampled audio data; acquiring a second covariance matrix of pure noise data based on a Gaussian mixture model; acquiring a third covariance matrix of pure speech data according to a matrix difference obtained by subtracting the first covariance matrix from the second covariance matrix; acquiring a target eigenvector of the third covariance matrix; and synthesizing target audio data according to the target eigenvector and the sampled audio data. The multi-channel audio data processing method classifies and extracts noise and speech by using the clustering idea, estimates the distribution of pure noise data and mixed sound data in the sampled audio data by using a maximum likelihood method, improves the robustness of noise in the sampled audio data, and can better process noise data in the sampled audio data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio data processing, and particularly relates to a multi-channel audio data processing method and device, computer equipment and a storage medium. BACKGROUND

[0002] At present, the speech enhancement technology mainly removes the noise in the speech data and the interference of the noise on the speech, and improves the performance of the speech recognition system. According to the number of microphones for collecting speech, the speech enhancement can be divided into single-channel speech enhancement method and multi-channel speech enhancement method. The traditional multi-channel speech enhancement method usually extracts and processes the noise based on energy. However, in the actual application environment, the noise and interference in the audio data collected by the multi-channel microphone array are often complex, and the influence of the noise on the audio data is difficult to estimate, resulting in poor speech enhancement effect. SUMMARY

[0003] Therefore, it is necessary to provide a multi-channel audio data processing method, device, computer equipment and storage medium to solve the problem of poor speech enhancement effect of the existing multi-channel speech enhancement technology on audio data.

[0004] A multi-channel audio data processing method comprises the following steps: acquiring sample audio data collected by a multi-channel microphone array; the sample audio data comprises pure noise data and mixed sound data; calculating a first covariance matrix of the sample audio data; acquiring a second covariance matrix of the pure noise data based on a Gaussian mixture model; acquiring a third covariance matrix of pure speech data according to a matrix difference obtained by subtracting the second covariance matrix from the first covariance matrix; acquiring a target eigenvector of the third covariance matrix; and synthesizing target audio data according to the target eigenvector and the sample audio data.

[0005] The multi-channel audio data processing method converts the sampled audio data collected by the multi-channel microphone array into a first covariance matrix. The distribution of pure noise data in the sampled audio data is simulated and estimated by using a Gaussian mixture model to obtain a second covariance matrix of the pure noise data. The second covariance matrix is subtracted from the first covariance matrix, and the calculated matrix difference is taken as a third covariance matrix of pure speech data. The target audio data is synthesized according to the eigenvectors of the third covariance matrix and the sampled audio data. The multi-channel audio data processing method provided by the present disclosure classifies and extracts noise and speech by using the clustering idea, estimates the distribution of pure noise data and mixed sound data in the sampled audio data by using the maximum likelihood method, and improves the robustness of noise in the sampled audio data. By improving the robustness of noise data in the audio data, the noise data and speech data in the sampled audio data can be better separated. Further, the noise contained in the audio data collected in different environments is well processed.

[0006] In one of the embodiments, the calculating the first covariance matrix of the sampled audio data comprises: performing frame windowing on the sampled audio data to obtain a plurality of frames of audio data; performing fast Fourier transform on the plurality of frames of audio data to obtain the sampled audio data in a time-frequency domain; and substituting the sampled audio data in the time-frequency domain into a covariance calculation formula of mixed sound data to obtain the first covariance matrix.

[0007] In one of the embodiments, the sampled audio data in the time-frequency domain comprises:

[0008]

[0009] In the formula, y f,t represents the value of the sampled audio data after the fth fast Fourier transform at the t time, v represents any one of pure noise data n or mixed sound data (x+n), s f,t represents audio data generated by a real sound source, r f,t represents a propagation function from the real sound source to the microphone.

[0010] The covariance calculation formula of the mixed sound data comprises:

[0011]

[0012] In the formula, Y represents the first covariance matrix, y f,t represents the value of the sampled audio data after the fth fast Fourier transform at the t time, represents the conjugate matrix of y f,t , and T represents the audio time length of the sampled audio data.

[0013] In one embodiment, the method of obtaining the second covariance matrix of the pure noise data based on the Gaussian mixture model comprises obtaining an initial setting value of the speech arrival vector of the pure noise data; calculating the Gaussian mixture distribution coefficient of the pure noise data according to the initial setting value; calculating the parameter probability in the Gaussian mixture model of the pure noise data according to the Gaussian mixture distribution coefficient; calculating a new speech arrival vector of the pure noise data according to the Gaussian mixture distribution coefficient and the parameter probability; repeating the iteration process of calculating a new Gaussian mixture distribution coefficient and a new parameter probability using the new speech arrival vector until the iteration converges; substituting the parameter probability of the pure noise data at the time of iteration convergence into the covariance calculation formula of the pure noise data to obtain the second covariance matrix.

[0014] In one embodiment, in the iteration process of calculating a new Gaussian mixture distribution coefficient and a new parameter probability using the new speech arrival vector until the iteration converges, the method further comprises obtaining a fourth covariance matrix of the mixed sound data based on the Gaussian mixture model; comparing the characteristic parameters of the second covariance matrix obtained in each iteration with the characteristic parameters of the fourth covariance matrix obtained in each iteration; and correcting the second covariance matrix and the fourth covariance matrix according to the comparison result.

[0015] In one embodiment, the method of calculating the Gaussian mixture distribution coefficient comprises:

[0016]

[0017] wherein, represents the Gaussian mixture distribution coefficient, M represents the number of channels of the microphone in the multi-channel microphone array, tr() represents the trace of the matrix, y f,t represents the value of the sampled audio data after the fast Fourier transform at the t time, f represents the fth microphone in the multi-channel microphone array, represents the conjugate matrix of y f,t , represents the speech arrival vector;

[0018] The method of calculating the parameter probability comprises:

[0019]

[0020] wherein, represents the parameter probability, P(y f,t |d f,t = v, Θ) represents y f,t is the probability density function of the pure noise data n or y f,t is the probability density function of the mixed sound data (x + n), ∑ v P(y f,t|d f,t represents the probability density function of the pure noise data n or y f,t represents the probability density function of the mixed sound data (x+n) ; f,t represents the sum of the probability density functions of the mixed sound data (x+n) ;

[0021] The method for calculating the speech arrival vector comprises:

[0022]

[0023] wherein, represents the speech arrival vector, represents the parameter probability, represents the sum of the parameter probabilities at all time points, represents the coefficient of the Gaussian mixture distribution, y f,t represents the value of the sampled audio data after the f th fast Fourier transform at the time t, represents the conjugate matrix of y f,t ;

[0024] The covariance calculation formula of the pure noise data comprises:

[0025]

[0026] wherein, represents the covariance of the pure noise data, y f,t represents the value of the sampled audio data after the f th fast Fourier transform at the time t, represents the conjugate matrix of y f,t ; represents the parameter probability of the pure noise data.

[0027] In one of the embodiments, the method for obtaining the target eigenvector of the third covariance matrix comprises performing eigen decomposition on the third covariance matrix to obtain a plurality of eigenvalues and a plurality of eigenvectors, one of the eigenvalues corresponding to one of the eigenvectors; and defining the eigenvector corresponding to the eigenvalue meeting the preset condition as the target eigenvector.

[0028] In one of the embodiments, the target eigenvector and the sampled audio data are taken as the input of the beamforming algorithm to obtain the target audio data.

[0029] An audio data processing apparatus comprises a data acquisition module configured to acquire sampled audio data collected by a multi-channel microphone array; the sampled audio data comprises pure noise data and mixed sound data; a data processing module configured to calculate a first covariance matrix of the sampled audio data; a model generation module configured to acquire a second covariance matrix of the pure noise data based on a Gaussian mixture model and the first covariance matrix, and further configured to acquire a third covariance matrix of pure speech data according to a matrix difference between the second covariance matrix and the first covariance matrix; and an audio processing module configured to acquire a target eigenvector of the third covariance matrix, and further configured to synthesize target audio data according to the target eigenvector and the sampled audio data.

[0030] A computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements steps of the multi-channel audio data processing method according to any one of the above embodiments when executing the computer program.

[0031] A computer readable storage medium stores a computer program, and the computer program implements steps of the multi-channel audio data processing method according to any one of the above embodiments when executed by a processor.

[0032] A computer program product comprises a computer program, and the computer program implements steps of the multi-channel audio data processing method according to any one of the above embodiments when executed by a processor. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present specification, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0034] Figure 1 A method flowchart of the multi-channel audio data processing method of one embodiment of the present disclosure;

[0035] Figure 2 A method flowchart of acquiring the first covariance matrix of one embodiment of the present disclosure;

[0036] Figure 3 A method flowchart of acquiring the second covariance matrix of one embodiment of the present disclosure;

[0037] Figure 4 A method flowchart of correcting the second covariance matrix and the fourth covariance matrix of one embodiment of the present disclosure;

[0038] Figure 5 A flowchart of a method for obtaining a target eigenvector of a third covariance matrix according to an embodiment of the present disclosure;

[0039] Figure 6 A block diagram of a multi-channel audio data processing device according to an embodiment of the present disclosure;

[0040] Figure 7 A block diagram of an audio data processing device or system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] For the purpose of promoting an understanding of the application, the application will be described in greater detail below with reference to the drawings. The preferred embodiments of the application are shown in the drawings. However, the application can be realized in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0043] Figure 1 A flowchart of a method for processing multi-channel audio data according to an embodiment of the present disclosure, in one embodiment, the method for processing multi-channel audio data includes the following steps S100-S600.

[0044] Step S100: Obtain sampled audio data collected by a multi-channel microphone array, the sampled audio data including pure noise data and mixed sound data.

[0045] The multi-channel microphone array is used to collect audio data, and the audio data collected by each channel is integrated into the sampled audio data. Since noise exists at all times in any application scenario, current audio collection technology cannot directly collect pure speech data without noise interference, and therefore it can be assumed that the sampled audio data includes two types of data, one being pure noise data that only collects noise, and the other being mixed sound data that collects speech mixed with noise.

[0046] Step S200: Calculate a first covariance matrix of the sampled audio data.

[0047] The covariance can be used to measure the degree of each dimension deviating from its mean value, and when the value of the covariance is positive, the two are positively correlated, and when the value of the covariance is negative, the two are negatively correlated. The covariance of the audio data at any two time points in the sampled audio data can be calculated, and correspondingly, the first covariance matrix corresponding to the sampled audio data can be calculated by calculating all the data in the sampled audio data.

[0048] Step S300: obtaining the second covariance matrix of the pure noise data based on the Gaussian mixture model.

[0049] The Gaussian mixture model (GMM) is a mixture model composed of multiple single Gaussian models, and is a probabilistic clustering method and a generative model. The Gaussian mixture model assumes that all data samples are generated by a given parameter of a multivariate Gaussian distribution. The probability density function of a Gaussian mixture model can be represented by a mixture distribution composed of K multivariate Gaussian distributions. Each distribution is referred to as a component in the Gaussian mixture model, and the proportion of samples generated by the corresponding component to the total samples is selected as a mixture component. According to the probability density function of the mixture component, a corresponding sample is generated by sampling. The clustering process using GMM is the inverse process of generating data samples using GMM. Given the number of clusters K, the parameters of each mixture component are derived from the given data set.

[0050] The Gaussian mixture model is not commonly used in audio processing methods, but in this embodiment, the Gaussian mixture model is applied innovatively. By using the Gaussian mixture model to cluster the pure noise data in the sampled audio data, the parameters of the mixture components of the pure noise data in the sampled audio data are obtained, and thus the second covariance matrix of the pure noise data is obtained.

[0051] Step S400: obtaining the third covariance matrix of the pure speech data according to the matrix difference obtained by subtracting the first covariance matrix from the second covariance matrix of the simulated pure noise data.

[0052] In this embodiment, the overall sampled audio data collected by the multi-channel microphone array can be regarded as a mixed sound data, which collects both speech data and noise data. At the same time, the sampled audio data can also include pure noise data. Therefore, by subtracting the second covariance matrix of the simulated pure noise data from the first covariance matrix, the third covariance matrix of the pure speech data can be obtained, that is, the speech data in the sampled audio data is separated, and thus the optimization processing for the speech data part in the audio data can be realized.

[0053] In one of the embodiments, the second covariance matrix is R(n), and the first covariance matrix is R(x+n). That is, the third covariance matrix of the pure speech data is obtained by subtracting the second covariance matrix R(n) from the first covariance matrix R(x+n), and the third covariance matrix of the pure speech data is R(x)=R(x+n)-R(n).

[0054] Step S500: obtaining a target eigenvector of the third covariance matrix.

[0055] Step S600: synthesizing target audio data according to the target eigenvector and the sampled audio data.

[0056] The eigenvectors that meet the preset target condition are defined as target eigenvectors by performing eigendecomposition on the third covariance matrix. The target eigenvectors of the third covariance matrix can be used as the angle of arrival vector of the pure speech data. The target eigenvectors and the sampled audio data are taken as inputs of a related audio processing algorithm to synthesize target audio data.

[0057] In the prior art, when processing the audio data collected by a microphone, the performance parameters of the microphone need to be obtained to further process the collected audio data. Since the Gaussian mixture model can use known data to solve unknown data, when applied to an audio data processing method, the simulation value can be made to approximate the true value through iterative optimization by giving a hypothetical value. It can be seen that when the multi-channel audio data processing method provided by the present disclosure is used to process the sampled audio data, even if the performance parameters of the microphone are unknown, the noise in the sampled audio data can also be well processed, thereby achieving enhancement of the speech part of the sampled audio data.

[0058] The multi-channel audio data processing method uses the clustering idea of the Gaussian mixture model to classify and extract noise and speech, estimates the distribution of pure noise data and mixed sound data in the sampled audio data through the maximum likelihood method, and improves the robustness of the noise in the sampled audio data. By improving the robustness of the noise data in the audio data, the noise data and speech data in the sampled audio data can be better separated. Further, the noise contained in the audio data collected in different environments can be well processed.

[0059] Figure 2 The method flowchart for obtaining the first covariance matrix of one of the embodiments of the present disclosure includes the following steps S210 to S230.

[0060] Step S210: frame and window the sampled audio data to obtain multiple frames of audio data.

[0061] Since the input signal is required to be stationary when performing the fast Fourier transform, and the audio data is not stationary in the macroscopic view but is stationary in the microscopic view, the audio data is divided into many short segments with stationarity by framing the audio data, and each segment can be referred to as a chunk. Each chunk of the audio data obtained after framing has stationarity, and thus can be used as the input of the fast Fourier transform.

[0062] In some other embodiments of the present disclosure, instead of back-to-back truncation, the adjacent two chunks can be partially overlapped when framing. The time difference between the start positions of the adjacent two chunks is referred to as the chunk shift.

[0063] In digital processing, the long signal sequence can also be truncated by using a windowing operation. The windowing operation refers to multiplying the audio data after framing with a window function, and the windowing operation can make the global signal more continuous and avoid the Gibbs phenomenon. After windowing, some speech signals originally without periodicity in the audio data can exhibit some characteristics of periodic functions, and thus the audio data can be better expanded by the Fourier transform.

[0064] In some other embodiments of the present disclosure, the audio data can also be pre-emphasized for speech signals, and the purpose is to emphasize the high-frequency part of the speech, remove the influence of the lip radiation, and increase the high-frequency resolution of the speech. The pre-emphasis can be achieved by using a high-pass digital filter with a first-order transfer function.

[0065] Step S220: performing the fast Fourier transform on the multiple chunks of the audio data to obtain the audio data in the time-frequency domain.

[0066] Step S220: performing the fast Fourier transform on the multiple chunks of the audio data to obtain the audio data in the time-frequency domain.

[0067] In one embodiment, if the classification assumption that the audio data only includes pure noise data and mixed sound data is correct, the audio data in the time-frequency domain at any observation time can be described by the following expression, and the audio data in the time-frequency domain at any observation time is:

[0068]

[0069] In the expression, y f,t represents the value of the audio data after the fth fast Fourier transform at the time t, v represents any one of the pure noise data n or the mixed sound data (x+n), s f,trepresents audio data generated by a real sound source, r f,t represents a propagation function between the real sound source and the microphone.

[0070] Step S230: Substituting the time-frequency domain sampled audio data into the covariance calculation formula of the mixed sound data, a first covariance matrix is obtained.

[0071] In one embodiment, the covariance calculation formula of the mixed sound data comprises:

[0072]

[0073] wherein, represents the covariance of the mixed sound data, y f,t represents the value of the fthfast Fourier transform of the sampled audio data at time t, represents the conjugate matrix of y f,t , and T represents the audio time length of the sampled audio data.

[0074] In this embodiment, since the sampled audio data collected by the multi-channel microphone array generally collects both speech data and noise data, the sampled audio data as a whole can be regarded as a mixed sound data. Substituting the time-frequency domain sampled audio data obtained by fast Fourier transform in step S220 into the above-mentioned covariance calculation formula of the mixed sound data, a first covariance matrix of the sampled audio data can be obtained. In the subsequent data processing process, the first covariance matrix can be used as calibration data.

[0075] Figure 3 The method flowchart for obtaining the second covariance matrix of one embodiment of the present disclosure comprises the following steps S310 to S360.

[0076] Step S310: Obtain the initial setting value of the speech arrival vector of the pure noise data.

[0077] In some embodiments of the present disclosure, the maximum likelihood method is used to estimate the distribution of the pure noise data in the sampled audio data. Since the Gaussian mixture model can use known data to solve unknown data, when simulating the simulation value of the pure noise data, the initial setting value of the speech arrival vector of the pure noise data can be pre-set, so that the simulation value of the pure noise data is approximated to the real value by iterative optimization. In this embodiment, the initial setting value can be set as the eigenvector of the unit matrix.

[0078] Step S320: Calculate the Gaussian mixture distribution coefficient of the pure noise data according to the initial setting value.

[0079] In one embodiment, the method for calculating the Gaussian mixture distribution coefficient includes:

[0080]

[0081] In the formula, The coefficients represent the Gaussian mixture distribution coefficients, M represents the number of microphone channels in the multi-channel microphone array, tr() represents the trace of the matrix, and y f,t This represents the value of the sampled audio data after the f-th Fast Fourier Transform at time t. Indicates y f,t The conjugate matrix, Represents the speech arrival vector;

[0082] Since this embodiment is used for simulation of pure noise data, the formula for calculating the Gaussian mixture distribution coefficient of pure noise data should be:

[0083]

[0084] After obtaining the initial setting value of the speech arrival vector of the pure noise data, substitute the initial setting value of the speech arrival vector into the above formula for calculating the Gaussian mixture distribution coefficient, that is, let the above formula... The value is the initial setting of the speech arrival vector for pure noise data. Simultaneously, the number of microphone channels M in the multi-channel microphone array, and the value y of the sampled audio data after the f-th fast Fourier transform at time t. f,t and y f,t conjugate matrix Since all values ​​are known, the Gaussian mixture distribution coefficients for obtaining pure noise data can be calculated.

[0085] Step S330: Calculate the parameter probabilities in the Gaussian mixture model of pure noise data based on the Gaussian mixture distribution coefficients.

[0086] In one embodiment, the method for calculating the parameter probability includes:

[0087]

[0088] In the formula, P(y) represents the parameter probability. f,t |d f,t =v,Θ) represents y f,t The probability density function or y of pure noise data n f,t Let ∑ be the probability density function of the mixed sound data (x+n). v P(y f,t |d f,t =v,Θ) represents the relationship between y f,t The probability density function or y of pure noise data nf,t Sum the probability density function of the mixed sound data (x+n).

[0089] Since this embodiment is used for simulation of pure noise data, the formula for calculating the parametric probability of pure noise data should be:

[0090]

[0091] Substituting the Gaussian mixture distribution coefficients of the pure noise data into the probability density function P(y) of the Gaussian mixture distribution in the above formula f,t |d f,t =n,Θ), to find y f,t The probability value for all y when n is pure noise data. f,t By summing the probability values ​​for pure noise data n, we can then calculate the parametric probability of obtaining pure noise data.

[0092] Step S340: Calculate the new speech arrival vector of the pure noise data based on the Gaussian mixture distribution coefficient and parametric probability.

[0093] In one embodiment, the method for calculating the speech arrival vector includes:

[0094]

[0095] In the formula, Represents the speech arrival vector. Represents the probability of the parameter. This represents the summation of the parameter probabilities over all time points. y represents the Gaussian mixture distribution coefficient. f,t This represents the value of the sampled audio data after the f-th Fast Fourier Transform at time t. Indicates y f,t The conjugate matrix.

[0096] Since this embodiment is used for simulation of pure noise data, the formula for calculating the speech arrival vector of pure noise data should be:

[0097]

[0098] Substitute the Gaussian mixture distribution coefficients of the pure noise data obtained in step S320 and the parametric probabilities of the pure noise data obtained in step S330 into the above formula. Simultaneously, the value y of the sampled audio data after the f-th Fast Fourier Transform at time t... f,t and y f,t conjugate matrix Since all values ​​are known, the speech arrival vector of the pure noise data can be calculated.

[0099] Step S350: Repeat the iteration process of calculating new Gaussian mixture distribution coefficients and new parameter probabilities using the new speech arrival vector until the iteration converges.

[0100] The speech arrival vector calculated in step S350 is used as the new speech arrival vector The new speech arrival vector is used to calculate new Gaussian mixture distribution coefficients and new parameter probabilities

[0101] The basic idea of the EM algorithm (Expectation-Maximization Algorithm) is to estimate the value of the model parameters according to the given observation data, then estimate the value of the missing data according to the parameter value estimated in the previous step, and then re-estimate the parameter value according to the estimated missing data and the data that has been observed before, and then iterate repeatedly until the final convergence, and the iteration ends.

[0102] In some embodiments of the present disclosure, the EM algorithm is used to approximate the maximum likelihood Gaussian mixture distribution of the pure noise data in the audio data. By pre-setting the initial value of the speech arrival vector , the value of the Gaussian mixture distribution coefficient is estimated according to the initial value of the speech arrival vector , the value of the parameter probability is estimated according to the value of the Gaussian mixture distribution coefficient , the new speech arrival vector is estimated according to the value of the Gaussian mixture distribution coefficient and the value of the parameter probability , the Gaussian mixture distribution coefficient and the parameter probability are re-estimated according to the estimated new speech arrival vector , and then iterated repeatedly until the final convergence, and the iteration ends.

[0103] Step S360: Substitute the parameter probability of the pure noise data at the time of iteration convergence into the covariance calculation formula of the pure noise data to obtain a second covariance matrix.

[0104] Assuming that the sound source signal s (n) and s (x+n) represent the pure noise signal and the mixed sound signal respectively, and the pure noise signal and the mixed sound signal are subject to circularly symmetric distribution, which is represented as follows:

[0105]

[0106] ​​Therefore, based on the expression for the time-frequency domain sampled audio data in the above embodiments, the following can be derived:

[0107]

[0108] in, Therefore, the audio transfer function r can be derived from the estimated speech arrival vector R of the pure noise data.

[0109] Assumption Let (f, t) represent the probability that the time-frequency point (f, t) is only noise, and simply assume that speech and noise are linearly independent. Then, based on the covariance calculation formula for mixed sound data in the above embodiment, we can derive:

[0110]

[0111] In the formula, y represents the covariance of pure noise data. f,t This represents the value of the sampled audio data after the f-th Fast Fourier Transform at time t. Indicates y f,t The conjugate matrix, T represents the audio duration of the sampled audio data; The parameter probability represents pure noise data.

[0112] Substitute the parameter probabilities of the pure noise data obtained during the iterative convergence in step S350 into the covariance calculation formula for the pure noise data. Simultaneously, the value y of the sampled audio data after the f-th Fast Fourier Transform at time t... f,t and y f,t conjugate matrix Since all values ​​are known, the formula for calculating the second covariance of pure noise data can be obtained. Because the parameter probabilities of pure noise data are estimated using an algorithm, the formula for calculating the second covariance of pure noise data is an algorithmic estimate.

[0113] Figure 4 This is a flowchart illustrating a method for correcting the second covariance matrix and the third covariance matrix according to one embodiment of the present disclosure. In one embodiment, the method further includes the following steps S370 to S390 when the iterative process of repeatedly using the new speech arrival vector to calculate the new Gaussian mixture distribution coefficient and the new parameter probability is completed until the iteration converges.

[0114] Step S370: Obtain the fourth covariance matrix of the mixed sound data based on the Gaussian mixture model.

[0115] Similarly, the fourth covariance matrix of the mixed sound data can also be obtained based on a Gaussian mixture model. The simulation process of the fourth covariance matrix of the mixed sound data is similar to the simulation process of the second covariance matrix of the pure noise data in the above embodiment. By pre-setting the initial setting value of the speech arrival vector of the mixed sound data, the simulated value of the mixed sound data can be made to approximate the true value through iterative optimization.

[0116] Since this embodiment is used for simulation of mixed sound data, the formula for calculating the Gaussian mixture distribution coefficient of the mixed sound data should be:

[0117]

[0118] Obtain the initial setting value of the speech arrival vector of the mixed sound data, and substitute this initial setting value into the calculation formula of the Gaussian mixture distribution coefficient above, that is, let the above formula... The value is the initial setting of the speech arrival vector of the mixed sound data. Simultaneously, the number of microphone channels M in the multi-channel microphone array, and the value y of the sampled audio data after the f-th fast Fourier transform at time t. f,t and y f,t conjugate matrix Since all values ​​are known, the Gaussian mixture distribution coefficients of the mixed sound data can be calculated.

[0119] Since this embodiment is used for simulation of mixed sound data, the formula for calculating the parametric probabilities of mixed sound data should be:

[0120]

[0121] Substituting the Gaussian mixture distribution coefficients of the mixed audio data into the probability density function P(y) of the Gaussian mixture distribution in the above formula... f,t |d f,t =n,Θ), to find y f,t Let be the probability value for the mixed sound data (x+n), for all y f,t By summing the probability values ​​for the mixed sound data (x+n), we can then calculate the parametric probabilities of acquiring the mixed sound data.

[0122] Since this embodiment is used for simulation of mixed audio data, the formula for calculating the speech arrival vector of mixed audio data should be:

[0123]

[0124] The Gaussian mixture distribution coefficient of the mixed sound data obtained in step S320 and the parameter probability of the mixed sound data obtained in step S330 are substituted into the above formula. Meanwhile, the value y f,t and the conjugate matrix f,t of y f,t are known values, so the speech arrival vector of the mixed sound data can be calculated.

[0125] In some embodiments of the present disclosure, an EM algorithm (Expectation-Maximization Algorithm) is used to approximate the maximum likelihood Gaussian mixture distribution of the mixed sound data in the sampled audio data. By presetting the initial value of the speech arrival vector , the value of the Gaussian mixture distribution coefficient is estimated according to the initial value of the speech arrival vector , the value of the parameter probability is estimated according to the value of the Gaussian mixture distribution coefficient , the new speech arrival vector is estimated according to the value of the Gaussian mixture distribution coefficient and the value of the parameter probability , and the new speech arrival vector is estimated according to the estimated new speech arrival vector , and the Gaussian mixture distribution coefficient and the parameter probability are estimated again, and then the iteration is repeated until the final convergence, and the iteration is ended.

[0126] When the iteration converges, the parameter probability of the mixed sound data obtained is substituted into the covariance calculation formula of the mixed sound data. Meanwhile, the value y f,t and the conjugate matrix f,t of y are known values, so the second covariance calculation formula of the mixed sound data can be obtained. Since the parameter probability of the mixed sound data is also estimated by the algorithm in the present embodiment, the fourth covariance calculation formula of the mixed sound data is also an algorithm estimation value.

[0127] Step S380: The characteristic parameters of the second covariance matrix obtained in each iteration are compared with the characteristic parameters of the fourth covariance matrix obtained in each iteration.

[0128] Step S390: The second covariance matrix and the fourth covariance matrix are corrected according to the comparison result.

[0129] After the parameter probability of the pure noise data and the parameter probability of the mixed sound data are obtained in each calculation, the parameter probability of the pure noise data is substituted into the covariance calculation formula of the pure noise data to obtain the second covariance matrix at this time, and the parameter probability of the mixed sound data is substituted into the covariance calculation formula of the mixed sound data to obtain the fourth covariance matrix at this time.

[0130] Since the pure noise data and the mixed sound data may be mixed in the iteration process, the covariance matrices corresponding to the pure noise data n and the mixed sound data (x+n) need to be corrected after each iteration process of the pure noise data n and the mixed sound data (x+n).

[0131] The second covariance matrix at this time is subjected to eigenvalue decomposition, and a plurality of eigenvalues corresponding to the second covariance matrix at this time can be obtained. The plurality of eigenvalues corresponding to the second covariance matrix at this time are multiplied by each other, and the obtained product is taken as the eigenvalue parameter of the second covariance matrix at this time. Similarly, the fourth covariance matrix at this time is subjected to eigenvalue decomposition, and a plurality of eigenvalues corresponding to the fourth covariance matrix at this time can be obtained. The plurality of eigenvalues corresponding to the fourth covariance matrix at this time are multiplied by each other, and the obtained product is taken as the eigenvalue parameter of the fourth covariance matrix at this time.

[0132] The eigenvalue parameter of the second covariance matrix obtained in each iteration is compared with the eigenvalue parameter of the fourth covariance matrix obtained in each iteration. In this embodiment, the covariance matrix corresponding to the eigenvalue parameter with the larger eigenvalue product value of the two eigenvalue parameters is defined as the fourth covariance matrix R(x+n) of the mixed sound data (x+n), and the covariance matrix corresponding to the eigenvalue parameter with the smaller eigenvalue product value of the two eigenvalue parameters is defined as the second covariance matrix R(n) of the pure noise data n, so as to realize the correction of the second covariance matrix and the fourth covariance matrix according to the comparison result, to improve the accuracy of the classification result and enhance the robustness of the noise.

[0133] Figure 5 The method flowchart for obtaining the target eigenvector of the fourth covariance matrix of one embodiment of the present disclosure is shown in the following steps S510 to S520.

[0134] Step S510: The third covariance matrix is subjected to eigenvalue decomposition to obtain a plurality of eigenvalues and a plurality of eigenvectors, one eigenvalue corresponding to one eigenvector.

[0135] After the third covariance matrix of the pure speech data is obtained by the EM algorithm, eigenvalue decomposition is performed on the third covariance matrix. Eigenvalue decomposition refers to a method of decomposing a matrix into a product of matrices represented by eigenvalues and eigenvectors. After eigenvalue decomposition, a plurality of eigenvalues of the third covariance matrix are obtained, and one eigenvalue corresponds to one eigenvector.

[0136] Step S520: Defining the eigenvector corresponding to the eigenvalue meeting the preset condition as a target eigenvector.

[0137] In this embodiment, by comparing the sizes of the eigenvalues, the eigenvector corresponding to the maximum eigenvalue can be defined as the target eigenvector of the pure speech data, that is, the preset condition is to select the eigenvalue with the largest value as the target eigenvector. The multi-channel audio data processing method provided by the present disclosure assumes that the pure noise data and the mixed sound data are subject to complex Gaussian distribution, which is closer to the distribution state of noise and speech under random state. The EM algorithm is used to iteratively solve the parameter probability of the Gaussian mixture distribution model, so that the method is more universal and can better process noise in different environments.

[0138] In one of the embodiments, the target eigenvector and the sampled audio data are used as inputs of a beamforming algorithm to obtain target audio data. MVDR is an adaptive beamforming algorithm based on the maximum signal-to-noise ratio (SINR) criterion. According to the idea of beamforming, the sound in the direction of the sound source (i.e., the direction with the strongest sound intensity) can be enhanced, while the sound in other directions can be suppressed. The MVDR algorithm can adaptively minimize the power and maximize the signal-to-noise ratio in the desired direction.

[0139] The target eigenvector corresponding to the maximum eigenvalue is defined as the arrival angle vector of the pure speech data. By using the target eigenvector and the sampled audio data as inputs of a beamforming algorithm, the MVDR algorithm can be used to enhance the pure speech data in the sampled audio data according to the arrival angle vector of the pure speech data, and synthesize a single-channel audio data, i.e., target audio data, thereby realizing positioning and enhancement of the pure speech data in the sampled audio data.

[0140] In one of the embodiments, after obtaining the target audio data, the target audio data can also be substituted into a single-channel speech enhancement post-processing algorithm to further optimize the speech enhancement of the target audio data. In this embodiment, the single-channel speech enhancement post-processing algorithm can be an OMLSA (Optimally-Modified Log-Spectral Amplitude) algorithm.

[0141] It should be understood that, although Figures 1-5The steps in the flowchart are shown in sequence according to the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the steps are not strictly limited in sequence, and the steps can be executed in other sequences. Moreover, Figures 1-5 At least one of the steps in the flowchart can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of the steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least one of the other steps or the steps or stages in the other steps.

[0142] Based on the description of the above-mentioned multi-channel audio data processing method embodiment, the present disclosure also provides a multi-channel audio data processing device. The device can include a system (including a distributed system), software (application), module, component, server, client, etc. using the method described in the embodiments of the present disclosure, and a device combined with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided by the embodiments of the present disclosure is described in the following embodiments. Since the implementation scheme of the device to solve the problem is similar to the method, the implementation of the specific device in the embodiments of the present disclosure can be referred to the implementation of the foregoing method, and the repeated parts will not be described. The term "unit" or "module" used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware is also possible and is conceived.

[0143] Figure 6 The structural block diagram of the multi-channel audio data processing device of one embodiment of the present disclosure is shown in FIG. 1. In one embodiment, the multi-channel audio data processing device can be a terminal, a server, or a module, component, device, unit, etc. integrated in the terminal. The multi-channel audio data processing device can include a data acquisition module 100, a data processing module 200, a model generation module 300, and an audio processing module 400.

[0144] The data acquisition module 100 is configured to acquire sampled audio data collected by a multi-channel microphone array. The sampled audio data includes pure noise data and mixed sound data. The data processing module 200 is configured to calculate a first covariance matrix of the sampled audio data. The model generation module 300 is configured to obtain a second covariance matrix of the pure noise data based on a Gaussian mixture model and the first covariance matrix, and to obtain a third covariance matrix of the pure speech data according to a matrix difference obtained by subtracting the second covariance matrix from the first covariance matrix. The audio processing module 400 is configured to obtain a target eigenvector of the third covariance matrix, and to synthesize target audio data according to the target eigenvector and the sampled audio data.

[0145] As to the apparatus in the above embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0146] It can be understood that each of the above-described embodiments of the method, the apparatus, etc. is described in a progressive manner, and the same / similar parts between the embodiments can be referred to each other, and each of the embodiments mainly describes the difference from other embodiments. The relevant parts can be referred to the description of the other embodiments.

[0147] Figure 7 A block diagram of a multi-channel audio data processing apparatus or system according to an embodiment of the present disclosure. Referring to Figure 7 , the multi-channel audio data processing apparatus or system S00 includes a processing component S20, which further includes one or more processors, and a memory resource represented by a memory S22, for storing instructions, such as an application program, executable by the processing component S20. The application program stored in the memory S22 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component S20 is configured to execute the instructions to perform the above method.

[0148] The multi-channel audio data processing apparatus or system S00 can also include a power component S24 configured to perform power management of the multi-channel audio data processing apparatus or system S00, a wired or wireless network interface S26 configured to connect the multi-channel audio data processing apparatus or system S00 to a network, and an input / output (I / O) interface S28. The multi-channel audio data processing apparatus or system S00 can operate based on an operating system stored in the memory S22, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0149] In an exemplary embodiment, a computer-readable storage medium including instructions, such as the memory S22 including instructions, is also provided, and the above-described instructions can be executed by the processor of the multi-channel audio data processing apparatus or system S00 to complete the above-described method. The storage medium can be a computer-readable storage medium, such as a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0150] In an exemplary embodiment, a computer program product including instructions is also provided, and the above-described instructions can be executed by the processor of the multi-channel audio data processing apparatus or system S00 to complete the above-described method.

[0151] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the hardware + program type embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments.

[0152] It should be noted that the above-described apparatus, electronic device, server, etc. according to the description of the method embodiments can also include other embodiments, and the specific implementation can refer to the description of the related method embodiments. Meanwhile, the mutual combination of the features of each method and the apparatus, device, server embodiments constitutes a new embodiment, which still belongs to the implementation range covered by the present disclosure, and will not be described one by one here.

[0153] In the description of the specification, the description of the terms "some embodiments", "other embodiments", "ideal embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example.

[0154] Each of the technical features of the above-described embodiments can be combined arbitrarily, and in order to make the description simple, each technical feature in the above-described embodiments is not described all possible combinations, however, as long as the combination of these technical features does not exist contradictory, it should be considered as the scope of the present disclosure.

[0155] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method of processing multi-channel audio data, characterized by, The method comprises: acquiring sampled audio data collected by a multi-channel microphone array; the sampled audio data comprises pure noise data and mixed sound data; calculating a first covariance matrix of the sampled audio data; acquiring a second covariance matrix of the pure noise data based on a Gaussian mixture model; acquiring a third covariance matrix of pure speech data according to a matrix difference obtained by subtracting the second covariance matrix from the first covariance matrix; acquiring a target eigenvector of the third covariance matrix; synthesizing target audio data according to the target eigenvector and the sampled audio data; the method of acquiring the second covariance matrix of the pure noise data based on the Gaussian mixture model comprises: acquiring an initial setting value of a speech arrival vector of the pure noise data; calculating a Gaussian mixture distribution coefficient of the pure noise data according to the initial setting value; calculating a parameter probability in the Gaussian mixture model of the pure noise data according to the Gaussian mixture distribution coefficient; calculating a new speech arrival vector of the pure noise data according to the Gaussian mixture distribution coefficient and the parameter probability; repeating an iteration process of calculating a new Gaussian mixture distribution coefficient and a new parameter probability using a new speech arrival vector until iteration converges; substituting the parameter probability of the pure noise data at the time of iteration convergence into a covariance calculation formula of the pure noise data to acquire the second covariance matrix; in the iteration process of repeating calculation of a new Gaussian mixture distribution coefficient and a new parameter probability using a new speech arrival vector until iteration converges, the method further comprises: acquiring a fourth covariance matrix of the mixed sound data based on the Gaussian mixture model; comparing the feature parameters of the second covariance matrix acquired in each iteration with the feature parameters of the fourth covariance matrix acquired in each iteration; correcting the second covariance matrix and the fourth covariance matrix according to the comparison result; the method of acquiring the target eigenvector of the third covariance matrix comprises: performing eigenvalue decomposition on the third covariance matrix to acquire a plurality of eigenvalues and a plurality of eigenvectors, one eigenvalue corresponding to one eigenvector; defining the eigenvector corresponding to the eigenvalue meeting the preset condition as the target eigenvector; the method of comparing the feature parameters of the second covariance matrix acquired in each iteration with the feature parameters of the fourth covariance matrix acquired in each iteration and correcting the second covariance matrix and the fourth covariance matrix according to the comparison result comprises: after each iteration process, performing eigenvalue decomposition on the second covariance matrix to obtain a plurality of eigenvalues corresponding to the second covariance matrix; multiplying the plurality of eigenvalues corresponding to the second covariance matrix with each other, and taking the obtained product as the feature parameter of the second covariance matrix. Multiplying the eigenvalues corresponding to the fourth covariance matrix with each other, taking the obtained product as a characteristic parameter of the fourth covariance matrix; defining the covariance matrix corresponding to the characteristic parameter with larger eigenvalue product as a fourth covariance matrix R(x+n) of mixed sound data (x+n), and defining the covariance matrix corresponding to the characteristic parameter with smaller eigenvalue product as a second covariance matrix R(n) of pure noise data n.

2. The multi-channel audio data processing method of claim 1, wherein, The calculating the first covariance matrix of the sampled audio data comprises: frame windowing the sampled audio data to obtain multiple frames of audio data; performing fast Fourier transform on the multiple frames of audio data to obtain the sampled audio data in time-frequency domain; substituting the sampled audio data in time-frequency domain into a covariance calculation formula of mixed sound data to obtain the first covariance matrix.

3. The multi-channel audio data processing method of claim 2, wherein, The sampled audio data in time-frequency domain comprises: In the formula, denotes the value of the sampled audio data after the fth fast Fourier transform at time t, v denotes any one of pure noise data n or mixed sound data (x + n), denotes the audio data generated by the real sound source, denotes the propagation function between the real sound source and the microphone; The covariance calculation formula of mixed sound data comprises: ； wherein denotes the covariance of the mixed sound data, denotes the value of the f-th fast Fourier transform of the sampled audio data at time t, denotes the conjugate matrix of T, T denotes the audio time length of the sampled audio data.

4. The multi-channel audio data processing method of claim 1, wherein, The method for calculating the Gaussian mixture distribution coefficient comprises: ； wherein denotes the Gaussian mixture distribution coefficient, M denotes the number of channels of the microphone in the multi-channel microphone array, tr( ) denotes the trace of a matrix, denotes the value of the sampled audio data after the fast Fourier transform of the fth at time t, denotes the conjugate matrix of denotes the speech arrival vector; The method for calculating the parameter probability comprises: ； wherein denotes the parametric probability, denotes is the probability density function of the pure noise data n or is the probability density function of the mixed sound data (x + n), denotes the sum over is the probability density function of the pure noise data n or is the probability density function of the mixed sound data (x + n). The method for calculating the speech arrival vector comprises: ； wherein denotes the speech arrival vector, denotes the parameter probability, denotes the sum of the parameter probabilities over all time instants, denotes the Gaussian mixture distribution coefficient, denotes the value of the sampled audio data after a fast Fourier transform at time instant t and for the f-th, denotes the conjugate matrix of The covariance calculation formula of pure noise data comprises: ； wherein denotes the covariance of the pure noise data, denotes the value of the f-th fast Fourier transform of the sampled audio data at time t, denotes the conjugate matrix of denotes the audio time length of the sampled audio data; denotes the parameter probability of pure noise data.

5. The multi-channel audio data processing method of claim 1, wherein, taking the target feature vector and the sampled audio data as inputs of a beamforming algorithm to obtain target audio data.

6. A multi-channel audio data processing apparatus, characterized by comprising: comprises: a data acquisition module configured to acquire sampled audio data collected by a multi-channel microphone array; the sampled audio data comprises pure noise data and mixed sound data; a data processing module configured to calculate a first covariance matrix of the sampled audio data; a model generation module configured to obtain a second covariance matrix of pure noise data based on a Gaussian mixture model and the first covariance matrix, and to obtain a third covariance matrix of pure speech data according to a matrix difference obtained by subtracting the second covariance matrix from the first covariance matrix; an audio processing module configured to obtain a target feature vector of the third covariance matrix, and to synthesize target audio data according to the target feature vector and the sampled audio data; the model generation module is further configured to obtain an initial setting value of a speech arrival vector of the pure noise data; calculate a Gaussian mixture distribution coefficient of the pure noise data according to the initial setting value; calculate a parameter probability in the Gaussian mixture model of the pure noise data according to the Gaussian mixture distribution coefficient; calculate a new speech arrival vector of the pure noise data according to the Gaussian mixture distribution coefficient and the parameter probability; repeat the iteration process of calculating a new Gaussian mixture distribution coefficient and a new parameter probability by using the new speech arrival vector until the iteration converges; substitute the parameter probability of the pure noise data at the time of iteration convergence into a covariance calculation formula of pure noise data to obtain a second covariance matrix; in the iteration process of calculating a new Gaussian mixture distribution coefficient and a new parameter probability by using the new speech arrival vector until the iteration converges: obtain a fourth covariance matrix of mixed sound data based on the Gaussian mixture model; comparing the eigenvalue of the second covariance matrix obtained in each iteration with the eigenvalue of the fourth covariance matrix obtained in each iteration; correcting the second covariance matrix and the fourth covariance matrix according to the comparison result; the obtaining of the target eigenvector of the third covariance matrix comprises: performing eigenvalue decomposition on the third covariance matrix to obtain a plurality of eigenvalues and a plurality of eigenvectors, one eigenvalue corresponding to one eigenvector; defining the eigenvector corresponding to the eigenvalue meeting the preset condition as the target eigenvector; the comparing the eigenvalue of the second covariance matrix obtained in each iteration with the eigenvalue of the fourth covariance matrix obtained in each iteration; and the correcting the second covariance matrix and the fourth covariance matrix according to the comparison result, comprise: after each iteration, performing eigenvalue decomposition on the second covariance matrix to obtain a plurality of eigenvalues corresponding to the second covariance matrix; multiplying the plurality of eigenvalues corresponding to the second covariance matrix with each other, and taking the obtained product as the eigenvalue of the second covariance matrix; multiplying the plurality of eigenvalues corresponding to the fourth covariance matrix with each other, and taking the obtained product as the eigenvalue of the fourth covariance matrix; defining the covariance matrix corresponding to the eigenvalue with larger eigenvalue product value as the fourth covariance matrix R(x+n) of the mixed sound data (x+n), and defining the covariance matrix corresponding to the eigenvalue with smaller eigenvalue product value as the second covariance matrix R(n) of the pure noise data n. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The processor executes the computer program to realize the steps of the multi-channel audio data processing method in any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the multi-channel audio data processing method in any one of claims 1 to 5.

9. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the multi-channel audio data processing method in any one of claims 1 to 5.

Images