Beamforming methods, apparatus, devices and computer-readable storage media

Abstract

This invention discloses a beamforming method, apparatus, device, and computer-readable storage medium. The method includes: performing unscented Kalman filtering on the previous frame of a target frame audio signal to obtain a state noise covariance matrix corresponding to the target frame audio signal; determining the state noise covariance matrix as the target noise covariance matrix corresponding to the target frame audio signal when performing beamforming on the target frame audio signal; calculating a target weight matrix for beamforming based on the target noise covariance matrix; and performing beamforming on the target frame audio signal according to the target weight matrix to obtain an output beam. This invention improves the accuracy of calculating the noise covariance matrix in complex environments, thereby improving the adaptive beamforming effect.

Description

Technical Field

[0001] This invention relates to the field of audio processing technology, and in particular to a beamforming method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the development of audio signal processing technology, single-channel speech processing technology and traditional filtering and noise reduction methods can no longer meet the current requirements for speech enhancement and noise reduction. Microphone array technology, due to its strong adaptability in complex acoustic environments, fewer restrictions on sound sources and pickup-related devices, strong control over array directivity, strong spatial filtering effect, and flexible spatiotemporal domain transformation, has been widely studied and applied in audio signal processing.

[0003] Microphone array-based adaptive beamforming algorithms are widely used in fields such as voice call enhancement. The noise covariance matrix is ​​an important parameter in the adaptive beamforming process. Due to the phenomena such as sound wave reflection and reverberation that may occur in actual acoustic scenarios, the noise covariance matrix cannot be accurately estimated during the adaptive beamforming process, thus affecting the processing effect of adaptive beamforming. Summary of the Invention

[0004] The main objective of this invention is to provide a beamforming method, apparatus, device, and computer-readable storage medium, which aims to improve the accuracy of the noise covariance matrix in complex sound environments and enhance the processing effect of adaptive beamforming.

[0005] To achieve the above objectives, the present invention provides a beamforming method, the beamforming method comprising the following steps:

[0006] The state noise covariance matrix corresponding to the target frame audio signal is obtained by performing unscented Kalman filtering on the audio signal of the frame preceding the target frame audio signal.

[0007] The state noise covariance matrix is ​​determined as the target noise covariance matrix corresponding to the target frame audio signal when beamforming is performed on the target frame audio signal.

[0008] The target weight matrix for beamforming is calculated based on the target noise covariance matrix, and beamforming is performed on the target frame audio signal according to the target weight matrix to obtain the output beam.

[0009] Optionally, the step of performing unscented Kalman filtering on the previous frame audio signal of the target frame audio signal to obtain the state noise covariance matrix corresponding to the target frame audio signal includes:

[0010] Multiple sampling points are obtained by sampling the previous frame audio signal of the target frame audio signal according to the unscented transform sampling principle;

[0011] The prediction points are obtained by iteratively updating each of the sampling points, and the prediction point weights corresponding to each prediction point are determined, wherein the prediction point weights include the prediction point mean weights and the prediction point covariance weights.

[0012] Based on each prediction point and its corresponding prediction point weight, calculate the prediction state mean and prediction covariance matrix corresponding to the target frame audio signal.

[0013] Based on the predicted state mean and the predicted covariance matrix, the error covariance matrix and the observation covariance matrix corresponding to the target frame sound signal are calculated.

[0014] The prediction covariance matrix is ​​corrected by the error covariance matrix and the observation covariance matrix to obtain the state noise covariance matrix corresponding to the target frame audio signal.

[0015] Optionally, the step of correcting the prediction covariance matrix based on the error covariance matrix and the observation covariance matrix to obtain the state noise covariance matrix corresponding to the target frame audio signal includes:

[0016] The mean of the predicted state corresponding to the target frame sound signal is obtained by weighting and summing the prediction points according to the mean weight of each prediction point.

[0017] The prediction covariance matrix corresponding to the target frame audio signal is calculated based on the mean of the predicted state, each of the predicted points, and the prediction point covariance weights corresponding to each of the predicted points.

[0018] Optionally, the step of calculating the prediction covariance matrix corresponding to the target frame audio signal based on the mean of the prediction state, each of the prediction points, and the prediction point covariance weights corresponding to each of the prediction points includes:

[0019] The difference matrix is ​​formed by subtracting the mean of the predicted state from each predicted point. The product matrix is ​​obtained by multiplying the difference matrix and the transpose of the difference matrix.

[0020] The intermediate covariance matrix is ​​obtained by weighting the product matrix using the covariance weight matrix composed of the covariance weights of each prediction point.

[0021] Calculate the Gaussian white noise covariance matrix corresponding to the Gaussian white noise in the previous frame audio signal;

[0022] The sum of the intermediate covariance matrix and the Gaussian white noise covariance matrix is ​​calculated to obtain the prediction covariance matrix corresponding to the target sound signal.

[0023] Optionally, the step of correcting the prediction covariance matrix based on the error covariance matrix and the observation covariance matrix to obtain the state noise covariance matrix corresponding to the target frame audio signal includes:

[0024] Multiple transformation points are obtained by performing an unscented transformation on the predicted state mean and the predicted covariance matrix. Multiple observation points are obtained by performing a nonlinear transformation on each of the transformation points through the observation equation in the unscented Kalman filter. The observation point weights corresponding to each of the observation points are determined, wherein the observation point weights include the observation point mean weight and the observation point covariance weight.

[0025] The observation points are weighted and summed using the mean weight of each observation point to obtain the mean observation state of the target frame sound signal.

[0026] The error covariance matrix corresponding to the target frame sound signal is calculated based on the mean of the observed state, each of the observed points, each of the predicted points, the mean of the predicted state, and the covariance weight of each of the observed points.

[0027] The observation covariance matrix corresponding to the target frame sound signal is calculated based on the mean of the observation state, each of the observation points, and the covariance weights of each observation point.

[0028] Optionally, the step of correcting the prediction covariance matrix using the error covariance matrix and the observation covariance matrix to obtain the state noise covariance matrix corresponding to the target frame audio signal includes:

[0029] Calculate the filter gain matrix corresponding to the target frame audio signal based on the error covariance matrix and the observation covariance matrix;

[0030] The correction reference matrix is ​​obtained by multiplying the observation covariance matrix, the filter gain matrix, and the transpose of the filter gain matrix.

[0031] The state noise covariance matrix corresponding to the target frame audio signal is obtained by subtracting the correction reference matrix from the prediction covariance matrix.

[0032] Optionally, the step of calculating the target weight matrix for beamforming based on the target noise covariance matrix includes:

[0033] Based on the constraints of beamforming, the weight calculation formula with the noise covariance matrix as the independent variable is obtained by calculating the cost function of beamforming. The constraints are that the noise covariance matrix is ​​minimized after beamforming and the sound signal is undistorted.

[0034] The target noise covariance matrix is ​​used as the independent variable in the weight calculation formula to calculate the target weight matrix for beamforming.

[0035] To achieve the above objectives, the present invention also provides a beamforming apparatus, the beamforming apparatus comprising:

[0036] The filtering module is used to perform unscented Kalman filtering on the previous frame audio signal of the target frame audio signal to obtain the state noise covariance matrix corresponding to the target frame audio signal;

[0037] The determination module is used to determine the state noise covariance matrix as the target noise covariance matrix corresponding to the target frame sound signal when beamforming is performed on the target frame sound signal;

[0038] The beamforming module is used to calculate the target weight matrix for beamforming based on the target noise covariance matrix, and to perform beamforming on the target frame audio signal according to the target weight matrix to obtain the output beam.

[0039] To achieve the above objectives, the present invention also provides a beamforming apparatus, the beamforming apparatus comprising: a memory, a processor, and a beamforming program stored in the memory and executable on the processor, wherein the beamforming program, when executed by the processor, implements the steps of the beamforming method as described above.

[0040] Furthermore, to achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a beamforming program, which, when executed by a processor, implements the steps of the beamforming method as described above.

[0041] In this invention, the state noise covariance matrix corresponding to the target frame audio signal is obtained by performing unscented Kalman filtering on the previous frame audio signal of the target frame audio signal; the state noise covariance matrix is ​​determined as the target noise covariance matrix corresponding to the target frame audio signal when beamforming is performed on the target frame audio signal; the target weight matrix for beamforming is calculated based on the target noise covariance matrix, and beamforming is performed on the target frame audio signal according to the target weight matrix to obtain the output beam.

[0042] Currently, adaptive beamforming algorithms typically calculate the noise covariance matrix based on the assumption that noise signals are uncorrelated. However, in complex acoustic environments, phenomena such as sound wave reflection and reverberation lead to high correlation of noise signals, making it difficult for adaptive beamforming algorithms to accurately estimate the noise covariance matrix in complex acoustic environments.

[0043] Unscented Kalman filtering estimates the signal state at the current moment based on the signal state at the previous moment, and updates the estimated signal state by observing the signal state at the current moment to obtain the accurate signal state at the current moment. Unscented Kalman filtering also estimates and updates the interference signal simultaneously to obtain the covariance matrix of the true interference signal at the current moment. In this invention, noise signal is treated as interference signal in the audio signal. Unscented Kalman filtering is used to process the audio signal to obtain the covariance matrix of the noise signal in the audio signal. This avoids discussing the correlation between noise signals when calculating the noise covariance matrix, enabling stable and accurate calculation of the noise covariance matrix in complex audio environments, thereby improving the accuracy of the noise covariance matrix in complex audio environments and enhancing beamforming performance. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the hardware operating environment involved in the embodiments of the present invention;

[0045] Figure 2 This is a schematic flowchart of the first embodiment of the beamforming method of the present invention;

[0046] Figure 3 This is a comparison diagram of the effects before and after beamforming according to an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of the functional modules of a preferred embodiment of the beamforming apparatus of the present invention.

[0048] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0050] like Figure 1 As shown, Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention.

[0051] It should be noted that the beamforming device in the embodiments of the present invention can be a device equipped with a microphone array, such as headphones, speakers, etc., or a device that establishes a communication connection with a device equipped with a microphone array, such as a smartphone, personal computer, server, etc., without specific limitations.

[0052] like Figure 1 As shown, the beamforming device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to establish communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM or a stable, non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0053] Those skilled in the art will understand that Figure 1 The device structure shown does not constitute a limitation on the beamforming device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0054] like Figure 1 As shown, the memory 1005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a beamforming program. The operating system is a program that manages and controls the device's hardware and software resources, supporting the operation of the beamforming program and other software or programs. Figure 1 In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the server; and the processor 1001 can be used to call the beamforming program stored in the memory 1005 and perform the following operations:

[0055] The state noise covariance matrix corresponding to the target frame audio signal is obtained by performing unscented Kalman filtering on the audio signal of the frame preceding the target frame audio signal.

[0056] The state noise covariance matrix is ​​determined as the target noise covariance matrix corresponding to the target frame audio signal when beamforming is performed on the target frame audio signal.

[0057] The target weight matrix for beamforming is calculated based on the target noise covariance matrix, and beamforming is performed on the target frame audio signal according to the target weight matrix to obtain the output beam.

[0058] Furthermore, the operation of performing unscented Kalman filtering on the previous frame audio signal of the target frame audio signal to obtain the state noise covariance matrix corresponding to the target frame audio signal includes:

[0059] Multiple sampling points are obtained by sampling the previous frame audio signal of the target frame audio signal according to the unscented transform sampling principle;

[0060] The prediction points are obtained by iteratively updating each of the sampling points, and the prediction point weights corresponding to each prediction point are determined, wherein the prediction point weights include the prediction point mean weights and the prediction point covariance weights.

[0061] Based on each prediction point and its corresponding prediction point weight, calculate the prediction state mean and prediction covariance matrix corresponding to the target frame audio signal.

[0062] Based on the predicted state mean and the predicted covariance matrix, the error covariance matrix and the observation covariance matrix corresponding to the target frame sound signal are calculated.

[0063] The prediction covariance matrix is ​​corrected by the error covariance matrix and the observation covariance matrix to obtain the state noise covariance matrix corresponding to the target frame audio signal.

[0064] Further, the operation of correcting the prediction covariance matrix based on the error covariance matrix and the observation covariance matrix to obtain the state noise covariance matrix corresponding to the target frame audio signal includes:

[0065] The mean of the predicted state corresponding to the target frame sound signal is obtained by weighting and summing the prediction points according to the mean weight of each prediction point.

[0066] The prediction covariance matrix corresponding to the target frame audio signal is calculated based on the mean of the predicted state, each of the predicted points, and the prediction point covariance weights corresponding to each of the predicted points.

[0067] Further, the operation of calculating the prediction covariance matrix corresponding to the target frame audio signal based on the predicted state mean, each of the predicted points, and the prediction point covariance weights corresponding to each of the predicted points includes:

[0068] The difference matrix is ​​formed by subtracting the mean of the predicted state from each predicted point. The product matrix is ​​obtained by multiplying the difference matrix and the transpose of the difference matrix.

[0069] The intermediate covariance matrix is ​​obtained by weighting the product matrix using the covariance weight matrix composed of the covariance weights of each prediction point.

[0070] Calculate the Gaussian white noise covariance matrix corresponding to the Gaussian white noise in the previous frame audio signal;

[0071] The sum of the intermediate covariance matrix and the Gaussian white noise covariance matrix is ​​calculated to obtain the prediction covariance matrix corresponding to the target sound signal.

[0072] Further, the operation of correcting the prediction covariance matrix based on the error covariance matrix and the observation covariance matrix to obtain the state noise covariance matrix corresponding to the target frame audio signal includes:

[0073] Multiple transformation points are obtained by performing an unscented transformation on the predicted state mean and the predicted covariance matrix. Multiple observation points are obtained by performing a nonlinear transformation on each of the transformation points through the observation equation in the unscented Kalman filter. The observation point weights corresponding to each of the observation points are determined, wherein the observation point weights include the observation point mean weight and the observation point covariance weight.

[0074] The observation points are weighted and summed using the mean weight of each observation point to obtain the mean observation state of the target frame sound signal.

[0075] The error covariance matrix corresponding to the target frame sound signal is calculated based on the mean of the observed state, each of the observed points, each of the predicted points, the mean of the predicted state, and the covariance weight of each of the observed points.

[0076] The observation covariance matrix corresponding to the target frame sound signal is calculated based on the mean of the observation state, each of the observation points, and the covariance weights of each observation point.

[0077] Further, the operation of correcting the prediction covariance matrix using the error covariance matrix and the observation covariance matrix to obtain the state noise covariance matrix corresponding to the target frame audio signal includes:

[0078] Calculate the filter gain matrix corresponding to the target frame audio signal based on the error covariance matrix and the observation covariance matrix;

[0079] The correction reference matrix is ​​obtained by multiplying the observation covariance matrix, the filter gain matrix, and the transpose of the filter gain matrix.

[0080] The state noise covariance matrix corresponding to the target frame audio signal is obtained by subtracting the correction reference matrix from the prediction covariance matrix.

[0081] Furthermore, the operation of calculating the target weight matrix for beamforming based on the target noise covariance matrix includes:

[0082] Based on the constraints of beamforming, the weight calculation formula with the noise covariance matrix as the independent variable is obtained by calculating the cost function of beamforming. The constraints are that the noise covariance matrix is ​​minimized after beamforming and the sound signal is undistorted.

[0083] The target noise covariance moment is used as the independent variable in the weight calculation formula to calculate the target weight matrix for beamforming.

[0084] Based on the above structure, various embodiments of the beamforming method are proposed.

[0085] Reference Figure 2 , Figure 2 This is a schematic flowchart of the first embodiment of the beamforming method of the present invention.

[0086] This invention provides an embodiment of a beamforming method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order. In this embodiment, the executing entity of the beamforming method can be a device such as headphones, a personal computer, or a smartphone; this is not limited in this embodiment. For ease of description, the execution entity is omitted from the description of each embodiment. In this embodiment, the beamforming method includes:

[0087] Step S10: Perform unscented Kalman filtering on the previous frame audio signal of the target frame audio signal to obtain the state noise covariance matrix corresponding to the target frame audio signal;

[0088] In this embodiment, the sound signal that needs to be beamformed is called the target frame sound signal, and the sound signal of the frame preceding the target frame sound signal is called the previous frame sound signal.

[0089] Specifically, unscented Kalman filtering is performed on the previous frame of the target frame audio signal to obtain the posterior state estimation covariance matrix, which is also the state noise covariance matrix, corresponding to the target frame audio signal. Unscented Kalman filtering estimates the prior state estimation covariance matrix of the target frame audio signal based on the previous frame audio signal; observation of the target frame audio signal yields the observation state estimation covariance matrix; and the prior state estimation covariance matrix is ​​corrected using the observation state estimation covariance matrix to obtain the accurate posterior state estimation covariance matrix, which is also the state noise covariance matrix, corresponding to the target frame audio signal.

[0090] Furthermore, in one feasible implementation, when the unscented Kalman filter is used for the first time, the state noise covariance matrix can be initialized, that is:

[0091]

[0092] Where X0 is the initial sampling point, The initial mean value.

[0093] Step S20: Determine the state noise covariance matrix as the target noise covariance matrix corresponding to the target frame audio signal when beamforming is performed on the target frame audio signal;

[0094] In this embodiment, the state noise covariance matrix is ​​determined as the noise covariance matrix corresponding to the target frame audio signal when beamforming is performed on the target frame audio signal (hereinafter referred to as the target noise covariance matrix for distinction), so as to perform beamforming on the target frame audio signal according to the target noise covariance matrix.

[0095] Step S30: Calculate the target weight matrix for beamforming based on the target noise covariance matrix, and perform beamforming on the target frame audio signal according to the target weight matrix to obtain the output beam.

[0096] In this embodiment, a beamforming weight matrix (hereinafter referred to as the target weight matrix for distinction) is calculated based on the target noise covariance matrix. The target frame audio signal is then beamformed according to the target weight matrix to obtain the output beam. Specifically, each weight in the target weight matrix corresponds to each microphone element in the microphone array. The output beam is obtained by weighting and summing the audio signals collected by each microphone element at the target time using the weight matrix.

[0097] Specifically, in one feasible implementation, a formula for calculating weights with the covariance matrix as the independent variable can be preset, and the target weight matrix can be calculated by using the target noise covariance matrix as the independent variable in the formula; in another feasible implementation, the target weight matrix can also be calculated using the target noise covariance matrix according to the constraints of beamforming, and there is no limitation here.

[0098] Further, in a feasible implementation, step S30 includes:

[0099] Step S301: Based on the constraints of beamforming, the weight calculation formula with the noise covariance matrix as the independent variable is calculated through the cost function of beamforming. The constraints are that the noise covariance matrix is ​​minimized after beamforming and the sound signal is undistorted.

[0100] In this embodiment, the target weight matrix for beamforming is calculated using the target noise covariance matrix based on the constraints of beamforming.

[0101] Specifically, the constraints are that the noise covariance matrix is ​​minimized after beamforming, and the audio signal is distortion-free. The constraint of minimizing the noise covariance matrix is ​​expressed as:

[0102]

[0103] ω H Let R denote the transpose and conjugate of the weight matrix, ω denote the weight matrix, and R n This represents the noise covariance matrix.

[0104] Specifically, in this embodiment, the cost function of beamforming is differentiated, and the derivative is set to 0 to obtain the expression for the weight matrix when the noise covariance matrix is ​​minimized. Specifically, the cost function of beamforming is:

[0105]

[0106] The expression for the weight matrix when the noise covariance matrix is ​​minimized is:

[0107]

[0108] Substitute the expression for the weight matrix into the constraint condition for distortion-free audio signal. We can obtain:

[0109]

[0110] Combining the above formulas (1) and (2), we obtain the formula for calculating the weight matrix with the noise covariance matrix as the independent variable (hereinafter referred to as the weight calculation formula for distinction):

[0111]

[0112] Step S302: Input the target noise covariance moment as an independent variable into the weight calculation formula to obtain the target weight matrix for beamforming.

[0113] By using the target noise covariance moment as the independent variable in the weight calculation formula, the target weight matrix for beamforming is obtained.

[0114] In this embodiment, the previous frame audio signal of the target frame audio signal is subjected to unscented Kalman filtering to obtain the state noise covariance matrix corresponding to the target frame audio signal; the state noise covariance matrix is ​​determined as the target noise covariance matrix corresponding to the target frame audio signal when beamforming is performed on the target frame audio signal; the target weight matrix for beamforming is calculated based on the target noise covariance matrix, and beamforming is performed on the target frame audio signal according to the target weight matrix to obtain the output beam.

[0115] In this embodiment, the noise signal is used as the interference signal in the sound signal. The sound signal is processed by unscented Kalman filtering to obtain the covariance matrix of the noise signal in the sound signal. This avoids discussing the correlation between noise signals when calculating the noise covariance matrix, so that the noise covariance matrix can be calculated stably and accurately in complex sound environments, thereby improving the accuracy of the noise covariance matrix in complex sound environments and enhancing the beamforming effect.

[0116] Exemplarily, in one feasible implementation, reference is made to Figure 3 , Figure 3 This is a comparison diagram of the beamforming effects according to one embodiment of the present invention, wherein, Figure 3 (1) is the input sound signal before beamforming. Figure 3 (2) The output sound signal after beamforming according to an embodiment of the present invention is compared with... Figure 3 (1) and Figure 3 (2) It can be seen from the signal that, compared with the input sound signal before beamforming, the output sound signal after beamforming in this embodiment has reduced noise and significantly increased signal-to-noise ratio.

[0117] Furthermore, based on the first embodiment described above, a second embodiment of the present invention is proposed. In this embodiment, step S10 includes:

[0118] Step S101: Sample the previous frame audio signal of the target frame audio signal according to the unscented transform sampling principle to obtain multiple sampling points;

[0119] In this embodiment, the state noise covariance matrix corresponding to the target frame audio signal is obtained by unscented Kalman filtering.

[0120] Specifically, the audio signal of the previous frame, acquired at the previous moment, is sampled according to the unscented transform sampling principle to obtain multiple sampling points. In specific implementations, the sampling strategy follows the unscented transform sampling strategy, which can be proportional sampling or proportional interval sampling, and is not limited here.

[0121] Step S102: Iteratively update each of the sampling points to obtain prediction points, and determine the prediction point weights corresponding to each prediction point, wherein the prediction point weights include the prediction point mean weights and the prediction point covariance weights.

[0122] In this embodiment, the state equation of the unscented Kalman filter is used to iteratively update each sampling point to obtain new sample points (hereinafter referred to as prediction points for distinction).

[0123] Specifically, the iterative formula can be:

[0124]

[0125] in, Indicates the prediction point. denoted by , and q represents the non-zero mean of the noise state.

[0126] In this embodiment, the weights corresponding to each prediction point are determined, hereinafter referred to as prediction point weights. These prediction point weights include the prediction point mean weight used to calculate the state mean and the prediction point covariance weight used to calculate the estimated covariance matrix. Specifically, the prediction point mean weight and the prediction point covariance weight can be calculated in the same way, specifically:

[0127]

[0128] Among them, W i (m) W is the weight of the predicted point mean. i (c) The covariance weights of the prediction points are denoted by n, where n represents the index of the prediction point.

[0129] Step S103: Calculate the mean of the predicted state and the prediction covariance matrix corresponding to the target frame sound signal based on each of the predicted points and the prediction point weights corresponding to each of the predicted points.

[0130] Based on each prediction point and its corresponding prediction point weight, the prior state mean of the target frame sound signal is calculated, which is also the prediction state mean and the prior covariance matrix, which is also the prediction covariance matrix.

[0131] Step S104: Based on the predicted state mean and the predicted covariance matrix, calculate the error covariance matrix and the observation covariance matrix corresponding to the target frame sound signal.

[0132] An unscented transformation is performed based on the predicted state mean and the predicted covariance matrix to obtain the error covariance matrix and the observation covariance matrix corresponding to the target frame audio signal, which are used to correct the predicted covariance matrix.

[0133] Step S105: Correct the prediction covariance matrix using the error covariance matrix and the observation covariance matrix to obtain the state noise covariance matrix corresponding to the target frame sound signal;

[0134] In this embodiment, the prediction covariance matrix is ​​corrected based on the error covariance matrix and the observation covariance matrix to obtain the state noise covariance matrix corresponding to the target frame audio signal. Specifically, the correction process may involve: calculating the filter gain matrix corresponding to the target frame audio signal based on the error covariance matrix and the observation covariance matrix; and correcting the prediction covariance matrix using the filter gain matrix and the observation covariance matrix to obtain the state noise covariance matrix corresponding to the target frame audio signal.

[0135] Further, in one feasible embodiment, step S103 includes:

[0136] Step S1031: The prediction points are weighted and summed using the mean weight of the prediction points corresponding to each prediction point to obtain the mean prediction state of the target frame sound signal.

[0137] In this embodiment, the predicted state mean and the predicted covariance matrix are calculated. Specifically, the predicted points are weighted and summed using the weights of their respective predicted point mean values ​​to obtain the predicted state mean corresponding to the target frame audio signal. The calculation formula can be:

[0138]

[0139] Among them, W i (m) As the weight of the predicted points, This is the prediction point.

[0140] Step S1032: Calculate the prediction covariance matrix corresponding to the target frame audio signal based on the mean of the prediction state, each prediction point, and the prediction point covariance weights corresponding to each prediction point.

[0141] The prediction covariance matrix corresponding to the target frame audio signal is calculated based on the predicted state mean, each prediction point, and the corresponding prediction point covariance weights. In a specific implementation, Gaussian white noise can also be considered when calculating the prediction covariance matrix.

[0142] Further, in one feasible embodiment, step S1032 includes:

[0143] Step S10321: Subtract the mean of the predicted state from each of the predicted points to obtain a difference matrix composed of the differences. Calculate the product of the difference matrix and the transpose of the difference matrix to obtain a product matrix. Then, use the covariance weight matrix composed of the covariance weights of each of the predicted points to weight the product matrix to obtain an intermediate covariance matrix.

[0144] In this embodiment, Gaussian white noise is considered when calculating the prediction covariance matrix. For any target point among all prediction points, the difference matrix is ​​formed by subtracting the mean of the prediction state from each prediction point. The product matrix is ​​obtained by multiplying the difference matrix and its transpose. The intermediate covariance matrix is ​​then obtained by weighting the product matrix using the covariance weight matrix formed by the covariance weights of each prediction point. The specific calculation formula can be:

[0145]

[0146] in, For the prediction point, To predict the state mean, W i (c) The covariance weights for the prediction points.

[0147] Step S10323: Calculate the Gaussian white noise covariance matrix corresponding to the Gaussian white noise in the previous frame audio signal;

[0148] Calculate the Gaussian white noise covariance matrix corresponding to the Gaussian white noise in the previous frame of the audio signal.

[0149] Step S10323: Calculate the sum of the intermediate covariance matrix and the Gaussian white noise covariance matrix to obtain the prediction covariance matrix corresponding to the target sound signal.

[0150] The sum of the intermediate covariance matrix and the Gaussian white noise covariance matrix is ​​calculated to obtain the prediction covariance matrix corresponding to the target frame audio signal. The specific calculation formula is as follows:

[0151] P k|k-1 =P′ k|k-1 +Q k

[0152] Among them, Q k This represents the covariance matrix of Gaussian white noise.

[0153] Further, in one feasible embodiment, step S104 includes:

[0154] Step S1041: Perform an unscented transformation on the predicted state mean and the predicted covariance matrix to obtain multiple transformation points; perform a nonlinear transformation on each transformation point through the observation equation in the unscented Kalman filter to obtain multiple observation points; and determine the observation point weight corresponding to each observation point, wherein the observation point weight includes the observation point mean weight and the observation point covariance weight.

[0155] In this embodiment, the predicted state mean and predicted covariance matrix are subjected to unscented transformation to obtain multiple nonlinearly transformed sample points (hereinafter referred to as transformation points for distinction). The observation equation in the unscented Kalman filter is used to perform nonlinear transformation on each transformation point to obtain multiple observation points, and the observation point weights corresponding to each observation point are determined. The observation point weights include the observation point mean weight and the observation point covariance weight.

[0156] Specifically, in one feasible implementation, the specific calculation formula for obtaining multiple transformation points by performing an unscented transformation on the predicted state mean and the predicted covariance matrix can be:

[0157] in, The mean of the predicted state is the mean of the 0th observation point.

[0158] Where i = 1, 2, 3, 4, ..., n, the above formula is used to calculate the first n prediction points.

[0159] Where i = n+1, n+2, n+3, n+4, ..., 2n, that is, the above formula is used to calculate the (n+1)th to 2nth prediction points.

[0160] The specific calculation formula for obtaining multiple observation points by performing a nonlinear transformation on each transformation point using the observation equation in unscented Kalman filtering can be:

[0161]

[0162] Among them, V k This represents Gaussian observation noise.

[0163] Step S1042: Use the average weight of each observation point to perform a weighted summation on each observation point to obtain the average observation state corresponding to the target frame sound signal;

[0164] The mean of the observations at each observation point is weighted and summed to obtain the mean of the observation state corresponding to the target frame audio signal. The specific calculation formula can be:

[0165]

[0166] in, For the observation point, W i (m) The weights are the average values ​​of the observation points.

[0167] Step S1043: Calculate the error covariance matrix corresponding to the target frame sound signal based on the mean of the observation state, each of the observation points, each of the prediction points, the mean of the prediction state, and the covariance weights of each observation point.

[0168] Based on the mean of the observed state, each observed point, each predicted point, the mean of the predicted state, and the covariance weights of each observed point, the error covariance matrix corresponding to the target frame audio signal is calculated. The specific calculation formula can be:

[0169]

[0170] in, The covariance weights for the observation points.

[0171] Step S1044: Calculate the observation covariance matrix corresponding to the target frame sound signal based on the mean of the observation state, each observation point, and the covariance weights of each observation point.

[0172] The observation covariance matrix corresponding to the target frame audio signal is calculated based on the mean of the observation state, each observation point, and the corresponding covariance weights of each observation point. In this embodiment, Gaussian white noise may or may not be considered when calculating the observation covariance matrix.

[0173] In one feasible implementation, Gaussian white noise is considered in the calculation to improve the accuracy of the state noise covariance matrix. The specific calculation process can be as follows: first, calculate the intermediate observation matrix; then, calculate the sum of the intermediate observation matrix and the Gaussian white noise observation covariance matrix to obtain the observation covariance matrix. The formula for calculating the intermediate observation matrix is:

[0174]

[0175] The formula for calculating the observation covariance matrix is:

[0176] P ZZ，k =P′ ZZ，k +R k

[0177] Among them, R k This is the observation covariance matrix of Gaussian white noise.

[0178] Further, in one feasible embodiment, step S106 includes:

[0179] Step S1051: Calculate the filter gain matrix corresponding to the target frame audio signal based on the error covariance matrix and the observation covariance matrix;

[0180] In this embodiment, the filter gain matrix corresponding to the target frame audio signal is calculated based on the error covariance matrix and the observation covariance matrix.

[0181] Specifically, the formula for calculating the filter gain matrix can be:

[0182] K k =P XZ，k / P ZZ，k

[0183] Among them, P XZ，k Let P be the error covariance matrix. ZZ，k To observe the covariance matrix.

[0184] Step S1052: Calculate the product of the observation covariance matrix, the filter gain matrix, and the transpose of the filter gain matrix to obtain the correction reference matrix;

[0185] In this embodiment, the product of the observation covariance matrix, the filter gain matrix, and the transpose of the filter gain matrix is ​​calculated to obtain the correction reference matrix.

[0186] Step S1053: Correct the prediction covariance matrix by subtracting the correction reference matrix from the prediction covariance matrix to obtain the state noise covariance matrix corresponding to the target frame audio signal.

[0187] The state noise covariance matrix corresponding to the target frame audio signal is obtained by correcting the predicted covariance matrix by subtracting the correction reference matrix from the predicted covariance matrix. The specific calculation formula can be:

[0188]

[0189] in, To correct the reference matrix.

[0190] In this embodiment, multiple sampling points are obtained by sampling the previous frame of the target frame audio signal according to the unscented transform sampling principle. Each sampling point is iteratively updated to obtain prediction points, and the prediction point weights corresponding to each prediction point are determined. These prediction point weights include the prediction point mean weight and the prediction point covariance weight. Based on each prediction point and its corresponding weight, the predicted state mean and prediction covariance matrix of the target frame audio signal are calculated. Based on the predicted state mean and prediction covariance matrix, the error covariance matrix and observation covariance matrix of the target frame audio signal are calculated. The prediction covariance matrix is ​​corrected using the error covariance matrix and observation covariance matrix to obtain the state noise covariance matrix of the target frame audio signal. This embodiment controls the error of the noise covariance matrix, thereby improving its accuracy and enhancing beamforming performance.

[0191] Furthermore, embodiments of the present invention also propose a beamforming apparatus, referring to... Figure 4 The beamforming apparatus includes:

[0192] The filtering module 10 is used to perform unscented Kalman filtering on the previous frame audio signal of the target frame audio signal to obtain the state noise covariance matrix corresponding to the target frame audio signal.

[0193] The determining module 20 is used to determine the state noise covariance matrix as the target noise covariance matrix corresponding to the target frame sound signal when beamforming is performed on the target frame sound signal;

[0194] The beamforming module 30 is used to calculate the target weight matrix for beamforming based on the target noise covariance matrix, and to perform beamforming on the target frame audio signal according to the target weight matrix to obtain the output beam.

[0195] Furthermore, the filtering module 10 is also used for:

[0196] Multiple sampling points are obtained by sampling the previous frame audio signal of the target frame audio signal according to the unscented transform sampling principle;

[0197] The prediction points are obtained by iteratively updating each of the sampling points, and the prediction point weights corresponding to each prediction point are determined, wherein the prediction point weights include the prediction point mean weights and the prediction point covariance weights.

[0198] Based on each prediction point and its corresponding prediction point weight, calculate the prediction state mean and prediction covariance matrix corresponding to the target frame audio signal.

[0199] Based on the predicted state mean and the predicted covariance matrix, the error covariance matrix and the observation covariance matrix corresponding to the target frame sound signal are calculated.

[0200] The prediction covariance matrix is ​​corrected by the error covariance matrix and the observation covariance matrix to obtain the state noise covariance matrix corresponding to the target frame audio signal.

[0201] Furthermore, the filtering module 10 is also used for:

[0202] The mean of the predicted state corresponding to the target frame sound signal is obtained by weighting and summing the prediction points according to the mean weight of each prediction point.

[0203] The prediction covariance matrix corresponding to the target frame audio signal is calculated based on the mean of the predicted state, each of the predicted points, and the prediction point covariance weights corresponding to each of the predicted points.

[0204] Furthermore, the filtering module 10 is also used for:

[0205] The difference matrix is ​​formed by subtracting the mean of the predicted state from each predicted point. The product matrix is ​​obtained by multiplying the difference matrix and the transpose of the difference matrix.

[0206] The intermediate covariance matrix is ​​obtained by weighting the product matrix using the covariance weight matrix composed of the covariance weights of each prediction point.

[0207] Calculate the Gaussian white noise covariance matrix corresponding to the Gaussian white noise in the previous frame audio signal;

[0208] The sum of the intermediate covariance matrix and the Gaussian white noise covariance matrix is ​​calculated to obtain the prediction covariance matrix corresponding to the target sound signal.

[0209] Furthermore, the filtering module 10 is also used for:

[0210] Multiple transformation points are obtained by performing an unscented transformation on the predicted state mean and the predicted covariance matrix. Multiple observation points are obtained by performing a nonlinear transformation on each of the transformation points through the observation equation in the unscented Kalman filter. The observation point weights corresponding to each of the observation points are determined, wherein the observation point weights include the observation point mean weight and the observation point covariance weight.

[0211] The observation points are weighted and summed using the mean weight of each observation point to obtain the mean observation state of the target frame sound signal.

[0212] The error covariance matrix corresponding to the target frame sound signal is calculated based on the mean of the observed state, each of the observed points, each of the predicted points, the mean of the predicted state, and the covariance weight of each of the observed points.

[0213] The observation covariance matrix corresponding to the target frame sound signal is calculated based on the mean of the observation state, each of the observation points, and the covariance weights of each observation point.

[0214] Furthermore, the filtering module 10 is also used for:

[0215] Calculate the filter gain matrix corresponding to the target frame audio signal based on the error covariance matrix and the observation covariance matrix;

[0216] The correction reference matrix is ​​obtained by multiplying the observation covariance matrix, the filter gain matrix, and the transpose of the filter gain matrix.

[0217] The state noise covariance matrix corresponding to the target frame audio signal is obtained by subtracting the correction reference matrix from the prediction covariance matrix.

[0218] Furthermore, the beamforming module 30 is also used for:

[0219] Based on the constraints of beamforming, the weight calculation formula with the noise covariance matrix as the independent variable is obtained by calculating the cost function of beamforming. The constraints are that the noise covariance matrix is ​​minimized after beamforming and the sound signal is undistorted.

[0220] The target noise covariance moment is used as the independent variable in the weight calculation formula to calculate the target weight matrix for beamforming.

[0221] The various embodiments of the beamforming apparatus of the present invention can be referred to the various embodiments of the beamforming method of the present invention, and will not be repeated here.

[0222] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a beamforming program, which, when executed by a processor, implements the steps of the beamforming method described below.

[0223] The various embodiments of the beamforming apparatus and computer-readable storage medium of the present invention can be referred to the various embodiments of the beamforming method of the present invention, and will not be repeated here.

[0224] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0225] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0226] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0227] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A beamforming method, characterized by, The beam forming method comprises the following steps: Unscented Kalman filtering is performed on a previous frame sound signal of a target frame sound signal to obtain a state noise covariance matrix corresponding to the target frame sound signal; The state noise covariance matrix is determined as a target noise covariance matrix corresponding to the target frame sound signal when beam forming is performed on the target frame sound signal; A target weight matrix of beam forming is calculated according to the target noise covariance matrix, and the target frame sound signal is beam formed according to the target weight matrix to obtain an output beam; The step of performing unscented Kalman filtering on the previous frame sound signal of the target frame sound signal to obtain the state noise covariance matrix corresponding to the target frame sound signal comprises: A plurality of sampling points are obtained by sampling the previous frame sound signal of the target frame sound signal according to an unscented transformation sampling principle; Each of the sampling points is iteratively updated to obtain a prediction point, and a prediction point weight corresponding to each of the prediction points is determined, wherein the prediction point weight comprises a prediction point mean weight and a prediction point covariance weight; A predicted state mean and a predicted covariance matrix corresponding to the target frame sound signal are calculated according to each of the prediction points and the prediction point weight corresponding to each of the prediction points; An error covariance matrix and an observation covariance matrix corresponding to the target frame sound signal are calculated according to the predicted state mean and the predicted covariance matrix; The predicted covariance matrix is corrected by the error covariance matrix and the observation covariance matrix to obtain the state noise covariance matrix corresponding to the target frame sound signal.

2. The beamforming method of claim 1, wherein, The step of correcting the predicted covariance matrix by the error covariance matrix and the observation covariance matrix to obtain the state noise covariance matrix corresponding to the target frame sound signal comprises: A predicted state mean corresponding to the target frame sound signal is obtained by weighted summation of each of the prediction points according to the prediction point mean weight corresponding to each of the prediction points; The predicted covariance matrix corresponding to the target frame sound signal is calculated according to the predicted state mean, each of the prediction points and the prediction point covariance weight corresponding to each of the prediction points.

3. The beamforming method of claim 2, wherein, The step of calculating the predicted covariance matrix corresponding to the target frame sound signal according to the predicted state mean, each of the prediction points and the prediction point covariance weight corresponding to each of the prediction points comprises: A difference matrix is obtained by subtracting the predicted state mean from each of the prediction points, respectively, a product matrix is calculated by multiplying the difference matrix and a transpose matrix of the difference matrix, and an intermediate covariance matrix is obtained by weighting the product matrix according to a covariance weight matrix composed of the prediction point covariance weight corresponding to each of the prediction points; A Gaussian white noise covariance matrix corresponding to a Gaussian white noise in the previous frame sound signal is calculated; The sum of the intermediate covariance matrix and the Gaussian white noise covariance matrix is calculated to obtain the predicted covariance matrix corresponding to the target frame sound signal.

4. The beamforming method of claim 1, wherein, The step of correcting the prediction covariance matrix based on the error covariance matrix and the observation covariance matrix to obtain the state noise covariance matrix corresponding to the target frame audio signal includes: Multiple transformation points are obtained by performing an unscented transformation on the predicted state mean and the predicted covariance matrix. Multiple observation points are obtained by performing a nonlinear transformation on each of the transformation points through the observation equation in the unscented Kalman filter. The observation point weights corresponding to each of the observation points are determined, wherein the observation point weights include the observation point mean weight and the observation point covariance weight. The observation points are weighted and summed using the mean weight of each observation point to obtain the mean observation state of the target frame sound signal. The error covariance matrix corresponding to the target frame sound signal is calculated based on the mean of the observed state, each of the observed points, each of the predicted points, the mean of the predicted state, and the covariance weight of each of the observed points. The observation covariance matrix corresponding to the target frame sound signal is calculated based on the mean of the observation state, each of the observation points, and the covariance weights of each observation point.

5. The beamforming method of claim 1, wherein, The step of correcting the prediction covariance matrix using the error covariance matrix and the observation covariance matrix to obtain the state noise covariance matrix corresponding to the target frame audio signal includes: Calculate the filter gain matrix corresponding to the target frame audio signal based on the error covariance matrix and the observation covariance matrix; The correction reference matrix is ​​obtained by multiplying the observation covariance matrix, the filter gain matrix, and the transpose of the filter gain matrix. The state noise covariance matrix corresponding to the target frame audio signal is obtained by subtracting the correction reference matrix from the prediction covariance matrix.

6. The beamforming method of any one of claims 1 to 5, wherein, The step of calculating the target weight matrix for beamforming based on the target noise covariance matrix includes: Based on the constraints of beamforming, the weight calculation formula with the noise covariance matrix as the independent variable is obtained by calculating the cost function of beamforming. The constraints are that the noise covariance matrix is ​​minimized after beamforming and the sound signal is undistorted. The target noise covariance matrix is ​​used as the independent variable in the weight calculation formula to calculate the target weight matrix for beamforming.

7. A beam forming apparatus characterized by comprising: The beamforming apparatus includes: The filtering module is used to perform unscented Kalman filtering on the previous frame audio signal of the target frame audio signal to obtain the state noise covariance matrix corresponding to the target frame audio signal; The determination module is used to determine the state noise covariance matrix as the target noise covariance matrix corresponding to the target frame sound signal when beamforming is performed on the target frame sound signal; The beamforming module is used to calculate the target weight matrix for beamforming based on the target noise covariance matrix, and to perform beamforming on the target frame audio signal according to the target weight matrix to obtain the output beam. The filter module is further configured to sample a previous frame of the target frame of sound signals according to a principle of unscented transform sampling to obtain a plurality of sampling points; update each of the sampling points to obtain a prediction point, and determine a prediction point weight corresponding to each of the prediction points, wherein the prediction point weight comprises a prediction point mean weight and a prediction point covariance weight; calculate a prediction state mean and a prediction covariance matrix corresponding to the target frame of sound signals according to each of the prediction points and the prediction point weight corresponding to each of the prediction points; calculate an error covariance matrix and an observation covariance matrix corresponding to the target frame of sound signals according to the prediction state mean and the prediction covariance matrix; and correct the prediction covariance matrix according to the error covariance matrix and the observation covariance matrix to obtain a state noise covariance matrix corresponding to the target frame of sound signals.

8. A beamforming device, characterized by The beamforming device comprises a memory, a processor, and a beamforming program stored on the memory and executable on the processor, and the beamforming program, when executed by the processor, implements the steps of the beamforming method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a beamforming program, and the beamforming program, when executed by the processor, implements the steps of the beamforming method according to any one of claims 1 to 6.

Images