Beamforming method, device and equipment
The beamforming method using a directional microphone array addresses the robustness issues of omnidirectional arrays by integrating amplitude responses and steering vectors, enhancing performance across frequency bands.
Patent Information
- Application Number
- CN202210995421.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-18
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-08-18
AI Technical Summary
Existing differential beamforming based on omnidirectional microphone arrays suffers from low robustness due to varying white noise gain at low frequencies and decreased directivity factor at high frequencies, leading to inconsistent beam patterns.
A beamforming method that utilizes a directional microphone array to determine target direction information, microphone pointing direction information, and microphone array shape information, incorporating amplitude responses and steering vectors to enhance beamforming performance.
The method improves white noise gain at low frequencies and directivity factor at high frequencies, resulting in a more frequency-invariant beam pattern and enhanced robustness.
Smart Images

Figure CN115515038B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech processing technology, and in particular to a beamforming method and device, and a sound pickup device. Background Art
[0002] Internet technology has brought about changes in people's communication tools, and audio and video conferencing systems based on cloud computing have gradually become popular. In noisy and reverberant environments, audio and video conferencing terminals use beamforming technology to achieve high-quality sound pickup, so beamforming in video conferencing environments has always been a hot topic of research.
[0003] Microphone array-based beamforming can enhance the desired direction of the target signal and suppress interference from undesired directions. A typical beamforming is differential beamforming using an omnidirectional microphone array. White noise gain (WNG) and directivity factor (DF) are two important indicators for evaluating the effectiveness of beamforming. White noise gain is used to measure the robustness of the beamformer, and directivity factor is used to evaluate the array performance in the presence of reverberation.
[0004] However, in the process of implementing the present invention, the inventors found that the existing solutions have at least the following problems: the beam pattern will change with frequency, specifically: the white noise gain (WNG) at low frequencies is low, and the directivity factor (DF) at high frequencies decreases. Therefore, the differential beamforming based on the omnidirectional microphone array has the problem of low robustness. Summary of the invention
[0005] The present application provides a beamforming method to solve the problem of low robustness of differential beamforming based on an omnidirectional microphone array. The present application also provides a beamforming device and a sound pickup device.
[0006] The present application provides a beamforming method for a sound pickup device, wherein the sound pickup device includes a directional microphone array, and the method includes:
[0007] Obtain target direction information, microphone pointing direction information and microphone array shape information;
[0008] Determine a response vector of the microphone array to the target direction according to the microphone pointing direction information and the target direction information;
[0009] Determining a first steering vector of the microphone array in the target direction according to the microphone array shape information and the target direction information;
[0010] A weight vector of a differential beamformer is determined according to the response vector in the target direction and the first steering vector in the target direction.
[0011] Optionally, the directional microphone array includes a linear array;
[0012] The microphone pointing direction information includes an angle between the microphone direction and the end-fire direction.
[0013] Optionally, the directional microphone array includes a linear array composed of directional microphones with non-uniform directions.
[0014] Optionally, determining a response vector of the microphone array to the target direction according to the microphone pointing direction information and the target direction information includes:
[0015] Get the angle between the target direction and the direction in which the microphone is pointing;
[0016] According to the angle, the amplitude response of each microphone to the target direction is obtained.
[0017] Optionally, determining a response vector of the microphone array to the target direction according to the microphone pointing direction information and the target direction information includes:
[0018] A response vector of the microphone array to the target direction is determined according to the microphone pointing direction information, the target direction information and the coefficient of the first-order directional microphone.
[0019] Optionally, also include:
[0020] Set different first-order directional microphone coefficients for different directional microphones.
[0021] Optionally, determining a first steering vector of the microphone array to the target direction according to the microphone array shape information and the target direction information includes:
[0022] According to the shape information of the microphone array, the distance value between each microphone and the reference microphone is obtained;
[0023] A first steering vector in the target direction is acquired according to the distance value and the target direction.
[0024] Optionally, determining a weight vector of a differential beamformer according to the response vector in the target direction and the first steering vector in the target direction includes:
[0025] Determining a second steering vector in the target direction according to the response vector in the target direction and the first steering vector in the target direction;
[0026] The weight vector is determined according to the second steering vector in the target direction.
[0027] Optionally, also include:
[0028] Acquire a direction vector and a constraint response vector corresponding to the direction vector, wherein the direction vector includes a target direction and a plurality of non-target directions;
[0029] Determining a response vector of the microphone array to the non-target direction according to the microphone pointing direction information and the non-target direction information;
[0030] Determine a first steering vector of the microphone array in the non-target direction according to the microphone array shape information and the non-target direction information;
[0031] The step of determining a weight vector of a differential beamformer according to the response vector in the target direction and the first steering vector in the target direction comprises:
[0032] A weight vector of a differential beamformer is determined according to the response vector of each incident direction, the first steering vector of each incident direction and the constrained response vector.
[0033] Optionally, determining a weight vector of a differential beamformer according to a response vector of each incident direction, a first steering vector of each incident direction, and the constrained response vector includes:
[0034] Determine a second steering vector for each incident direction according to the response vector for each incident direction and the first steering vector for each incident direction;
[0035] The weight vector is determined according to a plurality of second steering vectors corresponding to different incident directions and the constraint response vector.
[0036] The present application also provides a beamforming device, including:
[0037] An information acquisition unit, used to acquire target direction information, microphone pointing direction information and microphone array shape information;
[0038] A target direction response vector determination unit, used to determine a response vector of the microphone array to the target direction according to the microphone pointing direction information and the target direction information;
[0039] A target direction first steering vector determining unit, used to determine a first steering vector of the microphone array in the target direction according to the microphone array shape information and the target direction information;
[0040] The weight vector determining unit is used to determine the weight vector of the differential beamformer according to the response vector in the target direction and the first steering vector in the target direction.
[0041] Optionally, the device may further include:
[0042] A parameter setting unit, used to obtain a direction vector and a constraint response vector corresponding to the direction vector, wherein the direction vector includes a target direction and a plurality of non-target directions;
[0043] A non-target direction response vector determination unit, used to determine a response vector of the microphone array to the non-target direction according to the microphone pointing direction information and the non-target direction information;
[0044] A non-target direction first steering vector determining unit, used to determine a first steering vector of the microphone array in the non-target direction according to the microphone array shape information and the target direction information;
[0045] The weight vector determination unit is specifically configured to determine the weight vector of the differential beamformer according to the response vector of each incident direction, the first steering vector of each incident direction and the constrained response vector.
[0046] Optionally, the weight vector determination unit is specifically used to determine the second steering vector of each incident direction based on the response vector of each incident direction and the first steering vector of each incident direction; and determine the weight vector based on multiple second steering vectors corresponding to different incident directions and the constraint response vector.
[0047] The present application also provides a sound pickup device, comprising:
[0048] speaker;
[0049] Directional microphone array;
[0050] Processor; and
[0051] The memory is used to store a program for implementing the above beamforming method. The terminal is powered on and runs the program of the method through the processor.
[0052] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the computer-readable storage medium is run on a computer, the computer executes the above-mentioned various methods.
[0053] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the above-mentioned various methods.
[0054] Compared with the prior art, this application has the following advantages:
[0055] The beamforming method provided in the embodiment of the present application obtains target direction information, microphone pointing direction information and microphone array shape information; determines the response vector of the microphone array to the target direction according to the microphone pointing direction information and the target direction information; determines the first steering vector of the microphone array to the target direction according to the microphone array shape information and the target direction information; determines the weight vector of the differential beamformer according to the response vector of the target direction and the first steering vector of the target direction. This processing method enables differential beamforming based on a directional microphone array, and adds an amplitude response obtained based on the microphone pointing direction information to the differential beamforming, so that a higher white noise gain can be obtained in the low frequency band, a higher directivity factor can be obtained in the high frequency band, and the beam pattern is more frequency invariant; therefore, the robustness of differential beamforming can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1 A schematic diagram of a flow chart of an embodiment of a beamforming method provided by the present application;
[0057] Figure 2 A schematic diagram of a scenario of an embodiment of a beamforming method provided in the present application;
[0058] Figure 3 A schematic diagram of a microphone array of an embodiment of a beamforming method provided in the present application;
[0059] Figure 4 A schematic structural diagram of an embodiment of a beamforming device provided in the present application. DETAILED DESCRIPTION
[0060] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.
[0061] In the present application, a beamforming method and device, and a conference terminal are provided. Various solutions are described in detail one by one in the following embodiments.
[0062] First embodiment
[0063] Please see Figure 1 , which is a flow chart of an embodiment of the beamforming method of the present application. In this embodiment, the method may include the following steps:
[0064] Step S101: Acquire target direction information, microphone pointing direction information and microphone array shape information.
[0065] The method provided in the embodiment of the present application is used for a sound pickup device, which includes but is not limited to an audio and video conference terminal, such as a speakerphone. The sound pickup device includes a directional microphone array, which includes but is not limited to: a cardioid, a supercardioid, a gun-shaped, a bidirectional type, and the microphone array can be such as Figure 2 The linear array shown may also be an array of other geometric shapes, such as a square array, a triangular array, a circular array, an L-shaped array, etc., or an array of irregular geometric shapes.
[0066] Superdirective Beamforming is a spatial filter algorithm based on microphone array. The so-called spatial filtering is to set a target direction. Signals within the target direction will be picked up, while signals outside the target direction will be suppressed. The target direction is the desired direction. Beamforming can show a distortion-free response in the desired direction and a certain distortion response in the undesired direction. The target direction can be a set direction. Based on the beamforming algorithm, the microphone array can pick up sounds in a specific direction range, while sounds outside the specific direction will be suppressed.
[0067] The differential array processes data by comparing the spatial sound pressure differences of the signals collected by adjacent array elements in the array. Its response is the spatial derivative of the sound pressure field, that is, the response of the N-order differential array can be seen as a linear combination of the zero- to N-order spatial derivatives of the sound pressure. The existing differential beamforming method uses a uniform linear array of omnidirectional microphones for differential beamforming, which only uses the array shape information of the omnidirectional microphone array. The method provided in the embodiment of the present application, on the basis of the existing differential beamforming method, also includes the directional characteristics of the microphone, and integrates this acoustic characteristic parameter (i.e., the microphone directional direction information) into the calculation of the differential beamforming, so it is necessary to obtain the microphone directional direction information.
[0068] Directional microphones have different sensitivities to sounds from different directions. Figure 3 As shown in FIG. 1 , the microphone pointing direction information is related to the microphone orientation and the geometric shape of the array. Taking a linear array as an example, the microphone pointing direction information is the direction of the microphone and the geometric shape of the array. Figure 2 The angle between the end-fire directions shown. The pointing direction of each microphone in the array can be different, rather than all microphone pointing directions being unified as perpendicular to one side of the array. For ease of description, this embodiment refers to a linear array of directional microphones with different pointing directions as a linear array of directional microphones with non-uniform pointing directions. Taking a circular array as an example, the pointing direction of the microphone is the sum of the angle between the microphone and the horizontal direction and the angle between the microphone direction and the diameter direction, that is, the angle between the microphone direction and the horizontal direction. The angle between the direction of each microphone in the array and the diameter direction can be the same or different.
[0069] The array shape information is related to the geometric shape of the array. Taking a linear array as an example, Figure 2 As shown, the array shape information may include information such as the distance between microphones. In this embodiment, the distances between different adjacent microphones may be the same or different. Taking a circular array as an example, the array shape information may include information such as the radius, the angle between the microphone and the horizontal direction of the microphone array, etc.
[0070] Step S103: determining a response vector of the microphone array to the target direction according to the microphone pointing direction information and the target direction information.
[0071] The method provided by the present application is a beamforming method based on a directional microphone array. When the directional microphone collects signals, for each microphone itself, after collecting sounds from different directions, the directional microphone itself will have an amplitude response (also called amplitude response) caused by the angle between the microphone pointing direction and the signal incident direction. If different microphones point to different directions, then for a speech signal in a certain direction, there will be different amplitude responses caused by different angles between the microphones. The amplitude response is caused by the amplitude difference brought by the directional microphone itself, which is related to the orientation of the directional microphone and has nothing to do with the position difference between the microphones. The response vector includes the amplitude responses corresponding to multiple microphones.
[0072] In specific implementation, step S103 can be implemented in the following manner: obtaining the angle between the target direction and the direction in which the microphone is pointing, which is also called the off-axis angle of the incident signal relative to the microphone; and obtaining the amplitude response of the directional microphone to the target direction based on the angle.
[0073] In one example, step S103 can be implemented in the following manner: according to the microphone pointing direction information, the target direction information and the coefficient of the first-order directional microphone, the response vector of the microphone array to the target direction is determined. The coefficient of the first-order directional microphone can be represented by p, p=0 forms a dipole beam pattern, p=0.5 forms a cardioid beam pattern, and p=1 is an omnidirectional microphone. Using a first-order directional microphone can effectively save hardware costs and reduce the size of the microphone array.
[0074] In the case of using a first-order directional microphone, the step of obtaining the amplitude response of the directional microphone to the target direction according to the angle can be performed in the following manner: obtaining the cosine value of the angle between the target direction and the direction in which the microphone is pointing; and obtaining the amplitude response of the directional microphone to the target direction according to the cosine value. In specific implementation, the amplitude response of the directional microphone can be calculated using the following formula:
[0075] u m(p,θ)=p+(1-p)cos(θ-α m )
[0076] In this formula, u m (p,θ) represents the amplitude response of the mth directional microphone to the sound signal from the θ direction, θ represents the incident direction of the signal (which can be the target direction or the non-target direction), α m is the pointing direction of the mth directional microphone. In this case, the response vector can be expressed as follows:
[0077] u(p,θ)=[u1,u2,…,u m ,…,u M ]
[0078] In this formula, M indicates that the microphone array includes M microphones.
[0079] It can be seen that when a first-order directional microphone is used, when the incident direction of the signal is α m When they are consistent, the angle between the two is 0, and the amplitude response of the microphone to the signal in the incident direction is 1; when the signal incident direction is perpendicular to the direction in which the microphone is pointing, the angle is 90 degrees, and the amplitude response is p; when the angle between the signal incident direction and the direction in which the microphone is pointing is between 0 and 90 degrees, the amplitude response is between p and 1. If the coefficient of the first-order directional microphone is p = 0, a dipole beam pattern is formed, then the amplitude response is between 0 and 1. If the coefficient of the first-order directional microphone is p = 0.5, a cardioid beam pattern is formed, then the amplitude response is between 0.5 and 1.
[0080] In one example, the method may further include the following steps: setting different first-order directional microphone coefficients p for different directional microphones. By controlling p for each directional microphone individually, such as p=1, 0.5 or 0, more degrees of freedom can be introduced, which is suitable for some special cases.
[0081] It should be noted that the method provided in the embodiment of the present application is not limited to a first-order directional microphone, but can also achieve high-order directionality. The amplitude response matrix of the directional microphone is a high-order response matrix.
[0082] Step S105: determining a first steering vector of the microphone array in the target direction according to the microphone array shape information and the target direction information.
[0083] The first steering vector (array steering vector) represents the phase difference relationship between different microphones in the array and the relationship between the directions of the sound source waves. The first steering vector can describe the spatial phase difference. The structure of the first steering vector is related to the relative position between the array elements. It is the steering vector of the traditional omnidirectional microphone array.
[0084] In one example, the directional microphone array is a linear array with non-uniform directional characteristics, and step S105 can be implemented in the following manner: based on the shape information of the microphone array, the distance value between each microphone and the reference microphone is obtained, and the reference microphone can be the microphone ranked first in the linear array from left to right; based on the distance value and the target direction, the first steering vector is obtained.
[0085] In specific implementation, the phase difference of the mth directional microphone in the direction θ can be expressed by the following formula:
[0086]
[0087] In this formula, θ represents the incident direction of the signal, represents the imaginary unit, ω=2πf represents the angular frequency, f represents the time frequency (also called the timing frequency or the flashing frequency, referred to as the time frequency); δ is the distance between adjacent microphones, and (m-1)δ represents the distance between the mth microphone and the first microphone. In this case, the first steering vector can be expressed as follows:
[0088]
[0089] In this formula, M indicates that the microphone array includes M microphones, and T indicates the transpose operation of the matrix.
[0090] In another example, the directional microphone array is a uniform circular array, and step S105 can be implemented in the following manner: acquiring the first steering vector according to the microphone array radius information and the target direction information.
[0091] Step S107: determining a weight vector of a differential beamformer according to the response vector in the target direction and the first steering vector in the target direction.
[0092] Multi-channel beamforming is the process of adding weighted coefficients to the signals received by each microphone channel. The beam pattern is such that signals within the target direction are picked up, while signals outside the target direction are suppressed. The beam pattern is the product of the beamformer's weight vector and the steering vector.
[0093] In the prior art, the beam pattern is the product of the weight vector of the beamformer and the first steering vector. For each array element, assuming that the amplitude of the signal at a moment is the same, different weight vectors are used for weighted processing, and only the phase of the signal is adjusted without changing the amplitude.
[0094] In the method provided in the embodiment of the present application, the directional characteristics of the microphone are included, and this acoustic characteristic parameter is integrated into the calculation of beamforming. This step adds a degree of freedom on the basis of the existing differential beamforming, that is, based on the amplitude response obtained based on the microphone directional information, the weight vector of the beamformer is determined according to the response vector and the steering vector (i.e., the first steering vector) of the traditional omnidirectional microphone array to form the above-mentioned beam pattern.
[0095] In one example, step S107 may include the following sub-steps:
[0096] Step S1071: Determine a second steering vector in the target direction according to the response vector in the target direction and the first steering vector in the target direction.
[0097] The second steering vector is not only related to the phase difference, but also to the amplitude response caused by the microphone pointing direction. The second steering vector can be expressed as follows:
[0098] d(ω,θ)=[d1,d2,…,d m ,…,d N ] T
[0099] Wherein, M indicates that the microphone array includes M microphones.
[0100] In a specific implementation, if the microphone array is a linear array composed of directional microphones with non-uniform directions, the element d in the second steering vector m (ω,θ) can be calculated using the following formula:
[0101]
[0102] The following relationship can be obtained from this formula: the second steering vector of the microphone array can be the basic product of the response vector and the first steering vector, which can be formally expressed as:
[0103] d(ω,θ)=u(p,θ)oa(ω,θ)
[0104] Among them, u(p,θ)=[u1,u2,…,u m ,…,u M ]
[0105]
[0106] Step S1073: Determine the weight vector according to the second steering vector in the target direction.
[0107] In this embodiment, the beam pattern is the product of the weight vector of the beamformer and the second steering vector. For each array element, the amplitude of the signal at a moment is not the same, and different weight vectors are used for weighted processing, which not only adjusts the phase of the signal but also changes the amplitude. The product of the beamforming weight vector h(ω) and the second steering vector d(ω,θ) is 1, and the signal in a specific direction is not distorted, and the microphone array can achieve sound pickup in a specific direction range; the product is 0, and the signal outside the specific direction is distorted, and the microphone array can achieve that the sound outside the specific direction will be suppressed.
[0108] The traditional design of a differential beamformer based on a microphone array is based on a multi-order differential array (DMA), where an n-order DMA is obtained by subtracting the outputs of two N-1-order DMAs. However, this method is usually very difficult to design DMAs higher than the second order. To this end, a null-constrained method can be used to design a robust differential beamformer with maximum WNG. In this embodiment, differential beamforming is performed based on this null-constrained method. Accordingly, the method provided in the embodiment of the present application may also include the following steps:
[0109] Step S201: acquiring a direction vector and a constraint response vector corresponding to the direction vector, wherein the direction vector includes a target direction and a plurality of non-target directions.
[0110] In this embodiment, in order to design a differential beamformer h(ω) with zero constraints, the constraints can be expressed as a system of linear equations as follows:
[0111] R(ω,θ)h(ω)=c θ
[0112] Among them, the constraint matrix of size NXM is R(ω,θ) which can be expressed as follows:
[0113]
[0114] d H (ω,θ n ) is a steering vector of length M, n = 1, 2, ..., N;
[0115] θ=[θ1 θ2…θ N ] T
[0116] c θ =[c θ1 c θ2 …c θN ] T
[0117] Wherein, θ represents the direction vector, which is the signal incident angle parameter vector, θ1 represents the signal incident direction 1 (such as 0 degrees), θ2 represents the signal incident direction 2 (such as 5 degrees), and θ N Indicates the signal incident direction N (such as 90 degrees), θ1 to θ N Can be the design parameter of the differential beamformer; c θ represents the constraint response vector. Θ and c θ are vectors of length N, vectors Θ and c θ are the design parameters of the beamformer. θ1,…,θ N defines the desired or zero direction, c θ1 ,…,c θN are the corresponding responses in these directions. For example, if c θ1 =1, then the steering vector with a signal incident angle of θ1 follows d H (ω,θ1)h(ω)=1, indicating that θ1 is the desired direction of the beamformer. For the sound signal from direction θ1, the beamformer produces a distortion-free output. For other directions (θ2…θ N ), c θi = 0, that is, these θ i It can be seen that the characteristics of the beamformer are determined by the constraint vector c that needs to be specified in the design. θ and the angle parameter vector θ.
[0118] Step S203: determining a response vector of the microphone array to the non-target direction according to the microphone pointing direction information and the non-target direction information.
[0119] Step S205: determining a first steering vector of the microphone array in the non-target direction according to the microphone array shape information and the non-target direction information.
[0120] Step S203 corresponds to the above step S103, and step S205 corresponds to the above step S105. The only difference is that the signal incident direction is different, so it will not be repeated here.
[0121] Accordingly, step S107 may be implemented in the following manner: determining a weight vector of a differential beamformer according to the response vectors of each incident direction, the first steering vector, and the constrained response vector.
[0122] In specific implementation, step S107 may include the following sub-steps: determining the second steering vector of each incident direction based on the response vector of each incident direction and the first steering vector of each incident direction; determining the weight vector based on multiple second steering vectors corresponding to different incident directions and the constraint response vector.
[0123] The design of the above differential beamformer can be formulated as the following optimization problem:
[0124] Optimization function minh H (ω)h(ω)
[0125] Constraint R(ω,θ)h(ω)=c θ
[0126] In this case, the beamformer can be obtained as follows:
[0127] h(ω)=R H (ω,θ)[R(ω,θ)R H (ω,θ)] -1 c θ
[0128] If the directional microphone array is a linear array of directional microphones based on non-uniform directivity (LA-NSDM), the corresponding beamformer can be obtained as follows:
[0129] h LA-NSDM (ω)=R H (ω,θ)[R(ω,θ)R H (ω,θ)] -1 c θ
[0130] In the method provided in the embodiment of the present application, the directional characteristics of the microphone are included. This acoustic characteristic parameter is incorporated into the calculation of beamforming, which can be expressed as follows:
[0131] R(ω,θ)=U(p,θ)°A(ω,θ)
[0132] U(p,θ) is the directional microphone response matrix, which can be expressed as follows:
[0133]
[0134] A(ω,θ) is the conventional far-field steering vector of an omnidirectional microphone and can be expressed as follows:
[0135]
[0136] The weight vector h(ω) is the desired parameter and can be defined as follows:
[0137] h(ω)=[H1(ω),H2(ω),…,H m (ω),…,H M (ω)] T
[0138] The beam pattern can be expressed as follows:
[0139]
[0140] In this formula, H represents the conjugate transpose, θ desired Indicates the target direction (specific direction). This beam pattern means that the microphone array can pick up sound in a specific direction range, and sounds outside the specific direction will be suppressed.
[0141] In this embodiment, the weight vector can be implemented using the following formula:
[0142] h(ω)=A H (ω,θ)°U H (p,θ)[(U(p,θ)°A(ω,θ))(A H (ω,θ)°U H (p,θ))] -1 c θ
[0143] Wherein, A represents the first steering vector matrix of the microphone array, U represents the response vector matrix of the directional microphone array; o represents the basic product operation; [] H represents a conjugate matrix.
[0144] The vector θ in the above formula is the design parameter of the beamformer, θ1 to θ N Indicates the target direction or an empty direction. θ represents the response corresponding to the direction. For example, θ1 is the target direction, c θ1 =1 means the beam pattern on θ1 is 1, producing distortion-free output; c θi =0, i=2,…,N,θ i Determines the null value of the beam pattern. By the constraint vector c θ and the angle parameter vector θ to determine the properties of the beamformer, and the constraint vector c needs to be set in advance θ and the angle parameter vector θ.
[0145] From the above content, it can be seen that the beamforming method provided in the embodiment of the present application expands the existing differential beamforming method by introducing a degree of freedom, which is the directional microphone response matrix u(p,θ). In fact, when the microphone response matrix decays to a matrix (p=1 omnidirectional microphone), the existing differential beamforming method is a special case of the differential beamforming method provided in the embodiment of the present application.
[0146] For the differential beamforming method provided in the embodiment of the present application, the effect of the method can be evaluated by using white noise gain (WNG), beam pattern and directivity factor (DF). White noise refers to noise whose power spectral density is constant in the entire frequency domain. Random noise with the same energy density at all frequencies is called white noise. WNG can evaluate the ability of the beamformer to suppress spatially irrelevant noise, and can evaluate the sensitivity of the beamformer to certain defects, such as the noise generated by the directional microphone itself, microphone position error, etc. The ability of the beamformer to suppress spatially irrelevant noise is related to the weight vector of the beamforming. WNG can be defined using the following formula:
[0147]
[0148] The beam pattern describes the directional sensitivity of the beamformer to a plane wave incident on the array from an angle of incidence θ. The beam model can be defined using the following formula:
[0149]
[0150] Where B represents the beam pattern in the spherical coordinate system, θ represents the azimuth angle, Indicates the elevation angle.
[0151] DF is the ratio of the signal power at the array output in the desired steering vector to the average power in all directions and can be defined as follows:
[0152]
[0153] DL[h(ω)]=10log10(DF[h(ω)])
[0154] Experiments have shown that the method provided in the present application can significantly improve the white noise gain at low frequencies while obtaining the same DF index as the prior art, and the beam pattern is more frequency invariant, thereby improving the robustness of differential beamforming.
[0155] It can be seen from the above embodiments that the beamforming method provided by the embodiments of the present application obtains target direction information, microphone pointing direction information and microphone array shape information; determines the response vector of the microphone array to the target direction according to the microphone pointing direction information and the target direction information; determines the first steering vector of the microphone array to the target direction according to the microphone array shape information and the target direction information; determines the weight vector of the differential beamformer according to the response vector of the target direction and the first steering vector of the target direction. This processing method enables differential beamforming based on a directional microphone array, and adds an amplitude response obtained based on the microphone pointing direction information to the differential beamforming, so that a higher white noise gain can be obtained in the low frequency band, a higher directivity factor can be obtained in the high frequency band, and the beam pattern is more frequency invariant; therefore, the robustness of differential beamforming can be effectively improved.
[0156] Second embodiment
[0157] In the above-mentioned embodiment, a beamforming method is provided, and correspondingly, the present application also provides a beamforming device. The device corresponds to the embodiment of the above-mentioned method. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.
[0158] Please see Figure 4 , which is a schematic diagram of the structure of an embodiment of the beamforming device of the present application. In this embodiment, the device includes: an information acquisition unit 401, a target direction response vector determination unit 403, a target direction first steering vector determination unit 405, and a weight vector determination unit 407.
[0159] An information acquisition unit is used to acquire target direction information, microphone pointing direction information and microphone array shape information; a target direction response vector determination unit is used to determine the response vector of the microphone array to the target direction based on the microphone pointing direction information and the target direction information; a target direction first steering vector determination unit is used to determine the first steering vector of the microphone array to the target direction based on the microphone array shape information and the target direction information; a weight vector determination unit is used to determine the weight vector of the differential beamformer based on the response vector of the target direction and the first steering vector of the target direction.
[0160] In one example, the device further includes: a parameter setting unit, a non-target direction response vector determination unit, and a non-target direction first steering vector determination unit.
[0161] A parameter setting unit is used to obtain a direction vector and a constraint response vector corresponding to the direction vector, wherein the direction vector includes a target direction and multiple non-target directions; a non-target direction response vector determination unit is used to determine the response vector of the microphone array to the non-target direction according to the microphone pointing direction information and the non-target direction information; a non-target direction first steering vector determination unit is used to determine the first steering vector of the microphone array to the non-target direction according to the microphone array shape information and the target direction information; correspondingly, the weight vector determination unit is specifically used to determine the weight vector of the differential beamformer according to the response vector of each incident direction, the first steering vector of each incident direction and the constraint response vector.
[0162] In one example, the weight vector determination unit is specifically used to determine the second steering vector of each incident direction based on the response vector of each incident direction and the first steering vector of each incident direction; and determine the weight vector based on multiple second steering vectors corresponding to different incident directions and the constraint response vector.
[0163] The directional microphone array includes but is not limited to a linear array. The directional microphone array includes but is not limited to a linear array composed of directional microphones with non-uniform directions. If a linear array is used, the microphone directional direction information includes the angle between the microphone direction and the end-fire direction.
[0164] In one example, the target direction response vector determination unit is specifically configured to obtain an angle between the target direction and the direction in which the microphone is pointing; and obtain an amplitude response of each microphone to the target direction based on the angle.
[0165] In one example, the target direction response vector determination unit is specifically configured to determine a response vector of the microphone array to the target direction according to the microphone pointing direction information, the target direction information and the coefficient of the first-order directional microphone.
[0166] In one example, the apparatus may further include: a microphone property setting unit, configured to set different first-order directional microphone coefficients for different directional microphones.
[0167] In one example, the target direction first steering vector determination unit is specifically used to obtain the distance value between each microphone and the reference microphone according to the microphone array shape information; and obtain the first steering vector of the target direction according to the distance value and the target direction.
[0168] In one example, the weight vector determination unit is specifically used to determine the second steering vector of the target direction according to the response vector of the target direction and the first steering vector of the target direction; and determine the weight vector according to the second steering vector of the target direction.
[0169] Third embodiment
[0170] In the above-mentioned embodiment, a beamforming method is provided, and correspondingly, the present application also provides a sound pickup device. The device corresponds to the above-mentioned method embodiment. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.
[0171] The present application further provides a sound pickup device, comprising: a speaker; a directional microphone array; a processor; and a memory, wherein the memory is used to store a program for implementing the above-mentioned beamforming method, and the terminal is powered on and runs the program of the method through the processor.
[0172] The electronic device may be an audio or video conferencing terminal or a sound pickup device.
[0173] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.
[0174] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0175] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0176] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.
[0177] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
Claims
1. A beamforming method for a sound pickup device, wherein the sound pickup device comprises a directional microphone array, characterized in that: include: Obtain target direction information, microphone pointing direction information and microphone array shape information; According to the microphone pointing direction information and the target direction information, obtaining the angle between the target direction and the microphone pointing direction; According to the angle, the amplitude response of each microphone to the target direction is obtained to form a response vector; Determining a first steering vector of the microphone array in the target direction according to the microphone array shape information and the target direction information, wherein the first steering vector describes a spatial phase difference; A second steering vector in the target direction is determined based on the basic product of the response vector in the target direction and the first steering vector in the target direction; a weight vector of a differential beamformer is determined based on the second steering vector in the target direction, so that the product of the beamforming weight vector and the second steering vector is 1, and there is no distortion to the signal in the target direction; and the product of the weight vector and the second steering vector is 0, and there is distortion to the signal outside the target direction.
2. The method according to claim 1, characterized in that: The directional microphone array includes a linear array; The microphone pointing direction information includes an angle between the microphone direction and the end-fire direction.
3. The method according to claim 2, characterized in that The directional microphone array comprises a linear array composed of directional microphones with non-uniform directions.
4. The method according to claim 1, characterized in that: The step of obtaining the amplitude response of each microphone to the target direction according to the angle includes: According to the angle and the coefficient of the first-order directional microphone, the amplitude response of each microphone to the target direction is obtained.
5. The method according to claim 4, characterized in that Also includes: Set different first-order directional microphone coefficients for different directional microphones.
6. The method according to claim 1, characterized in that The step of determining a first steering vector of the microphone array in the target direction according to the microphone array shape information and the target direction information comprises: According to the shape information of the microphone array, the distance value between each microphone and the reference microphone is obtained; A first steering vector in the target direction is acquired according to the distance value and the target direction.
7. The method according to claim 1, characterized in that Also includes: Acquire a direction vector and a constraint response vector corresponding to the direction vector, wherein the direction vector includes a target direction and a plurality of non-target directions; Determining a response vector of the microphone array to the non-target direction according to the microphone pointing direction information and the non-target direction information; Determine a first steering vector of the microphone array in the non-target direction according to the microphone array shape information and the non-target direction information; Determine a second steering vector for each incident direction according to the response vector for each incident direction and the first steering vector for each incident direction; The step of determining a weight vector of a differential beamformer according to the second steering vector of the target direction comprises: The weight vector is determined according to a plurality of second steering vectors corresponding to different incident directions and the constraint response vector.
8. A beamforming device, characterized in that: include: An information acquisition unit, used to acquire target direction information, microphone pointing direction information and microphone array shape information; A target direction response vector determination unit, used to obtain an angle between the target direction and the microphone direction according to the microphone direction information and the target direction information; According to the angle, the amplitude response of each microphone to the target direction is obtained to form a response vector; A target direction first steering vector determining unit, used to determine a first steering vector of the microphone array to the target direction according to the microphone array shape information and the target direction information, wherein the first steering vector describes a spatial phase difference; A weight vector determination unit is used to determine a second steering vector in the target direction according to a basic product of a response vector in the target direction and a first steering vector in the target direction; and to determine a weight vector of a differential beamformer according to the second steering vector in the target direction, so that the product of the beamforming weight vector and the second steering vector is 1, and the signal in the target direction is not distorted; and the product of the weight vector and the second steering vector is 0, and the signal outside the target direction is distorted.
9. The device according to claim 8, characterized in that Also includes: a parameter setting unit, configured to obtain a direction vector and a constraint response vector corresponding to the direction vector, wherein the direction vector includes a target direction and a plurality of non-target directions; A non-target direction response vector determination unit, used to determine a response vector of the microphone array to the non-target direction according to the microphone pointing direction information and the non-target direction information; A non-target direction first steering vector determining unit, used to determine a first steering vector of the microphone array in the non-target direction according to the microphone array shape information and the target direction information; The weight vector determination unit is specifically used to determine the second steering vector of each incident direction based on the response vector of each incident direction and the first steering vector of each incident direction; and determine the weight vector based on multiple second steering vectors corresponding to different incident directions and the constraint response vector.
10. A sound pickup device, characterized in that: include: speaker; Directional microphone array; processor; as well as A memory is used to store a program for implementing the method according to any one of claims 1 to 7, and the sound pickup device is powered on and runs the program of the method through the processor.
