Beamforming method, head-mounted device, and computer storage medium

Abstract

The application discloses a beam forming method, a head-mounted device and a computer storage medium, relates to the technical field of beam forming, and comprises the following steps: acquiring a first steering vector of a target direction, a second steering vector of an interference direction and an original audio signal picked up by a microphone array; inputting the original audio signal into a beam former to obtain an initial beam signal, and determining a spatial power ratio based on the first steering vector and the second steering vector, wherein the spatial power ratio is used for representing the ratio of the power of the original audio signal in the target direction to the power in the interference direction; generating a mask according to the spatial power ratio, wherein the mask is larger when the power of the original audio signal in the target direction is larger; and adjusting the initial beam signal based on the mask to obtain an enhanced beam signal. The application improves the output signal-to-interference ratio of beam forming.

Description

Technical Field

[0001] This application relates to the field of signal processing technology, and in particular to a beamforming method, a head-mounted device, and a computer storage medium. Background Technology

[0002] Microphone array beamforming technology is widely used in smart wearable devices, conferencing systems, and hearing aids. Its basic task is to enhance speech from the target direction and suppress interference from other directions to improve speech clarity. In AR (Augmented Reality) / VR (Virtual Reality) devices, the wearer's near-field speech and the speech of the target in the far field are often picked up simultaneously. How to effectively separate the two is an important requirement in this field.

[0003] Existing beamforming methods maintain high gain in the target direction and attenuate to some extent in the interference direction by designing fixed or adaptive spatial filtering weights. However, in practical near-field mixed scenarios, these methods often fail to adequately suppress near-field interference, especially when the interference energy is strong or the target and interference are close in spatial angle. Significant interference components remain in the output signal, resulting in a low signal-to-interference ratio (SIR). Therefore, a beamforming scheme that can further improve the output SIR is urgently needed. Summary of the Invention

[0004] The main objective of this application is to provide a beamforming method, a head-mounted device, and a computer storage medium, aiming to solve the technical problem of how to improve the output signal-to-interference ratio of beamforming.

[0005] To achieve the above objectives, this application provides a beamforming method, which includes the following steps: The first steering vector in the target direction, the second steering vector in the interference direction, and the raw audio signal picked up by the microphone array are obtained. The original audio signal is input into a beamformer to obtain an initial beam signal, and the spatial power ratio is determined based on the first steering vector and the second steering vector, wherein the spatial power ratio is used to characterize the ratio of the power of the original audio signal in the target direction to the power in the interference direction; A mask is generated based on the spatial power ratio, wherein the spatial power ratio indicates that the larger the power of the original audio signal in the target direction, the larger the mask; The initial beam signal is adjusted based on the mask to obtain an enhanced beam signal.

[0006] In one embodiment, the step of determining the space power ratio based on the first steering vector and the second steering vector includes: The original audio signal is converted to the time-frequency domain to obtain a time-frequency speech signal; The squared modulus of the product of the conjugate transpose of the first steering vector and the time-frequency speech signal is taken as the target direction power; The maximum value between the squared modulus of the product of the second guide vector and the time-frequency speech signal and the preset regularization constant is selected as the interference direction power; The ratio between the power in the target direction and the power in the interference direction is defined as the spatial power ratio.

[0007] In one embodiment, the step of generating a mask based on the spatial power ratio includes: The spatial power ratio is input into the activation function, and the generated mask is output.

[0008] In one embodiment, the step of adjusting the initial beam signal based on the mask to obtain an enhanced beam signal includes: Determine the frequency corresponding to each time-frequency unit in the initial beam signal; If the frequency is less than or equal to a preset threshold, the initial beam signal of the time-frequency unit is adjusted using a preset upper limit mask; If the frequency is greater than the preset threshold, the initial beam signal of the time-frequency unit is adjusted using a mask generated based on the spatial power ratio; After each time-frequency unit of the initial beam signal has been adjusted, the enhanced beam signal is obtained.

[0009] In one embodiment, the step of adjusting the initial beam signal based on the mask to obtain an enhanced beam signal includes: The initial beam signal is transformed to the time-frequency domain to obtain the initial beam signal value of each time-frequency unit; The enhanced time-frequency signal value is obtained by multiplying the initial beam signal value of each time-frequency unit with the mask corresponding to the time-frequency unit. The enhanced time-frequency signal value is transformed back to the time domain to obtain the enhanced beam signal.

[0010] In one embodiment, the beamformer is an LCMV beamformer, and the step of inputting the original audio signal into the beamformer to obtain the initial beam signal includes: Obtain the pre-set initial filter coefficients; The original audio signal is input into the LCMV beamformer, which performs adaptive filtering starting from the initial filter coefficients and outputs the filtered initial beam signal.

[0011] In one embodiment, prior to the step of obtaining the pre-set initial filter coefficients, the method further includes: The interference speech signal picked up by the microphone array is obtained, wherein the interference speech signal is the signal picked up by the microphone array when the speech is emitted by the sound source at the preset interference position; The autocorrelation matrix of the interfering speech signal is used as the initial covariance matrix; Based on the initial covariance matrix, the predetermined constraint matrix, and the response vector, the initial filter coefficients are set.

[0012] In one embodiment, the step of obtaining the pre-set initial filter coefficients includes: Obtain the relative distance between each microphone in the microphone array; Input the relative distances into the assumed model of the diffuse noise field and output the initial covariance matrix; Based on the initial covariance matrix, the predetermined constraint matrix, and the response vector, the initial filter coefficients are set.

[0013] In addition, to achieve the above objectives, this application also provides a head-mounted device, characterized in that it includes: a microphone array and a processing unit, the processing unit being used to perform the steps of the beamforming method as described above.

[0014] In addition, to achieve the above objectives, this application also provides a readable storage medium, which is a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps of the beamforming method as described above.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the beamforming method described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: This application embodiment obtains a first steering vector in the target direction and a second steering vector in the interference direction, and combines this with the original audio signal picked up by a microphone array to obtain an initial beam signal through a beamformer. Simultaneously, the spatial power ratio is determined using the first and second steering vectors. This ratio reflects the power of the original audio signal in the target direction to the power in the interference direction. Based on this, a mask is generated according to the spatial power ratio; the larger the spatial power ratio, the larger the mask. This means that when the power in the target direction is higher than the power in the interference direction, the mask value is closer to 1, thus preserving a stronger initial beam signal. Conversely, when the power in the interference direction is relatively strong, the spatial power ratio decreases, and the mask becomes smaller accordingly, thus adaptively attenuating residual interference components in the initial beam signal. By adjusting the initial beam signal based on the mask to obtain an enhanced beam signal, this scheme introduces a post-processing stage dynamically related to the spatial power ratio on the basis of spatial filtering in traditional beamformers. This allows the output signal to automatically adjust its gain according to the actual power comparison between the target and the interference, effectively suppressing residual interference that is difficult to eliminate using traditional methods, thereby improving the output signal-to-interference ratio of the beamformer. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic flowchart of the first embodiment of the beamforming method of this application; Figure 2 This is a comparative schematic diagram of beamforming results according to an embodiment of the beamforming method of this application; Figure 3 This is a schematic diagram illustrating the filter coefficient setting method according to an embodiment of the beamforming method of this application; Figure 4 This is a schematic diagram illustrating another filter coefficient setting method according to an embodiment of the beamforming method of this application; Figure 5 This is a schematic diagram of the system architecture involved in an embodiment of the head-mounted device of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Beamforming technology, by weighted combining multiple signals acquired by a microphone array, can suppress signals from non-target directions and enhance signals from the target direction, thereby achieving focused sound pickup in a specific direction, effectively improving the signal-to-noise ratio of the received signal and achieving noise reduction. The Linearly Constrained Minimum Variance (LCMV) algorithm, a classic adaptive beamforming method in array signal processing, minimizes output power by optimizing the array weight vector under preset linear constraints, thus enhancing the target signal while suppressing interference and noise. Its core constraints include maintaining unity gain in the target direction and creating deep nulls in the interference direction, and it can flexibly support multi-target and multi-interference constraints. This algorithm obtains the optimal weights by solving a constrained convex optimization problem, possessing strong anti-interference capabilities and is widely used in microphone arrays, radar, and communication antenna systems, making it a mainstream method for achieving directional pickup and interference cancellation. Nulling technology is essentially a form of beamforming, which effectively suppresses interference signals from these directions by artificially creating nulls or low-gain regions in the direction of interference signals.

[0023] The key to achieving null technology lies in accurately determining the direction of the interference signal and adjusting the array weighting coefficients accordingly. An optimization algorithm is then used to calculate the optimal weighting vector, thereby creating a null in the interference direction while maintaining maximum gain in the target direction. However, in practical applications, it has been found that the null formed by LCMV still has insufficient suppression of near-field interference, especially when the near-field interference energy is strong or the angle between the interference and the target is close. The residual interference component in the output signal is quite noticeable, limiting its performance in mixed near-field and far-field scenarios such as augmented reality / virtual reality devices.

[0024] To address the aforementioned issues, the main solution of this application is as follows: First, a first steering vector in the target direction, a second steering vector in the interference direction, and the original audio signal picked up by the microphone array are obtained; the original audio signal is input into a beamformer to obtain an initial beam signal, and a spatial power ratio is determined based on the first and second steering vectors, wherein the spatial power ratio characterizes the ratio of the power of the original audio signal in the target direction to the power in the interference direction; a mask is generated based on the spatial power ratio, wherein the spatial power ratio indicates that the larger the power of the original audio signal in the target direction, the larger the mask; the initial beam signal is adjusted based on the mask to obtain an enhanced beam signal.

[0025] This application obtains an initial beam signal by acquiring a first steering vector in the target direction and a second steering vector in the interference direction, combined with the original audio signal picked up by a microphone array, and then using a beamformer. Simultaneously, the spatial power ratio is determined using the first and second steering vectors; this ratio reflects the power of the original audio signal in the target direction to the power in the interference direction. Based on this, a mask is generated according to the spatial power ratio; the larger the spatial power ratio, the larger the mask. This means that when the power in the target direction is higher than the power in the interference direction, the mask value is closer to 1, thus preserving a stronger initial beam signal. Conversely, when the power in the interference direction is relatively strong, the spatial power ratio decreases, and the mask becomes smaller accordingly, thus adaptively attenuating residual interference components in the initial beam signal. By adjusting the initial beam signal based on the mask to obtain an enhanced beam signal, this scheme introduces a post-processing stage dynamically related to the spatial power ratio on the basis of spatial filtering in traditional beamformers. This allows the output signal to automatically adjust its gain according to the actual power comparison between the target and the interference, effectively suppressing residual interference that is difficult to eliminate using traditional methods, thereby improving the output signal-to-interference ratio of the beamformer.

[0026] Based on this, this application proposes a beamforming method according to a first embodiment. In this embodiment, reference is made to... Figure 1 As shown, the beamforming method includes the following steps S10~S40: Step S10: Obtain the first steering vector in the target direction, the second steering vector in the interference direction, and the original audio signal picked up by the microphone array; A microphone array consists of multiple microphone units arranged in a specific geometry (such as linear, circular, or spherical) to simultaneously acquire multi-channel audio signals, referred to as the raw audio signal. The target direction usually refers to the direction of the sound source to be picked up, such as the direction of the far-field target speech in front of the AR / VR device; the interference direction refers to the direction of the sound source to be suppressed, such as the direction of the wearer's own near-field speech.

[0027] The first and second steering vectors represent the sound propagation models of the target direction and interference direction relative to the microphone array, respectively. Their elements reflect the relative phase delay and amplitude attenuation of the unit amplitude plane wave or spherical wave received by each microphone. The steering vectors can be pre-calculated based on the array geometry and the orientation of the sound source, or estimated in real time through a sound source localization algorithm.

[0028] Step S20: Input the original audio signal into a beamformer to obtain an initial beam signal, and determine the spatial power ratio based on the first steering vector and the second steering vector, wherein the spatial power ratio is used to characterize the ratio of the power of the original audio signal in the target direction to the power in the interference direction; The beamformer employs conventional fixed beamforming (such as delay summation) or adaptive beamforming (such as LCMV, MVDR) algorithms to perform weighted summation on each channel of the original audio signal, outputting an initial beam signal. This signal has spatially enhanced the target direction and suppressed some directions, but may still retain strong interference components.

[0029] Simultaneously, the spatial power ratio is calculated using the first steering vector and the second steering vector. One feasible approach is to calculate the projected power of the original audio signal in the target direction and the projected power in the interference direction separately. This is achieved by multiplying the covariance matrix of the original audio signal by the conjugate transpose of the first and second steering vectors, then multiplying by the steering vector itself, and finally dividing the power in the target direction by the power in the interference direction to obtain the spatial power ratio. This ratio reflects the relative strength of the target signal energy to the interference signal energy at the current moment (or within a short frame).

[0030] Step S30: Generate a mask based on the spatial power ratio, wherein the spatial power ratio indicates that the larger the power of the original audio signal in the target direction, the larger the mask; The mask is a real value (or a mask vector related to a frequency point) that takes a value between 0 and 1. The way it is generated makes the mask generally show a non-decreasing trend as the spatial power ratio increases: when the spatial power ratio is large, indicating that the target signal is significantly stronger than the interference signal, the mask approaches 1; when the spatial power ratio is small, indicating that the interference signal is dominant, the mask approaches 0.

[0031] A typical implementation involves inputting the spatial power ratio into a smooth mapping function, such as mask = spatial power ratio / (1 + spatial power ratio), or using a piecewise linear function, sigmoid function, etc. This design ensures that the mask value is larger when the power in the target direction is relatively high, thus retaining more of the initial beam signal; conversely, it attenuates the initial beam signal when the power is relatively low.

[0032] Step S40: Adjust the initial beam signal based on the mask to obtain an enhanced beam signal.

[0033] The initial beam signal is multiplied frame-by-frame or sample-by-sample by the generated mask. For example, in frequency domain processing, each frequency point of the initial beam signal can be multiplied by the corresponding mask value; in time domain processing, each sample point of the initial beam signal can be multiplied by a mask that varies over time.

[0034] Through this masking weighting operation, the interference components remaining in the initial beam signal that were not fully suppressed by the beamformer (especially strong near-field interference or interference at close angles) are significantly attenuated due to their relatively small spatial power at the corresponding time or frequency. Meanwhile, the target signal components are preserved or even relatively enhanced due to their relatively large spatial power. The final output signal is the enhanced beam signal, which has a certain improvement in signal-to-interference ratio compared to the initial beam signal.

[0035] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, the step of determining the spatial power ratio based on the first steering vector and the second steering vector includes: Step A10: Convert the original audio signal to the time-frequency domain to obtain a time-frequency speech signal; The original audio signal is a multi-channel time-domain signal acquired by a microphone array, with each channel corresponding to one microphone. To perform power analysis and masking in the frequency domain, a time-frequency transformation is first performed on each channel signal. Commonly used transformation methods include the Short-Time Fourier Transform (STFT), which frames and windows the time-domain signal, then uses the Fast Fourier Transform (FFT) to obtain complex values ​​at each time-frequency point. The resulting time-frequency speech signal is represented as X(k,n), where k is the frequency index and n is the time frame index. This signal retains both amplitude and phase information, facilitating subsequent conjugate transpose multiplication of the steering vector.

[0036] Step A20: The squared modulus of the product of the conjugate transpose of the first steering vector and the time-frequency speech signal is used as the target direction power; The first guiding vector is denoted as Its conjugate transpose is At each time frequency point, A complex multiplication is performed with the time-frequency speech signal vector X(k,n) (the dimension of which is the number of microphones) to obtain a scalar complex value. This value represents the result of phase alignment and weighted merging of the array signals in the target direction. The square of the modulus of this complex value is then taken. This represents the target direction power at the current time and frequency point. This power value reflects the amount of energy from the target direction in the original audio signal; when there is strong speech in the target direction, this power value is higher.

[0037] Step A30: Select the maximum value between the square of the modulus of the product of the conjugate transpose of the second steering vector and the time-frequency speech signal and the preset regularization constant, as the interference direction power; The second guiding vector is denoted as Its conjugate transpose is This vector reflects the acoustic propagation characteristics of the interference direction (e.g., the direction of the wearer's own near-field speech) relative to the microphone array. At each time-frequency point, Performing a complex multiplication with the time-frequency speech signal vector X(k,n) yields a scalar complex value, which represents the result of phase alignment and weighted merging of the array signals in the target direction. Taking the square of the modulus of this complex value, i.e. This value reflects the magnitude of the energy originating from the direction of interference in the original audio signal. Subsequently, a preset regularization constant ε (usually a small positive number) is introduced, and the squared modulus of the above signal is compared with ε, taking the maximum of the two. And this maximum value is taken as the power in the direction of interference.

[0038] The purpose of this design is to allow for direct use when the actual energy in the interference direction is extremely low or even zero. Using the denominator would cause the subsequent spatial power ratio to tend to infinity, resulting in numerical instability or masking errors. By taking the maximum value, it is ensured that the power in the interference direction is at least a regularization constant, thus ensuring that the spatial power ratio always has an upper bound, enhancing the numerical robustness and environmental adaptability of the algorithm.

[0039] Step A40: The ratio between the target direction power and the interference direction power is determined as the spatial power ratio.

[0040] Dividing the obtained target direction power by the obtained interference direction power yields the spatial power ratio γ(k,n), i.e.: .

[0041] This ratio quantifies the relative strength of the energy in the target direction and the energy in the interference direction at the current time and frequency point. When the power in the target direction is significantly greater than the power in the interference direction, the spatial power ratio is much greater than 1; when the power in the target direction is comparable to the power in the interference direction, the ratio is close to 1; and when the power in the target direction is much less than the power in the interference direction, the ratio approaches 0. It should be noted that because a regularization constant is introduced into the denominator, even if the power in the interference direction is zero, the ratio will not diverge, but will be equal to a finite value of the target direction power divided by the regularization constant.

[0042] The spatial power ratio reflects the degree to which the target signal occupies the original audio signal relative to the interference signal. The larger the value, the more dominant the target direction is, and the larger the mask generated subsequently. This enables adaptive gain adjustment of the initial beam signal, effectively suppressing residual interference and improving the output signal-to-interference ratio.

[0043] In one possible implementation, the step of generating a mask based on the spatial power ratio includes: Step B10: Input the spatial power ratio into the activation function and output the generated mask.

[0044] The spatial power ratio is a scalar value reflecting the relative strength of the power in the target direction and the power in the interference direction at the current time and frequency point. Its value ranges from 0 to positive infinity, according to the calculation method in step A40 (approaching 0 when the denominator is a regularization constant and the numerator is very small, and approaching 1 when the numerator is much larger than the denominator). To generate a mask with clear physical meaning, this ratio is mapped to an interval suitable for multiplicative adjustment of the initial beam signal; typically, the mask's value range is designed to be between 0 and 1.

[0045] The activation function achieves this mapping. Let the spatial power ratio be denoted as γ(k,n) and the mask as M(k,n), then M(k,n) = σ(γ(k,n)), where σ() is an activation function. This function satisfies the following: when the spatial power ratio is large (i.e., the power in the target direction far exceeds the power in the interference direction), the mask approaches 1, thus preserving the initial beam signal; when the spatial power ratio is small (i.e., the power in the interference direction is dominant), the mask approaches 0, thus effectively attenuating the interference component in the initial beam signal.

[0046] In a preferred embodiment, the activation function expression used is as follows: .

[0047] Here, c1, c2, and c3 are preset hyperparameters. Specifically, c1 is a positive number used to control the minimum value of the denominator, avoiding mask saturation caused by an excessively small denominator; it is usually taken to be close to 1, for example, c1=1, in which case the function form is similar to the standard Sigmoid but with an offset. c2 is a scaling parameter, controlling the steepness of the mapping curve: the larger c2 is, the more drastic the mask changes near γ(k,n)=1, and the more sensitive it is to distinguish the relative strength of the target and the interference; the smaller c2 is, the smoother the mapping. c3 is an offset constant used to adjust the lower limit of the mask, ensuring that the mask is not completely zero even when the spatial power ratio is extremely low, thereby preserving some background signal or avoiding discontinuities in the output signal; c3 is usually taken to be a small positive number close to 0.

[0048] The activation function has the following characteristics: When the spatial power ratio γ(k,n) is much greater than 1 (i.e., the power in the target direction significantly exceeds the power in the interference direction), the exponent term approaches 0, and M≈1 / c1+c3. If c1=1 and c3=0, the mask approaches 1. When γ(k,n) is much less than 1 (interference is dominant), the exponent term approaches infinity, the denominator approaches infinity, and the mask approaches c3. When γ(k,n)=1 (the target and interference power are equal), the exponent term is exp(0)=1, and the mask value is 1 / (c1+1)+c3, usually designed to be around 0.5. By adjusting c1, c2, and c3, the mapping range, sensitivity, and lower limit of the mask can be flexibly controlled, thereby adapting to different application scenarios and interference suppression requirements. The final generated mask M(k,n) takes values ​​between c3 and 1 / c1+c3. It is usually designed to fall within the [0,1] interval so that it can be multiplied with the initial beam signal to achieve adaptive suppression of residual interference, thereby improving the output signal-to-interference ratio of beamforming.

[0049] In one possible implementation, the step of adjusting the initial beam signal based on the mask to obtain an enhanced beam signal includes: Step C10: Determine the frequency corresponding to each time-frequency unit in the initial beam signal; After time-frequency transformation, the initial beam signal has a distinct frequency index at each time-frequency point. Let the obtained initial beam signal be Y(k,n), where k is the frequency index and n is the time frame index. For each time-frequency unit, its corresponding actual physical frequency value, in Hertz (Hz), is first determined. The determination of the frequency value depends on the parameters of the time-frequency transformation, such as the sampling rate and the number of points in the short-time Fourier transform. This frequency value will serve as the basis for subsequently determining which mask adjustment method to use.

[0050] Step C20: If the frequency is less than or equal to a preset threshold, then the initial beam signal of the time-frequency unit is adjusted using a preset upper limit mask; The preset threshold is a predetermined frequency boundary point. Considering the limitations of beamforming resolution in the low-frequency band, this preset threshold is preferably 1000Hz. When the frequency of a certain time-frequency unit is less than or equal to this preset threshold, it indicates that the current frequency band belongs to the low-frequency range. Within this frequency band, the directional resolution of the microphone array is relatively low, the beamformer's ability to suppress interference is inherently limited, and the energy of near-field interference is usually concentrated in the low-frequency part. If a mask generated based on the spatial power ratio is used directly, the target signal may be distorted or interference suppression may be unstable due to inaccurate estimation of the spatial power ratio. Therefore, a preset upper limit mask is used to adjust the initial beam signal of the time-frequency unit.

[0051] The preset upper limit mask is usually set to 1 (or a constant close to 1), meaning that the initial beam signal of the time-frequency unit is not attenuated or only slightly preserved. The advantage of this design is that it avoids the negative impact of mask estimation errors in the low-frequency band where the beamformer itself has weak directivity, ensuring the integrity of the target low-frequency speech. At the same time, although low-frequency interference exists, the human ear has a relatively high tolerance for low-frequency distortion, so appropriately preserving the original beam output will not significantly affect the final signal-to-interference ratio.

[0052] Step C30: If the frequency is greater than the preset threshold, the initial beam signal of the time-frequency unit is adjusted using a mask generated based on the spatial power ratio. When the frequency of the time-frequency unit exceeds a preset threshold, it indicates that the current frequency band belongs to the mid-to-high frequency range. Within this band, the microphone array has good spatial resolution, and the beamformer can form a narrower beam and a deeper null. At this point, the estimation of the spatial power ratio γ(k,n) is relatively accurate and reliable. Meanwhile, interference components in the mid-to-high frequency band have a more significant impact on speech intelligibility and require effective suppression.

[0053] Therefore, the initial beam signal of the time-frequency cell is adjusted using a mask M(k,n) generated based on the spatial power ratio. Specifically, the initial beam signal Y(k,n) is multiplied by the mask M(k,n). Through this operation, the gain can be dynamically adjusted according to the actual power ratio of the target and the interference in the mid-to-high frequency band: when the target is dominant, the mask is close to 1, and the signal is preserved; when the interference is dominant, the mask is close to 0, and the interference is effectively attenuated.

[0054] Step C40: After each time-frequency unit of the initial beam signal has been adjusted, the enhanced beam signal is obtained.

[0055] For all time-frequency units (i.e., all frequency indices k and time frame indices n) in the initial beam signal, weighted adjustments are made sequentially using either a preset upper limit mask or a dynamically generated spatial power ratio mask, following the judgment logic described above. Once all time-frequency units have completed the above operations, the resulting signal is the enhanced beam signal.

[0056] Subsequently, the enhanced beam signal can be subjected to inverse time-frequency transformation (such as inverse short-time Fourier transform) as needed to restore the time-domain waveform, which can then be output for subsequent applications such as speech recognition, communication, or recording. By distinguishing between low-frequency and mid-to-high-frequency frequencies and employing different masking strategies, while ensuring low-frequency robustness, the spatial resolution capability of the mid-to-high-frequency band is fully utilized to achieve adaptive interference suppression, thereby improving the overall signal-to-interference ratio of beamforming output.

[0057] In one possible implementation, the step of adjusting the initial beam signal based on the mask to obtain an enhanced beam signal includes: Step D10: Transform the initial beam signal to the time-frequency domain to obtain the initial beam signal value for each time-frequency unit; The initial beam signal is a time-domain signal output after beamforming, denoted as y(t). To perform time-frequency cell-by-time masking in the frequency domain, this time-domain signal needs to be transformed to the time-frequency domain. Specifically, Short-Time Fourier Transform (STFT) can be used to frame and window y(t), and then Fast Fourier Transform (FFT) can be used to obtain the complex spectrum value of each time-frequency cell. Let the transformed time-frequency domain signal be Y(k,n).

[0058] Step D20: Multiply the initial beam signal value of each time-frequency unit with the mask corresponding to the time-frequency unit to obtain the enhanced time-frequency signal value; After obtaining the time-frequency representation Y(k,n) of the initial beam signal, for each time-frequency unit, the corresponding mask M(k,n) generated in step B10 or step C20 / C30 is obtained. This mask is a real value, usually between 0 and 1 (or falls within a specific range after parameter design), reflecting the relative energy strength of the target signal and the interference signal in that time-frequency unit.

[0059] The initial beam signal value in the same time-frequency unit is multiplied by the mask using a complex number method, i.e., Y enhanced (k,n) = Y(k,n)M(k,n). Since the mask is a real number, this operation only adjusts the amplitude of the signal, without changing its phase. When the mask is close to 1, the initial beam signal is preserved; when the mask is close to 0, the energy of this time-frequency unit is significantly attenuated. Through this multiplicative weighting of time-frequency units, adaptive suppression of residual interference is achieved while preserving the dominant target signal component.

[0060] Step D30 involves transforming the enhanced time-frequency signal value back to the time domain to obtain the enhanced beam signal.

[0061] After masking and weighting all time-frequency units, the enhanced time-frequency domain signal Y is obtained. enhanced (k,n). To obtain a time-domain waveform that can be directly used for playback or subsequent processing, an inverse transform of this time-frequency representation is required. Specifically, the inverse short-time Fourier transform (ISTFT) is used, combined with the same window function and frame shift parameters as in step D10, to synthesize the time-frequency domain signal into a time-domain signal.

[0062] The time-domain signal output after inverse transformation is the final enhanced beam signal. This signal preserves the speech in the target direction while further suppressing the residual interference components in the initial beam signal, especially in the mid-to-high frequency bands and scenarios with strong interference energy, thus improving the output signal-to-interference ratio.

[0063] For example, in one specific embodiment, the same original audio signal is processed using both a traditional LCMV beamforming algorithm and the beamforming algorithm provided in this application. A comparison of the frequency domain and time domain of the processed signals is shown below. Figure 2 ,exist Figure 2 In the frequency and time domain comparison diagrams shown, the upper half of each sub-graph represents the processing result of the conventional method, and the lower half represents the processing result of the present application's method. The blue boxes indicate near-field interference signals, and the red boxes indicate far-field target signals. In the audio processed by the conventional method, the average RMS amplitude of the interference signal is -42.33 dB, and the average RMS amplitude of the target signal is -44.08 dB, resulting in a signal-to-interference ratio (SIR) of -1.75 dB (i.e., the target signal energy is approximately 1.75 dB lower than the interference signal). In the audio processed by the present application's method, the average RMS amplitude of the interference signal is -48.21 dB, and the average RMS amplitude of the target signal is -48.04 dB, resulting in an SIR of +0.17 dB (the target signal energy is slightly higher than the interference signal by approximately 0.17 dB). Compared to the conventional method, the present application's method improves the output SIR by approximately 1.92 dB. Therefore, the post-filtering method proposed in this application can effectively attenuate residual interference while better preserving the target speech energy, improving the output signal-to-interference ratio of beamforming, and improving speech clarity.

[0064] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the first and second embodiments described above can be referred to the above description and will not be repeated hereafter. Based on this, the beamformer is an LCMV beamformer, and the step of inputting the original audio signal into the beamformer to obtain the initial beam signal includes: Step E10: Obtain the pre-set initial filter coefficients; LCMV (Linearly Constrained Minimum Variance) beamformer is an adaptive filter whose core is to find the optimal weight vector that minimizes the output power while satisfying preset linear constraints (e.g., maintaining unity gain in the target direction and forming nulls in the interference direction).

[0065] To improve the convergence speed and stability of the adaptive algorithm, a set of initial filter coefficients is pre-set as the starting point for iterative optimization. The initial filter coefficients can be set using the weight vector of a conventional fixed beamformer (such as a delay summation beamformer) based on the array geometry and target direction, or they can be taken as empirical values.

[0066] These initial coefficients typically satisfy the linear constraints of LCMV, or serve as a baseline for subsequent adaptive updates, to ensure that the algorithm has basic beam pointing capability in the initial stage.

[0067] Step E20: The original audio signal is input into the LCMV beamformer. The LCMV beamformer performs adaptive filtering starting from the initial filter coefficients and outputs the filtered initial beam signal.

[0068] The multi-channel raw audio signals picked up by the microphone array are input into the LCMV beamformer. Under predefined linear constraints (e.g., the first steering vector constrains the target direction gain to 1, and the second steering vector constrains the interference direction gain to 0 or a minimum value), the beamformer uses the obtained initial filter coefficients as a starting point and employs an adaptive iterative algorithm (such as minimum mean square error (LMS), recursive least squares (RLS), or direct solution of constrained least squares problems) to continuously update the filter coefficients.

[0069] In each iteration, the algorithm adjusts the weight vector based on the error between the output signal and the constraints, gradually reducing the output power (variance) while strictly maintaining the linear constraints. After several iterations or real-time adaptive updates, the filter coefficients converge to the optimal solution.

[0070] The original audio signal is weighted and summed with the coefficients of the optimal filter to output a spatially filtered signal, denoted as the initial beam signal. This signal maintains a high gain in the target direction and forms a deep null in the interference direction. However, due to limitations in actual array resolution and near-field effects, some interference components may still remain, requiring further suppression in subsequent masking post-processing.

[0071] In one possible implementation, prior to the step of obtaining the pre-set initial filter coefficients, the method further includes: Step F10: Obtain the interference speech signal picked up by the microphone array, wherein the interference speech signal is the signal picked up by the microphone array when the speech is emitted by the sound source at the preset interference position; Before the beamformer operates, an interfering speech signal is acquired to initialize the filter coefficients. The preset interference location refers to the known location of the interfering sound source. For example, in AR / VR devices, the wearer's own mouth position can be considered a preset near-field interference location.

[0072] At the preset interference location, a sound source is placed or assumed to emit a speech signal (e.g., playing a standard speech or actual human voice), while a microphone array simultaneously picks up multi-channel signals. The picked-up signal is the interference speech signal, which mainly contains the propagation characteristics from the direction of interference.

[0073] The interfering speech signal can be used to estimate the spatial covariance information of the interference direction, thereby providing a basis for setting the initial filter coefficients.

[0074] Step F20: Use the autocorrelation matrix of the interfering speech signal as the initial covariance matrix; After acquiring the interfering speech signal, its autocorrelation matrix is ​​calculated. Specifically, the multi-channel interfering speech signal is represented as a vector, and its autocorrelation matrix is ​​calculated. This autocorrelation matrix reflects the power distribution of the interfering signal in each channel and the correlation between channels. It is used as the initial covariance matrix for subsequent filter coefficient calculation.

[0075] By using the autocorrelation matrix of the interfering speech signal as the initial covariance matrix, the initial filter coefficients can form preliminary nulls or low gains in the direction of interference, thereby improving the initial performance of adaptive filtering.

[0076] Step F30: Set the initial filter coefficients based on the initial covariance matrix, the predetermined constraint matrix, and the response vector.

[0077] The predetermined constraint matrix and response vector are the specific expressions of the linear constraint conditions of the LCMV beamformer. Each column of the constraint matrix C corresponds to a steering vector for a constraint direction (e.g., target direction and interference direction), and the response vector f specifies the expected gain for each constraint direction (e.g., 1 for the target direction and 0 for the interference direction).

[0078] Based on the obtained initial covariance matrix Rxx, combined with the constraint matrix C and the response vector f, the LCMV optimization problem is solved. This optimization problem can be expressed as: ; Its closed-form solution is: .

[0079] In actual calculations, the initial covariance matrix is ​​substituted into the closed-form solution to obtain the initial filter coefficients based on the assumption of diffuse noise field.

[0080] In one possible implementation, the step of obtaining the pre-set initial filter coefficients includes: Step G10: Obtain the relative distance between each microphone in the microphone array; A microphone array consists of multiple microphone units arranged in a specific geometry (such as linear, circular, or spherical). To construct a model of the diffuse noise field, the relative distance between each pair of microphones in the array is first obtained.

[0081] Specifically, the Euclidean distance between all microphone pairs can be calculated based on the array's geometric parameters (such as the spacing between adjacent microphones, array radius, etc.).

[0082] Step G20: Input the relative distances into the diffusion noise field assumption model and output the initial covariance matrix; A diffuse noise field is an idealized noise model that assumes noise in the environment comes from all directions and is uncorrelated, with its spatial correlation function depending only on the distance between microphones. Commonly used diffuse noise field models include spherical isotropic diffuse fields and cylindrical isotropic diffuse fields.

[0083] Under the assumption of a spherical isotropic diffusion field, the spatial correlation function between the two microphones is: .

[0084] Where f is the frequency, dij is the relative distance between the i-th and j-th microphones, and c is the speed of sound (usually taken as 340 m / s). Substitute all the obtained relative distances into the above formula, calculate the correlation value between all microphone pairs for each frequency point, and form an M×M correlation matrix Γ(f). Use this correlation matrix as the initial covariance matrix.

[0085] Step G30: Set the initial filter coefficients based on the initial covariance matrix, the predetermined constraint matrix, and the response vector.

[0086] Using the same calculation method as in step F30, the initial covariance matrix obtained in step G20 is substituted into the closed-form solution to obtain the initial filter coefficients based on the diffusion noise field assumption. These coefficients, while satisfying spatial constraints, can effectively suppress diffusion noise, providing a good starting point for subsequent adaptive filtering.

[0087] Furthermore, embodiments of this application also propose a head-mounted device, referring to... Figure 5 As shown, the head-mounted device includes a microphone array and a processing unit, the processing unit being used to perform the steps of the beamforming method described above.

[0088] The head-mounted device provided in this application, employing the beamforming method described in the above embodiments, can solve the technical problem of how to improve the output signal-to-interference ratio of beamforming. Compared with the prior art, the beneficial effects of the head-mounted device provided in this application are the same as those of the beamforming method provided in the above embodiments, and other technical features of this head-mounted device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0089] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0090] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0091] In addition, to achieve the above objectives, this application also provides a readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the beamforming method in the above embodiments.

[0092] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0093] The aforementioned computer-readable storage medium may be included in the head-mounted device; or it may exist independently and not assembled into the head-mounted device.

[0094] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a head-mounted device, cause the head-mounted device to perform the process steps of any embodiment of the beamforming method described above.

[0095] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0097] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0098] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described beamforming method, and can solve the technical problem of how to improve the output signal-to-interference ratio of beamforming. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the beamforming method provided in the above embodiments, and will not be repeated here.

[0099] Furthermore, embodiments of this application also propose a computer program product, including a computer program that, when executed by a processor, implements the steps of the beamforming method described above.

[0100] The specific implementation of the computer program product in this application is basically the same as the various embodiments of the beamforming method described above, and will not be repeated here.

[0101] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0102] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software sensor. This computer software sensor is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a head-mounted device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0104] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A beamforming method, characterized in that, The beamforming method includes the following steps: The first steering vector in the target direction, the second steering vector in the interference direction, and the raw audio signal picked up by the microphone array are obtained. The original audio signal is input into a beamformer to obtain an initial beam signal, and the spatial power ratio is determined based on the first steering vector and the second steering vector, wherein the spatial power ratio is used to characterize the ratio of the power of the original audio signal in the target direction to the power in the interference direction; A mask is generated based on the spatial power ratio, wherein the spatial power ratio indicates that the larger the power of the original audio signal in the target direction, the larger the mask; The initial beam signal is adjusted based on the mask to obtain an enhanced beam signal.

2. The beamforming method as described in claim 1, characterized in that, The step of determining the space power ratio based on the first steering vector and the second steering vector includes: The original audio signal is converted to the time-frequency domain to obtain a time-frequency speech signal; The squared modulus of the product of the conjugate transpose of the first steering vector and the time-frequency speech signal is taken as the target direction power; The maximum value between the squared modulus of the product of the second guide vector and the time-frequency speech signal and the preset regularization constant is selected as the interference direction power; The ratio between the power in the target direction and the power in the interference direction is defined as the spatial power ratio.

3. The beamforming method as described in claim 2, characterized in that, The step of generating a mask based on the spatial power ratio includes: The spatial power ratio is input into the activation function, and the generated mask is output.

4. The beamforming method as described in claim 1, characterized in that, The step of adjusting the initial beam signal based on the mask to obtain the enhanced beam signal includes: Determine the frequency corresponding to each time-frequency unit in the initial beam signal; If the frequency is less than or equal to a preset threshold, the initial beam signal of the time-frequency unit is adjusted using a preset upper limit mask; If the frequency is greater than the preset threshold, the initial beam signal of the time-frequency unit is adjusted using a mask generated based on the spatial power ratio; After each time-frequency unit of the initial beam signal has been adjusted, the enhanced beam signal is obtained.

5. The beamforming method as described in claim 1, characterized in that, The step of adjusting the initial beam signal based on the mask to obtain the enhanced beam signal includes: The initial beam signal is transformed to the time-frequency domain to obtain the initial beam signal value of each time-frequency unit; The enhanced time-frequency signal value is obtained by multiplying the initial beam signal value of each time-frequency unit with the mask corresponding to the time-frequency unit. The enhanced time-frequency signal value is transformed back to the time domain to obtain the enhanced beam signal.

6. The beamforming method according to any one of claims 1 to 5, characterized in that, The beamformer is an LCMV beamformer, and the step of inputting the original audio signal into the beamformer to obtain the initial beam signal includes: Obtain the pre-set initial filter coefficients; The original audio signal is input into the LCMV beamformer, which performs adaptive filtering starting from the initial filter coefficients and outputs the filtered initial beam signal.

7. The beamforming method as described in claim 6, characterized in that, Before the step of obtaining the pre-set initial filter coefficients, the method further includes: The interference speech signal picked up by the microphone array is obtained, wherein the interference speech signal is the signal picked up by the microphone array when the speech is emitted by the sound source at the preset interference position; The autocorrelation matrix of the interfering speech signal is used as the initial covariance matrix; Based on the initial covariance matrix, the predetermined constraint matrix, and the response vector, the initial filter coefficients are set.

8. The beamforming method as described in claim 6, characterized in that, The step of obtaining the pre-set initial filter coefficients includes: Obtain the relative distance between each microphone in the microphone array; Input the relative distances into the assumed model of the diffuse noise field and output the initial covariance matrix; Based on the initial covariance matrix, the predetermined constraint matrix, and the response vector, the initial filter coefficients are set.

9. A head-mounted device, characterized in that, The invention is characterized by comprising: a microphone array and a processing unit, wherein the processing unit is configured to perform the steps of the beamforming method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the beamforming method as described in any one of claims 1 to 8.

Images