Directional sound pickup method, apparatus, computer device and medium

Abstract

This invention relates to the field of directional sound pickup technology, and discloses a directional sound pickup method, apparatus, computer equipment, and medium. The method includes: acquiring time-domain speech signals collected by at least two microphones and performing time-frequency transformation; calculating the observed phase difference at each frequency point, and determining the target theoretical phase difference based on the user-specified target pickup direction and microphone geometric parameters; encoding the target theoretical phase difference into a query vector, encoding the observed phase difference into a key vector, and generating a spatial confidence mask through a geometric cross-attention mechanism; filtering the frequency domain signal using the mask, and outputting enhanced speech in the target direction after inverse time-frequency transformation. This invention effectively solves the phase winding problem through a physically prior-guided attention mechanism, achieving highly robust and high-fidelity speech pickup in any direction under strong reverberation and low signal-to-noise ratio environments.

Description

Technical Field

[0001] This invention relates to the field of directional sound pickup, and particularly to a directional sound pickup method, apparatus, computer equipment, and medium. Background Technology

[0002] With the increasing prevalence of smart voice devices in complex acoustic environments such as homes, vehicles, and conference rooms, directional voice pickup technology needs to accurately extract speech signals from a specified direction under conditions of strong noise and reverberation. Current mainstream solutions typically combine microphone array beamforming with post-processing filtering based on instantaneous angle of arrival (IDOA). These methods calculate the observed phase difference between microphone pairs and geometrically match it with the theoretical phase difference corresponding to the target direction (e.g., Euclidean distance) to generate a spatial gain mask to suppress components from non-target directions.

[0003] However, this type of method suffers from a fundamental flaw: its spatial matching mechanism relies on rigid geometric distance metrics and idealized phase statistics assumptions (such as Gaussian distribution). In real reverberant environments, multipath effects cause drastic phase response jumps and non-Gaussian characteristics, while the periodicity of the phase (i.e., phase wrapping) makes traditional distance metrics unable to accurately reflect directional similarity. More importantly, existing technologies lack a robust, learnable mechanism to align the physical geometric priors of the sound source direction with the actual observed signal characteristics, making it difficult for the system to distinguish the target speech from interference or reflection components from nearby directions.

[0004] Although deep learning methods have been introduced into the field of speech enhancement, most solutions either completely ignore the physical model or only use directional information as a static feature input, failing to establish a dynamic interaction between physical priors and observational data. Therefore, in complex sound fields, existing directional sound pickup systems generally suffer from insufficient spatial selectivity, severe speech distortion, and poor directional adaptability, making it difficult to meet the practical application requirements of high fidelity, low latency, and directional capability.

[0005] Therefore, there is an urgent need for a new spatial matching mechanism that can be guided by physical geometric priors and achieve robust discrimination of speech in the target direction through a data-driven approach. Summary of the Invention

[0006] This invention provides a directional sound pickup method, apparatus, computer device, and medium, aiming to solve the technical problem in the prior art that it is impossible to distinguish target speech from interference or reflection components from similar directions.

[0007] To achieve the above-mentioned objective, the first aspect of this invention provides a directional sound pickup method, comprising the following steps:

[0008] Acquire time-domain speech signals collected by at least two microphones;

[0009] The time-domain speech signal is subjected to time-frequency transformation to obtain the corresponding frequency-domain signal, and the observed phase difference at each frequency point at each time moment is calculated based on the frequency-domain signal; and,

[0010] Based on the target pickup direction specified by the user and the geometric parameters of the at least two microphones, determine the target theoretical phase difference corresponding to each frequency point;

[0011] The target theoretical phase difference is encoded as a query vector, the observed phase difference is encoded as a key vector, and a spatial confidence mask representing the confidence of speech presence from the target pickup direction is generated based on the similarity between the query vector and the key vector.

[0012] The frequency domain signal is filtered using the spatial confidence mask to obtain the enhanced spectrum of the target pickup direction;

[0013] The enhanced spectrum is subjected to inverse time-frequency transformation, and the pickup result of the target pickup direction is output.

[0014] Further, encoding the observed phase difference into a key vector includes:

[0015] The observed phase difference is mapped using trigonometric functions to generate a first vector containing sine and cosine components;

[0016] The first vector is concatenated with the amplitude spectrum of the frequency domain signal from at least one microphone to obtain the key vector.

[0017] Further, encoding the target theoretical phase difference into a query vector includes:

[0018] The target theoretical phase difference is mapped using trigonometric functions to generate a second vector containing sine and cosine components, which serves as the query vector.

[0019] Further, determining the target theoretical phase difference corresponding to each frequency point based on the user-specified target pickup direction and the geometric parameters of the at least two microphones includes:

[0020] Based on the target pickup direction, sound wave propagation speed, frequency values ​​at each frequency point, and the relative positions between the at least two microphones, the phase difference corresponding to the theoretical propagation delay of the sound source signal to each microphone is calculated and used as the target theoretical phase difference.

[0021] Further, generating a spatial confidence mask representing the confidence of speech presence from the target pickup direction based on the similarity between the query vector and the key vector includes:

[0022] Calculate the dot product of the query vector and the key vector at each frequency point and at each time point to obtain the original similarity score;

[0023] The original similarity scores are normalized to generate a spatial confidence mask with values ​​between zero and one.

[0024] Further, the step of filtering the frequency domain signal using the spatial confidence mask to obtain the enhanced spectrum of the target pickup direction includes:

[0025] The spatial confidence mask is spliced ​​with the amplitude spectrum of the frequency domain signal to obtain joint time-frequency features;

[0026] The joint time-frequency features are input into a pre-trained temporal neural network, which generates a complex ideal ratio mask.

[0027] The frequency domain signal is filtered in the complex domain based on the complex ideal ratio mask to obtain the enhanced spectrum in the target pickup direction.

[0028] Furthermore, the pre-trained temporal neural network is obtained through training in the following manner:

[0029] During the training phase, for each training sample, a target sound pickup direction is randomly sampled from a preset angle range, and the corresponding target sound source reference speech is obtained;

[0030] Based on the sampled target pickup direction and the geometric parameters of the at least two microphones, determine the target theoretical phase difference corresponding to each frequency point;

[0031] The target theoretical phase difference is encoded into a query vector, and the observed phase difference calculated from the frequency domain signal of the training samples is encoded into a key vector;

[0032] Based on the similarity between the query vector and the key vector, a spatial confidence mask is generated;

[0033] The spatial confidence mask is concatenated with the amplitude spectrum of the frequency domain signal of the training sample to obtain joint time-frequency features;

[0034] The joint time-frequency features are input into the temporal neural network to obtain the sound pickup result;

[0035] The loss function is calculated based on the difference between the sound pickup result and the reference speech of the target sound source, and the parameters of the temporal neural network are updated.

[0036] A second aspect of the present invention provides a directional sound pickup device, comprising:

[0037] An acquisition unit is used to acquire time-domain speech signals collected by at least two microphones;

[0038] The first calculation unit is used to perform time-frequency transformation on the time-domain speech signal to obtain the corresponding frequency-domain signal, and to calculate the observed phase difference of each frequency point at each time based on the frequency-domain signal; and,

[0039] The second calculation unit is used to determine the target theoretical phase difference corresponding to each frequency point based on the target pickup direction specified by the user and the geometric parameters of the at least two microphones.

[0040] The encoding similarity calculation unit is used to encode the target theoretical phase difference into a query vector, encode the observed phase difference into a key vector, and generate a spatial confidence mask representing the confidence of speech presence from the target pickup direction based on the similarity between the query vector and the key vector.

[0041] A filtering unit is used to filter the frequency domain signal using the spatial confidence mask to obtain the enhanced spectrum of the target pickup direction;

[0042] The output unit is used to perform inverse time-frequency transformation on the enhanced spectrum and output the pickup result of the target pickup direction.

[0043] A third aspect of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the directional sound pickup method as described in any of the preceding claims.

[0044] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the directional sound pickup method described in any of the preceding claims.

[0045] The directional sound pickup method of the present invention has the following beneficial effects:

[0046] By acquiring time-domain speech signals through multiple microphones, spatial orientation information is provided for subsequent direction determination, solving the problem that a single microphone cannot distinguish the direction of the sound source. Time-frequency transformation processing allows for precise extraction of observed phase differences for different frequency components, offering greater targeting compared to time-domain processing. The theoretical phase difference is determined based on the target pickup direction and microphone geometric parameters, relying on a physical acoustic model rather than idealized statistical assumptions. This maintains stable directional characteristics even in strong reverberation environments, avoiding misjudgment problems common in existing solutions. Encoding the phase difference as a vector and generating a mask based on similarity replaces traditional rigid geometric measurement methods, effectively solving the phase wrapping problem. This allows for adaptive matching of spatial features, resulting in more robust direction determination under strong reverberation and low signal-to-noise ratio conditions. No model retraining is required; simply adjusting the target direction switches the pickup angle, demonstrating strong directional adaptability. The final pickup result, after filtering and inverse time-frequency transformation, exhibits excellent spatial selectivity and low speech distortion, meeting the high-fidelity real-time requirements of intelligent voice devices. Attached Figure Description

[0047] Figure 1 A flowchart illustrating a directional sound pickup method according to an embodiment of the invention;

[0048] Figure 2 This is a schematic block diagram of a directional sound pickup device according to an embodiment of the invention;

[0049] Figure 3 This is a schematic block diagram of a computer device according to an embodiment of the invention. The realization of the object, functional features, and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of features, integers, steps, operations, elements, modules, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any modules and all combinations of one or more associated listed items.

[0052] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0053] Reference Figure 1 This invention provides a directional sound pickup method, comprising the following steps:

[0054] S1: Acquire time-domain speech signals collected by at least two microphones.

[0055] Time-domain speech signals refer to electrical signals continuously recorded over time, derived from sound wave vibrations, with time as the horizontal axis and signal amplitude as the vertical axis. A microphone array is a signal acquisition system composed of two or more microphones arranged in a specific geometric position, capable of capturing speech signals from different spatial locations. In a specific embodiment, a dual-microphone array with a spacing of d=2cm is selected and deployed on a conference terminal device to acquire the speaker's voice from the 0° direction directly in front of the speaker in the conference room, as well as interference noise from the 90° direction to the side, resulting in two time-domain speech signals x1(t) and x2(t), with a sampling rate F. s Set to 16kHz. Signals collected by two or more microphones contain spatial orientation information, providing basic data for subsequent calculation of phase difference and differentiation of sound source direction, thus solving the problem that a single microphone cannot determine the direction of a sound source.

[0056] S2: Perform time-frequency transformation on the time-domain speech signal to obtain the corresponding frequency-domain signal, and calculate the observation phase difference of each frequency point at each time based on the frequency-domain signal.

[0057] Time-frequency transformation is the operation of converting a time-domain signal into a frequency-time two-dimensional signal. This embodiment uses short-time Fourier transform (STFT), which can simultaneously reflect the time and frequency characteristics of the signal. The observed phase difference refers to the phase difference between the same signal acquired by different microphones at the same frequency point and the same time. The calculation formula is as follows: Where ∠X1(f,t) and ∠X2(f,t) are the phases of the two frequency domain signals, respectively. In a specific embodiment, a short-time Fourier transform is performed on x1(t) and x2(t), with a frame length of 512 points and a frame shift of 128 points, to obtain the frequency domain signals X1(f,t) and X2(f,t), where f ranges from 0 to 255 and t is the frame index. For a frequency point f=100 and a time t=50, ∠X1(100,50)=120° and ∠X2(100,50)=30° are calculated, and the observed phase difference Δ is... obs (100,50)=90°. After converting the time-domain signal to the frequency domain, different frequency components can be processed separately, improving the targeting of signal processing; calculating the observed phase difference can extract the spatial orientation features of the signal, providing a basis for subsequent direction matching, which is different from the existing technology that directly uses time-domain signal processing.

[0058] S3: Determine the target theoretical phase difference corresponding to each frequency point based on the target pickup direction specified by the user and the geometric parameters of the at least two microphones.

[0059] The target pickup direction is the azimuth angle θ of the sound source that the user expects to extract from the speech. target The microphone geometric parameters mainly refer to the microphone spacing d; the target theoretical phase difference is the phase difference value of the sound source signal arriving at different microphones in the target direction, calculated based on the acoustic propagation model. The calculation formula is as follows: Where NFFT represents the number of Fourier transform points, and c is the speed of sound, with a value of 340 m / s. In a specific embodiment, the user specifies the target pickup direction as θ. target =0°, microphone spacing d=0.02m, sampling rate F s =16kHz, Fourier transform (NFFT) points = 512. For the frequency point f = 100, substituting into the formula, we get... (Approximately 66.5°). Based on the calculation of theoretical phase difference using a physical acoustic model, without relying on idealized statistical assumptions such as Gaussian distribution in existing technologies, it can maintain stable directional characteristics even in strong reverberation environments, thus solving the problem of high misjudgment rate caused by the idealization of physical models in existing technologies.

[0060] S4: Encode the target theoretical phase difference into a query vector, encode the observed phase difference into a key vector, and generate a spatial confidence mask representing the confidence of speech presence from the target pickup direction based on the similarity between the query vector and the key vector.

[0061] The query vector is a vector obtained by encoding the theoretical phase difference of the target, representing the ideal spatial characteristics of the target direction; the key vector is a vector obtained by encoding the observed phase difference, representing the spatial characteristics of the actual acquired signal.

[0062] The core implementation mechanism for generating the spatial confidence mask based on the similarity calculation between the query vector and the key vector in this step is defined as the geometric cross-attention mechanism in this invention. This geometric cross-attention mechanism is an attention interaction method calculated along the physical feature dimension. Its core is to match the "theoretical spatial fingerprint" (query vector) generated by the physical model with the "real-world state features" (key vector) collected by the sensor in the feature space; essentially, it is physical manifold matching. This geometric cross-attention mechanism differs from traditional self-attention mechanisms calculated along the time dimension. It does not rely on a long-term history cache (KVCache), has low algorithm latency, and only requires information from the current frame to complete spatial filtering.

[0063] The spatial confidence mask is a matrix generated based on the similarity between the query and the key, with values ​​ranging from 0 to 1. It is used to represent the confidence that the signal belongs to the target direction at each frequency point and time.

[0064] In one embodiment, the Δ obtained in the above embodiment theo (f, 0°) is encoded using trigonometric functions as follows: This vector varies only with frequency f and target angle θ target The change, which does not change with time t, represents the physical prior characteristics of the target direction. The Δ obtained in the above embodiments... obs (f,t) is encoded as This vector dynamically changes with frequency f and time t, representing the spatial and energy characteristics of the actual observation. Q(f) and K(f,t) are mapped to the same hidden layer dimension Dmodel=128 through fully connected layers, placing them in the same feature space; the dot product similarity is then calculated. Then, normalization using the Sigmoid function is obtained. This is the spatial confidence mask.

[0065] Traditional methods use Euclidean distance to calculate the difference between theoretical and observed phase differences. When phase differences become entangled, such as between 350° and 10°, the Euclidean distance is considered extremely large. However, the geometric cross-attention mechanism used in this step, through trigonometric function encoding, ensures extremely high vector similarity between the two phase differences, effectively solving the phase entanglement problem. This geometric cross-attention mechanism abandons the Gaussian distribution assumption and rigid Euclidean distance metric of existing technologies. It adaptively learns spatial feature matching rules through a data-driven approach, improving the robustness of direction discrimination in environments with strong reverberation and low signal-to-noise ratio. This mechanism requires no historical buffering, relying only on current frame information, resulting in low algorithm latency, making it suitable for real-time voice interaction devices such as smart speakers and in-vehicle voice assistants. It achieves a decoupled architecture between "spatial geometric matching" and subsequent "temporal smoothing," preserving the stability of the physical model while avoiding the high computational overhead of traditional Transformer models.

[0066] S5: Filter the frequency domain signal using the spatial confidence mask to obtain the enhanced spectrum of the target pickup direction.

[0067] Filtering is a process of weighting a spatial confidence mask with a frequency domain signal to retain signal components in the target direction while suppressing noise and reverberation components in non-target directions. In a specific embodiment, the frequency domain signal X output by the delay-summing beamformer is selected. beam (f,t) is used as the reference signal, and it is multiplied by the AttnMap(f,t) obtained above, i.e. Y enhance (f,t) represents the enhanced spectrum. The filtering method based on spatial confidence masks is targeted and can accurately preserve speech in the target direction, unlike the speech distortion problem caused by indiscriminate filtering in existing technologies.

[0068] S6: Perform inverse time-frequency transformation on the enhanced spectrum and output the pickup result of the target pickup direction.

[0069] The inverse time-frequency transform (ISTFT) is the inverse operation of the short-time Fourier transform, which converts the enhanced spectrum in the frequency domain back to the time-domain speech signal. In a specific embodiment, for Y... enhance Perform an inverse short-time Fourier transform on (f,t), setting the same frame length and frame shift parameters as the STFT, to obtain the time-domain enhanced speech signal y. out (t) represents the sound pickup result in the target pickup direction. The signal processed in the frequency domain is converted back to the time domain to obtain a speech signal that can be directly played or further processed, completing the entire process of directional sound pickup and ensuring the usability of the output signal.

[0070] The directional sound pickup method disclosed in this embodiment employs a geometric cross-attention mechanism. This mechanism is a generalization of the principle behind the series of steps: encoding theoretical phase features into a query vector, encoding observed phase features into a key vector, and generating a spatial confidence mask based on similarity. Essentially, it uses the theoretical phase features generated by the physical acoustic model as the query and the actual observed phase features as the key, achieving dynamic matching between physical priors and observed data through attention interaction. This mechanism differs from the rigid geometric matching methods of existing technologies and also from the shortcomings of pure black-box deep learning models that ignore physical priors, establishing a hybrid architecture of "physical guidance + data-driven". The method in this embodiment can achieve high-fidelity directional sound pickup in environments with strong reverberation and low signal-to-noise ratio, exhibiting strong spatial selectivity and effectively distinguishing speech from the target direction from interference from non-target directions. The method does not rely on complex statistical models, demonstrating strong robustness. Furthermore, the pickup direction can be adjusted by changing the query vector without retraining the model, exhibiting good directional adaptability and meeting the practical application needs of devices such as smart speakers, conference terminals, and in-vehicle voice assistants.

[0071] In one embodiment, encoding the observed phase difference into a key vector includes:

[0072] S401: Perform trigonometric function mapping on the observed phase difference to generate a first vector containing sine and cosine components.

[0073] Trigonometric function mapping converts the phase difference in angular form into sin(Δ). ) and cos(Δ The operation of a two-dimensional vector composed of π / 2 can effectively solve the phase winding problem. Phase winding refers to the phenomenon that when the phase difference exceeds ±π, the angle value will jump (e.g., the angle value of 350° and 10° is large, but the actual phase difference is close), causing traditional distance calculation to fail. In a specific embodiment, for the observed phase difference Δ obtained above... obs (100,50)=90°, performing trigonometric function mapping yields the first vector V1=[sin(90°),cos(90°)]=[1,0]; if the observed phase difference is 350°, the mapped vector is [sin(350°),cos(350°)]≈[-0.17,0.98], which has a high similarity to the vector [sin(10°),cos(10°)]≈[0.17,0.98] mapped from a 10° phase difference, effectively solving the phase entanglement problem. Trigonometric function mapping converts the angular phase difference into a two-dimensional spatial vector, eliminating the numerical jump problem caused by phase entanglement. Compared with the existing technology that directly uses angle values ​​to calculate distance, it significantly improves the robustness of phase features.

[0074] S402: The first vector is concatenated with the amplitude spectrum of the frequency domain signal from at least one microphone to obtain the key vector.

[0075] The amplitude spectrum is the amplitude value of a frequency domain signal, reflecting the energy magnitude of different frequency components. Vector concatenation is the operation of combining multiple vectors into a higher-dimensional vector. For example, concatenating the first vector V1=[1,0] obtained above with the amplitude spectrum |X1(100,50)|=0.8 of X1(100,50) in the above embodiment yields the key vector K(100,50)=[1,0,0.8]. After concatenating the amplitude spectrum information, the key vector simultaneously contains the spatial direction features of the phase and the energy features of the signal, enabling the subsequent attention mechanism to combine signal-to-noise ratio information to determine the target direction signal, thus improving the accuracy of direction determination in low signal-to-noise ratio environments.

[0076] This embodiment addresses the phase entanglement problem in existing technologies by processing the observed phase difference through trigonometric function mapping. Simultaneously, it enriches the feature dimensions of the key vector by splicing amplitude spectrum information, forming the basis of the geometric perception cross-attention mechanism. This ensures that the key vector accurately represents the spatial and energy characteristics of the actual acquired signal, significantly different from the rigid geometric matching methods of existing technologies. The key vector generation method in this embodiment effectively handles the phase entanglement problem and improves the robustness of phase features. Combined with amplitude spectrum information, the key vector's feature representation capability is stronger, making subsequent similarity calculations more accurate, thereby improving the overall effect of directional sound pickup, especially suitable for complex acoustic scenarios with strong reverberation and low signal-to-noise ratio.

[0077] In one embodiment, encoding the target theoretical phase difference into a query vector includes:

[0078] S411: Perform trigonometric function mapping on the target theoretical phase difference to generate a second vector containing sine and cosine components, which serves as the query vector.

[0079] The target theoretical phase difference is the ideal phase difference calculated based on the physical model. The purpose of trigonometric function mapping is to transform it into a vector with the same feature space as the key vector, which facilitates subsequent similarity calculation. For example, the theoretical phase difference Δ at the target direction of 0° and frequency point f=100 in the above embodiment... theo (100, 0°) ≈ 1.16 rad. After trigonometric function mapping, the second vector Q(100) = [sin(1.16), cos(1.16)] ≈ [0.92, 0.39] is obtained. This vector is the query vector. The query vector and the key vector use the same trigonometric function encoding method, ensuring they are in the same feature space. This makes subsequent similarity calculations more reasonable and accurate, unlike existing technologies that directly use features of different dimensions to calculate distance.

[0080] This embodiment converts the theoretical phase difference into a query vector through trigonometric function mapping, ensuring its consistency with the feature space of the key vector. This step is crucial for realizing the physical prior-guided attention mechanism. The query vector represents the ideal spatial characteristics of the target direction, providing an accurate reference standard for subsequent matching with actual observed signals. Furthermore, the query vector only changes with the target pickup direction and microphone geometry, allowing adjustment of the pickup direction without retraining the model. This achieves method controllability and solves the problem of poor directional adaptability in existing technologies.

[0081] In one embodiment, determining the target theoretical phase difference corresponding to each frequency point based on the user-specified target pickup direction and the geometric parameters of the at least two microphones includes:

[0082] S31: Based on the target pickup direction, sound wave propagation speed, frequency values ​​at each frequency point, and the relative positions between the at least two microphones, calculate the phase difference corresponding to the theoretical propagation delay of the sound source signal reaching each microphone, and use this as the target theoretical phase difference.

[0083] The theoretical propagation delay is the time difference between the arrival of a sound source signal from the target direction at different microphones, and the calculation formula is: The relationship between phase difference and time delay is Δ =2πfτ, combined with the parameters of the short-time Fourier transform, finally yields the theoretical phase difference formula. The relative position of the microphones mainly refers to the microphone spacing d, while the sound wave propagation speed c is a constant. In one specific embodiment, the user specifies the target sound pickup direction θ. target =30°, microphone spacing d=0.02m, sound speed c=340m / s, sampling rate F s =16kHz, Fourier transform (NFFT) points = 512, for a frequency point f = 200, the theoretical time delay τ is calculated to be approximately 2.941 × 10⁻⁶. -5 s, theoretical phase difference Δ theo (200, 30°) ≈ 0.037 rad (approximately 2.12°). The theoretical phase difference is calculated based on the physical laws of sound wave propagation, without relying on the statistical assumptions of existing technologies. Even in a strong reverberant environment, if reflected sound causes a jump in the observed phase difference, the theoretical phase difference can still remain stable, providing a reliable reference for direction determination.

[0084] This embodiment clarifies the physical basis for calculating the theoretical phase difference, namely, it is based on the relationship between propagation delay and phase difference, combined with microphone geometric parameters and target direction. It introduces the core of physical acoustic priors, fundamentally different from existing technologies that rely on the Gaussian distribution assumption. The theoretical phase difference calculation method in this embodiment, based on a rigorous physical acoustic model, possesses extremely high stability and reliability, solving the problem of high misjudgment rates caused by the idealization of physical models in existing technologies. Simultaneously, this calculation method is simple, efficient, and has low computational overhead, making it suitable for the real-time processing requirements of embedded devices.

[0085] In one embodiment, generating a spatial confidence mask representing the confidence of speech presence from the target pickup direction based on the similarity between the query vector and the key vector includes:

[0086] S431: Calculate the dot product of the query vector and the key vector at each frequency point and at each time point to obtain the original similarity score.

[0087] Dot product similarity is the core metric in the geometric cross-attention mechanism described above, used to measure the degree of feature matching between the query and the key. The calculation formula is as follows: D model The dimension is the vector after mapping, and the denominator is the normalization factor used to avoid distortion of similarity values ​​due to excessively high dimensionality. In a specific embodiment, the query vector Q(100)=[0.92,0.39] and key vector K(100,50)=[1,0,0.8] obtained in the above embodiment are mapped to a Dmodel=128 dimension through a fully connected layer to obtain Q′(100) and K′(100,50). The dot product is then calculated to obtain Q′(100)×K′(100,50). T =85, then the original similarity score Score(100,50)≈7.52. This calculation process is a direct manifestation of the geometric cross-attention mechanism "physical manifold matching". By using vector dot product in high-dimensional space to replace the traditional Euclidean distance calculation in low-dimensional space, it is more suitable for handling nonlinear feature spaces, can effectively ignore the slight phase deviation caused by reverberation, and improve the robustness of direction discrimination.

[0088] S432: Normalize the original similarity scores to generate a spatial confidence mask with values ​​between zero and one.

[0089] Normalization is performed using the Sigmoid function, as shown in the formula. This function maps any real number to the interval 0-1, and the mapped value represents the confidence level that the signal belongs to the target direction. In a specific embodiment, for the original similarity score Score(100,50)=7.52 obtained in the above embodiment, after normalization by the Sigmoid function, AttnMap(100,50)≈0.999, indicating that the signal at this frequency point and at this time has a very high probability of belonging to the target direction; if the original similarity score is -2, the normalized value is approximately equal to 0.12, indicating that the signal is likely not to belong to the target direction. Based on the geometric cross-attention mechanism, through the similarity calculation and normalization in this step, the generated spatial confidence mask has a clear physical meaning. The higher the confidence value, the closer the signal is to the physical characteristics of the target direction. Compared with the probabilistic decision method of the prior art, the mask has stronger accuracy and robustness.

[0090] This embodiment replaces the Euclidean distance calculation and Gaussian probability decision of existing technologies with dot product similarity calculation and Sigmoid normalization. It achieves dynamic matching between physical prior features and actual observed features, overcoming the shortcomings of rigid geometric matching in existing technologies. The mask generation method in this embodiment employs a non-linear similarity calculation method, resulting in strong anti-reverberation capability. The normalized mask has clear physical meaning, and the filtering operation is highly targeted, effectively preserving speech in the target direction, suppressing interference and reverberation, and improving the sound quality and clarity of directional sound pickup.

[0091] In one embodiment, filtering the frequency domain signal using the spatial confidence mask to obtain the enhanced spectrum of the target pickup direction includes:

[0092] S51: The spatial confidence mask is spliced ​​with the amplitude spectrum of the frequency domain signal to obtain joint time-frequency features.

[0093] Joint time-frequency features are high-dimensional vectors that simultaneously contain spatial confidence features and frequency domain amplitude features, providing more comprehensive signal information for time-series neural networks. In a specific embodiment, the spatial confidence mask AttnMap(f,t) obtained in the above embodiment is combined with the frequency domain signal amplitude spectrum |X beam (f,t)∣ is concatenated to obtain joint time-frequency features. Where f is the frequency index and t is the frame index. The joint time-frequency features integrate spatial direction information and signal energy information, enabling subsequent temporal neural networks to make discriminations based on both direction confidence and signal energy, thus improving the accuracy of mask generation.

[0094] S52: Input the joint time-frequency features into a pre-trained temporal neural network, and generate a complex ideal ratio mask from the temporal neural network.

[0095] Temporal neural networks employ gated recurrent unit (GRU) networks to capture long-term correlations in speech signals; the complex ideal ratio mask (cIRM) is a complex mask that incorporates both amplitude and phase information, as shown in the formula... Where S(f,t) is the target speech frequency domain signal, and N(f,t) is the noise frequency domain signal. In a specific embodiment, the joint time-frequency feature Feature(f,t) is input into a pre-trained GRU network in chronological order, and the network outputs a complex ideal ratio mask M. cIRM (f,t), where the real and imaginary parts of the mask correspond to the gain coefficients of amplitude and phase, respectively. The GRU network can capture the long-term correlation of speech signals, replacing the first-order Markov chain in the existing technology, effectively solving the "musical noise" problem and improving the continuity and naturalness of speech; the complex ideal ratio mask processes both amplitude and phase information simultaneously, and compared with the real mask, the sound quality improvement is more significant.

[0096] S53: Perform complex domain filtering on the frequency domain signal based on the complex ideal ratio mask to obtain the enhanced spectrum of the target pickup direction.

[0097] Complex domain filtering involves multiplying the complex form of the frequency domain signal by a complex ideal ratio mask, as shown in the formula: , where X beam (f,t) represents the complex frequency domain signal output by the beamformer. In one specific embodiment, X from the above embodiment... beam (100, 50) = 0.5 + 0.3j and M cIRM Multiplying (100,50) = 0.9 + 0.1j, we get the enhanced spectrum Y. enhance (100,50)=(0.5+0.3j)×(0.9+0.1j)=0.42+0.32j, where "j" is the imaginary unit. Complex domain filtering adjusts both the amplitude and phase of the signal simultaneously. Compared to real-number filtering that only adjusts the amplitude, it can better preserve the phase information of speech, reduce speech distortion, and improve the output sound quality.

[0098] This embodiment achieves refined signal processing through feature concatenation, generation of complex masks using a temporal neural network, and complex domain filtering. The temporal neural network addresses the weakness of existing technologies in temporal modeling, while the use of complex masks enhances the filtering effect, significantly differentiating it from the simple gain filtering of existing techniques. The filtering method in this embodiment integrates spatial and energy features, resulting in high mask generation accuracy. The GRU network effectively eliminates "musical noise," improving speech continuity. Complex domain filtering reduces speech distortion, ultimately producing enhanced speech with high quality and clarity, meeting the requirements of high-fidelity directional sound pickup.

[0099] In one embodiment, the pre-trained temporal neural network is obtained by training in the following manner:

[0100] S101: During the training phase, for each training sample, a target sound pickup direction is randomly sampled from a preset angle range, and the corresponding target sound source reference speech is obtained.

[0101] Training samples are generated by convolution of clean target speech, interfering noise, and room impulse response (RIR); the preset angle range is [-90°, 90°], covering common sound source directions; the target sound source reference speech is a clean speech signal in the target direction. For example, the target pickup direction θ is randomly sampled from the preset angle range [-90°, 90°]. target =45°, select the speech signal S(t) from the clean speech library as the target sound source reference speech, and compare it with the room impulse response h 45° The training samples are obtained by convolving (t) and then superimposing the interference noise N(t). ,in This represents a convolution operation. The training method of randomly sampling the target sound pickup direction enables the neural network to learn feature matching rules for different directions, improving network controllability and solving the problem of poor directional adaptability in existing technologies.

[0102] S102: Based on the sampled target pickup direction and the geometric parameters of the at least two microphones, determine the target theoretical phase difference corresponding to each frequency point.

[0103] The theoretical phase difference in this step is completely consistent with the theoretical phase difference calculation method in the above embodiments, calculating the theoretical phase difference in the target direction based on a physical acoustic model. For example, for the sampled target direction θ target =45°, microphone spacing d=0.02m, calculate the theoretical phase difference Δ at each frequency point according to the formula in the above embodiment. theo (f, 45°). The same theoretical phase difference calculation method is used in the training phase as in the inference phase, which ensures the consistency between training and inference and improves the generalization ability of the network.

[0104] S103: Encode the target theoretical phase difference into a query vector, and encode the observed phase difference calculated from the frequency domain signal of the training samples into a key vector.

[0105] The query vector and key vector encoding methods in this step are consistent with those in the above embodiments, ensuring that the vector features in the training phase are consistent with those in the inference phase. For example, the Δ obtained in the above embodiments theo (f, 45°) is encoded as a query vector Q(f), and the training sample X is used as the query vector Q(f). mix(t) Δ obtained by time-frequency transformation and phase difference calculation obs (f,t) is encoded as a key vector K(f,t). Using a unified encoding method ensures the effectiveness of features during the training phase, enabling the network to accurately learn the matching rules between the query and the key.

[0106] S104: Generate a spatial confidence mask based on the similarity between the query vector and the key vector.

[0107] This step is consistent with the mask generation method in the above embodiments, generating a spatial confidence mask through dot product similarity calculation and sigmoid normalization. The similarity matching rules learned during the training phase can be directly transferred to the inference phase, improving the network's inference efficiency and accuracy.

[0108] S105: The spatial confidence mask is concatenated with the amplitude spectrum of the frequency domain signal of the training sample to obtain joint time-frequency features.

[0109] The amplitude spectrum stitching in this step is consistent with the feature stitching method in the above embodiment, fusing spatial confidence features and amplitude spectrum features. The fused features contain rich information, enabling the network to learn more comprehensive discrimination rules and improve the accuracy of mask generation.

[0110] S106: Input the joint time-frequency features into the time-series neural network to obtain the sound pickup result.

[0111] The temporal neural network is a GRU network. Its input is joint time-frequency features, and its output is a complex ideal ratio mask. Multiplying the mask by the frequency domain signal of the training samples and then performing an inverse time-frequency transform yields the time-domain sound pickup result. For example, using F... eature (f,t) is input into the GRU network to obtain the complex ideal ratio mask M. cIRM (f,t), and X mix After multiplying (f,t), perform ISTFT to obtain the time-domain pickup result y. pred (t). GRU networks can capture long-term correlations in speech, and the output pickup results have good continuity and no "musical noise".

[0112] S107: Calculate the loss function based on the difference between the sound pickup result and the target sound source reference speech, and update the parameters of the temporal neural network.

[0113] The loss function used is the joint loss function, and the formula is as follows: L mask To predict the mean square error (MSE) loss between the mask and the ideal mask, L timeThe scale-invariant signal-to-noise ratio (SI-SNR) loss between the pickup result and the reference speech is represented by λ1 and λ2, which are weighting coefficients with values ​​of 0.5 and 0.5, respectively. In a specific embodiment, the prediction mask M is calculated. pred (f,t) and the ideal mask M ideal MSE loss of (f,t) Calculate the pickup result y pred SI-SNR loss between (t) and reference speech S(t) S target For y pred The component in (t) that is related to S(t), n res This represents the residual noise component. The total loss is calculated based on the joint loss function, and the parameters of the GRU network are updated using the Adam optimizer. The joint loss function simultaneously optimizes both frequency domain masking accuracy and temporal auditory perception, resulting in a more efficient network with higher sound quality and clarity compared to a single loss function.

[0114] This embodiment discloses a training method for a temporal neural network, employing a dynamic angle injection training strategy. The target pickup direction is randomly sampled for training, giving the network controllability. This training method differs from existing static direction training methods, allowing adjustment of the pickup direction without retraining the network. Furthermore, a joint loss function optimizes network performance, addressing the problem of limited training strategies in existing technologies. This training method enables the neural network to have directional controllability; changing the target direction only requires adjusting the query vector, eliminating the need for retraining and improving the method's flexibility. The joint loss function ensures the frequency domain accuracy and temporal auditory quality of the output speech. The trained network performs excellently in environments with strong reverberation and low signal-to-noise ratio, achieving high-fidelity directional pickup.

[0115] Reference Figure 2 This invention also provides a directional sound pickup device, comprising:

[0116] Acquisition unit 10 is used to acquire time-domain speech signals collected by at least two microphones;

[0117] The first calculation unit 20 is used to perform time-frequency transformation on the time-domain speech signal to obtain the corresponding frequency-domain signal, and to calculate the observation phase difference of each frequency point at each time based on the frequency-domain signal; and,

[0118] The second calculation unit 30 is used to determine the target theoretical phase difference corresponding to each frequency point based on the target pickup direction specified by the user and the geometric parameters of the at least two microphones.

[0119] The encoding similarity calculation unit 40 is used to encode the target theoretical phase difference into a query vector, encode the observed phase difference into a key vector, and generate a spatial confidence mask representing the confidence of speech presence from the target pickup direction based on the similarity between the query vector and the key vector.

[0120] The filtering unit 50 is used to filter the frequency domain signal using the spatial confidence mask to obtain the enhanced spectrum of the target pickup direction;

[0121] The output unit 60 is used to perform inverse time-frequency transformation on the enhanced spectrum and output the pickup result of the target pickup direction.

[0122] As described above, the directional pickup device can implement the directional pickup method in any of the above embodiments.

[0123] Reference Figure 3 The present invention also provides a computer device, the internal structure of which can be as follows: Figure 3 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor is designed to provide computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores operating devices, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database stores relevant data. The network interface is used to communicate with external terminals via a network connection. Furthermore, the computer device may also include an input device and a display screen. When the computer program is executed by the processor, it implements the directional sound pickup method described in any of the above embodiments. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.

[0124] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the directional sound pickup method described in any of the above embodiments. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0125] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in this application and in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0126] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0127] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A directional sound pickup method, characterized in that, Includes the following steps: Acquire time-domain speech signals from at least two microphones; The time-domain speech signal is transformed by time-frequency to obtain the corresponding frequency-domain signal, and the observed phase difference at each frequency point at each time is calculated based on the frequency-domain signal. The observed phase difference refers to the phase difference value of the same signal collected by different microphones at the same frequency point and the same time. as well as, Based on the target pickup direction specified by the user and the geometric parameters of the at least two microphones, determine the target theoretical phase difference corresponding to each frequency point, including: based on the target pickup direction, the speed of sound propagation, the frequency value of each frequency point, and the relative position between the at least two microphones, calculate the phase difference corresponding to the theoretical propagation delay of the sound source signal to each microphone, and use it as the target theoretical phase difference; The target theoretical phase difference is encoded as a query vector, the observed phase difference is encoded as a key vector, and a spatial confidence mask representing the confidence of speech presence from the target pickup direction is generated based on the similarity between the query vector and the key vector. This includes: calculating the dot product of the query vector and the key vector at each frequency point and each time point to obtain the original similarity score; and performing Sigmoid normalization on the original similarity score to generate a spatial confidence mask with values ​​between zero and one. Filtering the frequency domain signal using the spatial confidence mask to obtain the enhanced spectrum of the target pickup direction includes: concatenating the spatial confidence mask with the amplitude spectrum of the frequency domain signal to obtain a joint time-frequency feature fusing spatial direction information and signal energy information; inputting the joint time-frequency feature into a pre-trained temporal neural network, which generates a complex ideal ratio mask. This complex ideal ratio mask is a complex-form mask that simultaneously contains amplitude and phase information, as shown in the formula: Where S(f,t) is the target speech frequency domain signal and N(f,t) is the noise frequency domain signal; Based on the complex ideal ratio mask, the frequency domain signal is filtered in the complex domain to obtain the enhanced spectrum of the target pickup direction; The enhanced spectrum is subjected to inverse time-frequency transformation, and the pickup result of the target pickup direction is output.

2. The directional sound pickup method according to claim 1, characterized in that, Encoding the observed phase difference into a key vector includes: The observed phase difference is mapped using trigonometric functions to generate a first vector containing sine and cosine components; The first vector is concatenated with the amplitude spectrum of the frequency domain signal from at least one microphone to obtain the key vector.

3. The directional sound pickup method according to claim 1, characterized in that, Encoding the target theoretical phase difference into a query vector includes: The target theoretical phase difference is mapped using trigonometric functions to generate a second vector containing sine and cosine components, which serves as the query vector.

4. The directional sound pickup method according to claim 1, characterized in that, The pre-trained temporal neural network is obtained through training in the following manner: During the training phase, for each training sample, a target sound pickup direction is randomly sampled from a preset angle range, and the corresponding target sound source reference speech is obtained; Based on the sampled target pickup direction and the geometric parameters of the at least two microphones, determine the target theoretical phase difference corresponding to each frequency point; The target theoretical phase difference is encoded into a query vector, and the observed phase difference calculated from the frequency domain signal of the training samples is encoded into a key vector; Based on the similarity between the query vector and the key vector, a spatial confidence mask is generated; The spatial confidence mask is concatenated with the amplitude spectrum of the frequency domain signal of the training sample to obtain joint time-frequency features; The joint time-frequency features are input into the temporal neural network to obtain the sound pickup result; The loss function is calculated based on the difference between the sound pickup result and the reference speech of the target sound source, and the parameters of the temporal neural network are updated.

5. A directional sound pickup device for implementing the directional sound pickup method according to any one of claims 1 to 4, characterized in that, include: An acquisition unit is used to acquire time-domain speech signals collected by at least two microphones; The first calculation unit is used to perform time-frequency transformation on the time-domain speech signal to obtain the corresponding frequency-domain signal, and calculate the observation phase difference of each frequency point at each time based on the frequency-domain signal. as well as, The second calculation unit is used to determine the target theoretical phase difference corresponding to each frequency point based on the target pickup direction specified by the user and the geometric parameters of the at least two microphones. The encoding similarity calculation unit is used to encode the target theoretical phase difference into a query vector, encode the observed phase difference into a key vector, and generate a spatial confidence mask representing the confidence of speech presence from the target pickup direction based on the similarity between the query vector and the key vector. A filtering unit is used to filter the frequency domain signal using the spatial confidence mask to obtain the enhanced spectrum of the target pickup direction; The output unit is used to perform inverse time-frequency transformation on the enhanced spectrum and output the pickup result of the target pickup direction.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the directional sound pickup method as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the directional sound pickup method as described in any one of claims 1 to 4.

Images