Directional speech enhancement method, device, storage medium and program product

Abstract

The application provides a directional speech enhancement method and device, a storage medium and a program product, relates to the technical field of speech processing, and the method comprises the following steps: extracting a corresponding target directional speech vector, a positive edge directional vector and a negative edge directional vector from original audio to be enhanced according to a target direction, a positive edge angle and a negative edge angle; performing a beamforming operation on the target directional speech vector, the positive edge directional vector and the negative edge directional vector to obtain corresponding beam enhanced speech; and inputting each beam enhanced speech and the original audio to a speech enhancement model to generate target enhanced speech corresponding to the target direction. Thus, by using a three-directional vector selection strategy, spatial information is effectively utilized, the system can dynamically adapt to enhance target speech while suppressing interfering speech from other directions, and the practicability and accuracy of speech enhancement are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing, and in particular to a directional speech enhancement method and device, a storage medium and a program product. BACKGROUND

[0002] In recent years, speech enhancement (SE) in extremely low signal-to-noise ratio (SNR) environments has received extensive attention because of its key applications in telecommunications, hearing aids, and automatic speech recognition (ASR) systems. In these environments, the main challenges include background noise, speech interference, and reverberation, which can severely degrade the intelligibility and quality of speech. Among them, the most challenging is human speech interference, because speech recognition systems have difficulty distinguishing target speech from interfering speech, resulting in a significant decrease in performance.

[0003] In order to effectively distinguish target speech from other interfering speech, some scholars in the industry have proposed some directional speech enhancement systems, which can enhance the speech of a target speaker according to a specified target direction and suppress the speech signals of interfering speakers from other directions. Specifically, non-neural network speech enhancement is achieved by using delay and sum beamforming, generalized sidelobe canceller (GSC) adaptive beamforming.

[0004] In addition, some technology companies have also proposed neural network-based speech enhancement systems, such as Google's BASNet and GSENet. However, the target enhancement direction of these neural network-based systems is usually set before training and can only be applied to cases where the target speaker's direction does not change. If the speaker's direction changes, the model needs to be retrained, which consumes a lot of resources. Specifically, the direction of the target speaker and the interfering speaker usually needs to be determined first, and then the model is trained to enhance the target direction and suppress the interfering direction. The directional speech enhancement model trained in this way can perform speech enhancement on a fixed direction. Therefore, when the target speaker changes, the speech enhancement performance of such a directional speech enhancement model is poor.

[0005] In view of the above problems, the industry has not yet proposed a better solution. SUMMARY

[0006] The application provides a directional speech enhancement method, device, storage medium and program product, to at least solve the problem that the performance of a speech enhancement system based on a neural network in the prior art is poor when the direction of a target speaker changes.

[0007] In a first aspect, the embodiments of the application provide a directional speech enhancement method, comprising: extracting a target directional speech vector, a forward edge directional vector and a reverse edge directional vector corresponding to a target direction, a forward edge angle and a reverse edge angle from original audio to be enhanced; performing a beamforming operation on the target directional speech vector, the forward edge directional vector and the reverse edge directional vector to obtain a corresponding beam enhanced speech; and inputting the beam enhanced speech and the original audio to a speech enhancement model to generate target enhanced speech corresponding to the target direction.

[0008] In a second aspect, the embodiments of the application provide an electronic device, comprising: at least one processor, and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the directional speech enhancement method of any of the embodiments of the application.

[0009] In a third aspect, the embodiments of the application provide a storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the steps of the directional speech enhancement method of any of the embodiments of the application.

[0010] In a fourth aspect, the embodiments of the application provide a computer program product comprising computer programs / instructions, which, when executed by a processor, implement the steps of the directional speech enhancement method of any of the embodiments of the application.

[0011] The embodiments of the application have the following beneficial effects:

[0012] By using a three-directional vector selection strategy, spatial information is effectively utilized, speech information in different directions can be flexibly extracted and distinguished, and even if the direction of the target speaker changes, the system can dynamically adapt, enhance the target speech while suppressing the interfering speech from other directions, and improve the practicability and accuracy of speech enhancement. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0014] Figure 1 A flow chart showing an example of a directional speech enhancement method according to an embodiment of the present application is shown;

[0015] Figure 2 A structural connection diagram showing an example of a speech enhancement model according to an embodiment of the present application is shown;

[0016] Figure 3 A schematic diagram showing a CDUNet architecture according to an embodiment of the present application is shown;

[0017] Figure 4 An effect diagram showing a simulation setting of a first fixed target data set is shown;

[0018] Figure 5 A structural diagram of an embodiment of an electronic device of the present application is shown. DETAILED DESCRIPTION

[0019] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0020] It is understood that humans can focus attention on a particular direction using binaural spatial information. Similarly, modern hearing aids, speech recognition systems, and other devices are equipped with multiple microphones. Therefore, multi-channel speech enhancement is becoming increasingly important, and many researchers have begun to explore the use of spatial features for speech separation or speaker extraction. Traditional multi-channel speech enhancement techniques, such as delay-and-sum and beamformers such as the Generalized Sidelobe Canceller (GSC), have limitations in performance. Recently, many neural network methods have outperformed traditional methods in terms of speech enhancement quality and intelligibility, such as EMGSE, VSEGAN, METRICGAN-U, HGCN, TF-GridNet, and FullSubNet+. However, these methods also have some limitations: first, they usually predefine the target speech region without inputting spatial information at the time of use, such as GSENet (guided speech enhancement network) and BASNet (binaural angular separation network). Second, they mainly focus on scenarios using three or more microphones, such as the multiple extraction method, JNF (Joint Normalization Function), and JNF-SSF (Joint Normalization Function with Single-Sided Filtering). Using too many microphones is not practical in device-side speech enhancement systems, as resource consumption increases rapidly with the number of microphones, and these systems are limited by memory and computing resources. In addition, the large-scale parameters of neural networks limit their feasibility in practical applications. Finally, previous work has mainly focused on improving speech quality indicators, often ignoring their impact on downstream tasks.

[0021] In the current related art, more than three microphones are usually used for directional enhancement of speech, which is costly, and in addition, neural network-based systems usually have a large number of parameters, making it difficult to use on end-side devices. In addition, when the target speaker changes, the phase of the incoming spectral graph needs to be manually changed to shift the target direction to the target direction during training, which consumes a lot of resources.

[0022] In view of this, in the embodiments of the present application, not only the incoming target angle, but also the "width" as a parameter to prompt the model of the interference direction are proposed, providing users with a more adaptable solution to various environments. The enhancement width can be set during training according to the actual environmental conditions, and at the same time, this also requires the model to have stronger adaptability and the ability to switch directions at any time.

[0023] Figure 1 A flowchart illustrating an example of a directional speech enhancement method according to an embodiment of this application is shown.

[0024] like Figure 1 As shown, in step S110, the target-guided speech vector, the forward edge-guided vector, and the reverse edge-guided vector are extracted from the original audio to be enhanced according to the target direction, the forward edge angle, and the reverse edge angle.

[0025] Here, the target direction can be the direction of the target speaker perceived by the user input or sensors, for example, the target direction can be accurately determined by direction estimation algorithms (such as sound source localization algorithms). The forward edge angle and the reverse edge angle define two boundary directions within a certain angular range related to the target direction in space. It should be understood that the forward edge angle and the reverse edge angle can be preset or pre-calibrated parameters.

[0026] Furthermore, the target-oriented speech vector is extracted based on the speech signal in the target direction, and this vector contains the speech information of the potential target speaker. The forward edge-oriented vector is extracted based on the audio signal at the forward edge angle relative to the target direction, and this vector may contain signals from this deviating direction (such as interference or noise). The reverse edge-oriented vector is extracted based on the audio signal at the reverse edge angle relative to the target direction, used to capture signals from the other side. By extracting oriented speech vectors in the target direction and at the edge angles on both sides, it is possible to effectively cover the speech information and interference information of the spatial region where the target speaker is located.

[0027] In step S120, beamforming is performed on the target-guided speech vector, the forward edge-guided vector, and the reverse edge-guided vector to obtain the corresponding beam-enhanced speech.

[0028] This can be achieved through various non-restricted beamforming algorithms. For example, by employing techniques such as delay-and-sum beamforming or adaptive beamforming (e.g., generalized sidelobe canceller, GSC), the speech signal from the target direction is enhanced by adjusting the phase and amplitude of the input signal, while suppressing interference signals from other directions, thus generating a corresponding beam-enhanced speech signal. Furthermore, for different steering vectors, corresponding beamforming parameters, such as gain and delay, can be configured separately to optimize the speech signal from the target direction. Therefore, by combining multiple steering vectors, more features of the target speech can be captured, resulting in a more intelligible and clearer enhanced speech signal.

[0029] In step S130, the beam enhanced speech and the original audio are input to the speech enhancement model to generate target enhanced speech corresponding to the target direction.

[0030] Here, the speech enhancement model can adopt a deep neural network-based model, such as a Convolutional Neural Network (CNN), a Long Short-Term Memory (LSTM), or a Transformer structure. In this way, through the learning ability of the neural network, the input audio features are analyzed, the speech signal of the target direction is further amplified, and the interference signal of the non-target direction is further reduced, achieving an optimized enhancement effect of the target speech.

[0031] In some examples of the embodiments of the present application, the angle width by which the forward edge angle and the reverse edge angle deviate from the target direction is equal. In this way, only by setting the angle width in the speech enhancement system, the corresponding forward edge angle and reverse edge angle can be automatically determined.

[0032] In some business scenarios, the angle width can be flexibly adjusted and can be set or updated according to actual interference directions or business needs.

[0033] Through experiments, it is known that the angle width is not less than 3°. When the width is less than 3°, the input width is close to the target direction, resulting in a weakened ability to obtain effective information, and at this time the performance is reduced. Exemplarily, the value range of the angle width is 3°~max(boundary angle), where the boundary angle is the estimated maximum angle between the target direction and the interference direction (which can be determined according to actual conditions), and more details will be evidenced and expanded in the experimental part below.

[0034] Figure 2 A structure connection diagram of an example of a speech enhancement model according to an embodiment of the present application is shown.

[0035] As shown in Figure 2 The speech enhancement model 200 includes a cascaded encoder 210, a noise reducer 220, and a decoder 230.

[0036] Here, the input audio signal is converted from the time domain to a representation space (or feature space) more suitable for neural network processing by the encoder 210. The original beam enhanced speech and the original audio are processed by the encoder to generate a feature sequence. Exemplarily, the encoder can use a convolutional neural network to extract local time domain features of the audio signal, for example, by processing the audio through a sliding window to capture frequency information at different time points, and map the time domain audio signal to a higher-dimensional feature space.

[0037] The noise reducer 220 includes multiple DPRNN (Dual-Path Recurrent Neural Network) blocks for sequence modeling, each DPRNN block including a frequency sequence layer and a long short-term memory (LSTM) layer. Specifically, the DPRNN block divides the input sequence into shorter blocks, models from two directions of recombination using two RNN layers, for local and global, i.e., shorter frequency dimension and longer time dimension, respectively.

[0038] Here, by performing noise reduction processing on the encoded audio feature sequence through the noise reducer 220, the DPRNN is able to process information in both time and frequency domains simultaneously, thus better removing noise. In each DPRNN block, the frequency dimension of the input features is modeled through the frequency sequence layer, enabling the network to capture frequency relationships and correlations in the features, thus distinguishing between speech and noise signals. The time dimension of the input features is processed through the LSTM layer, thus capturing the speech signal’s temporal variation patterns. In this way, the model is able to process long-time span audio data, effectively isolating and suppressing interference and noise in the time domain. Through the “dual-path” structure of the DPRNN, time and frequency domain features are alternately processed, enabling the network to capture dynamic changes in the audio signal in these two dimensions, thus more effectively performing speech enhancement. Through the processing of multiple DPRNN blocks, the system is able to work stably in dynamic and complex acoustic environments, not only suitable for single background noise, but also adaptable to mixed noise and multiple speaker scenarios, especially when used in hearing aid and speech recognition applications, having strong robustness and adaptability.

[0039] The feature sequence processed through the noise reducer is converted back to a time-domain audio signal through the decoder 230, thus generating the final enhanced speech or related speech feature representation. Specifically, the decoder can adopt a symmetric architecture to the encoder (e.g., using deconvolutional layers or up-sampling layers), thus realizing time-domain reconstruction of the features. Exemplarily, the decoder can include a deconvolution (Deconv) layer or a transpose convolutional layer to restore the features processed through the noise reduction to audio signals. By gradually enlarging the feature space, the decoder is able to recover the time-domain information of the original signal, and on the basis of the DPRNN processing result, the decoder can ensure that the generated enhanced speech has a high signal-to-noise ratio, while maintaining the naturalness and coherence of the speech signal.

[0040] Through the embodiments of the present application, based on the cascaded structure of the encoder, the noise reducer of multiple DPRNN blocks and the decoder, an effective processing flow from the time domain audio signal to the feature space and back to the time domain is realized. The encoder extracts features, the DPRNN in the noise reducer processes the time-frequency information of the audio signal through the double path, and the final decoder reconstructs the enhanced speech signal. Thus, in a complex acoustic environment, efficient and accurate speech enhancement effect can be achieved, making the target speech clearer and more natural, while effectively suppressing noise and improving the performance of speech recognition and communication systems.

[0041] In some examples of the embodiments of the present application, the encoder includes multiple encoder blocks, and the decoder includes multiple decoder blocks, at this time the speech enhancement model includes a CBAM module (Convolutional Block Attention Module), and the first CBAM module is used to cross-connect the output end of the first encoder block and the input end of the first decoder block. Here, the term "first" is not specific, and in some cases can be generic to any selected object.

[0042] Exemplarily, each encoder block can respectively adopt a two-dimensional convolution block, and each decoder block can respectively adopt a two-dimensional deconvolution block. In each encoder block, the convolution kernel is used to capture the local features in the audio data, so as to extract higher-level features layer by layer. Pooling layers can be introduced between each convolution block for down-sampling, which compresses the input time-frequency features to a lower dimension, not only reducing the calculation amount, but also improving the attention to key features. In each decoder block, the compressed features of the encoder are up-sampled and restored to ensure that key time-frequency information is not lost during the speech enhancement process. In addition, based on the skip connection, the output of the encoder block is connected to the input of the corresponding decoder block through the CBAM module, so as to retain the low-level time-frequency feature information and effectively improve the reconstruction ability of the decoder.

[0043] Here, the CBAM module includes a channel attention module and a spatial attention module, which recalibrates the feature map output by the encoder, adaptively allocates the weight of the feature map, and ensures that the model pays more attention to the time-frequency area that is beneficial to target enhancement. Based on the cross-connection mechanism of the CBAM module, the input of the decoder is further enhanced, so that the model can use more useful time-frequency information during the reconstruction process, greatly improving the accuracy and clarity of speech enhancement.

[0044] In some examples of the embodiments of the present application, the original audio is multi-channel audio, for example, containing audio data collected by two or more microphones (Mic.), each microphone can be arranged at different positions to collect audio signals in different directions, capturing sound information from different spatial angles. The collected original audio signals are subjected to short-time Fourier transform (STFT) to convert the time-domain audio signals into time-frequency domain features, and the frequency and phase information of different channel audio signals are extracted. The speech enhancement model is used to screen the nearest channel audio for the target speaker from the multi-channel audio, and generate the target enhanced speech according to the nearest channel audio and the mask signal output by the decoder, wherein the nearest channel audio is the channel audio with the smallest angle between the pickup direction and the target direction among the channel audios.

[0045] In some embodiments, the direction of the target speaker is determined by a sensor or a direction estimation algorithm (such as a sound source positioning technology based on a microphone array), and then the angle between the pickup direction and the target direction is calculated for the audio signal of each channel. According to the calculated deviation angle, the channel audio with the smallest deviation angle is selected as the "nearest channel audio", i.e. the channel audio closest to the target direction, thus containing the clearest target speech signal.

[0046] In addition, the decoder extracts the corresponding speech mask signal from the time-frequency features transmitted by the encoder and the noise reduction module, which is used to filter the input audio signal, suppress the interference noise and highlight the target speech signal. Here, the system applies the mask signal output by the decoder to the selected nearest channel audio to generate a mask time-frequency feature map, in which the interference components are greatly weakened and the time-frequency components of the target speech signal are significantly enhanced. Finally, the system converts the processed feature map back to the time-domain signal through inverse STFT to generate the target enhanced speech signal.

[0047] By the embodiments of the present application, the audio closest to the target direction is screened from the multi-channel audio, ensuring that the captured channel speech signal is the clearest. Through the mask signal generated by the decoder, the target speech can be efficiently enhanced, the interference noise and background sound can be suppressed, and the signal-to-noise ratio of the speech can be significantly improved. Then, by fusing the mask signal output by the decoder and the clearest channel speech signal for speech enhancement, the system can dynamically adapt to the changing sound source position and environmental noise, and is suitable for various complex multi-speaker and noise scenes, having strong adaptability and flexibility.

[0048] It should be noted that in the short-time Fourier transform domain, the target and the estimated output are represented as s and Preliminary studies show that using the scale-invariant signal-to-noise ratio (SI-SNR) loss function significantly enhances the stability of the learning process. However, when the network is trained using only the SI-SNR loss, it is observed that the network has a tendency to over-suppress low-frequency components. To address this issue, an innovative combined loss function is proposed that combines the SI-SNR and multi-resolution STFT (MR-STFT) losses.

[0049] More specifically, the loss function of the speech enhancement model is:

[0050] , Equation (1)

[0051] In the equation, denotes the loss function of the speech enhancement model, is a set of STFT points, denotes the MR-STFT loss of the th STFT point, denotes the SI-SNR loss of the th STFT point, and is a loss term weighting factor; is the target speech signal, representing the real speech signal; is the predicted enhanced speech signal, representing the model output speech signal.

[0052] By combining the MR-STFT loss and the SI-SNR loss in the embodiments of the present application, the loss function jointly optimizes speech enhancement in both the frequency domain and the time domain, ensuring that the model can simultaneously improve the spectral characteristics of the speech (through MR-STFT) and the time-domain signal-to-noise ratio (through SI-SNR), making the speech enhancement not only more delicate in the frequency spectrum but also significantly reducing noise and improving the overall signal-to-noise ratio of the enhanced speech. The model can maintain high enhancement effect in different noise scenarios, different frequency characteristics, and various complex environments.

[0053] It should be noted that in a multi-speaker scenario, using spatial features is crucial for enhancing the target speech. However, developing a compact multi-channel speech enhancement system is still challenging under the condition of limited microphone arrays, especially under extremely low signal-to-noise ratios.

[0054] In some examples of the embodiments of the present application, a neural-guided speech enhancement method for a dual-microphone array in a high-noise scene is provided, specifically a three-guide spatial selection method, which is a flexible framework using three guide vectors to guide enhancement and determine the enhancement range. Specifically, the speech enhancement model adopts a convolutional U-Net network architecture. By introducing a causal-directed U-Net (CDUNet) model, the model takes the original multi-channel speech and the desired enhancement width as input. This allows the guide vector to be dynamically adjusted according to the target direction, and the enhancement area is fine-tuned according to the angular separation between the target and interference signals. The model provided in the embodiments of the present application only uses a dual-microphone array and performs well in terms of speech quality and downstream task performance. It operates in real time with the fewest parameters, making it very suitable for low-latency, on-device streaming applications.

[0055] Specifically, first, speech and interference speech containing 0-180° target directions are generated as a training set to train the model to have the ability to enhance in various directions. During the training process, the correct direction of the model is input into the model, and the desired enhancement angle width is also input at the same time, so that the model generates three guide vectors according to the target direction and the direction of the target direction plus-minus width to perform beamforming, and then the results of these beamforming are compared with the original spectrum Figure 1 The input is designed into the UNet model. This helps the model to learn to enhance the speech in the target direction, while distinguishing the target speech and interference speech according to the input width, enhancing the model's grasp of real scenes. At the same time, the downstream speech recognition task is tested, and good test results are also obtained. Finally, a dual-microphone directional enhancement model is obtained, which can adjust the input width according to the real situation and does not need to adjust the input data.

[0056] Through the embodiments of the present application, a large amount of existing data can be fully utilized, and the robustness of the model is greatly improved. At the same time, the present scheme can generate three guide vectors according to the input accurate angle and the width to be enhanced, and can accurately divide the target range to be enhanced, and the quality score of the enhanced speech is high. In addition, the present scheme can flexibly adjust the input width to adapt to various scenes, has a wide range of applications and high flexibility. In addition, the model can be compressed and used on an end-side device; directional enhancement of the model can be realized using only a dual microphone, and the resource requirement is low. In addition, the downstream task of speech recognition is also tested, and the performance of the back end is good.

[0057] Specifically, a causal directionally oriented U-shaped network (CDUNet) model is proposed, which combines U-Net with beamforming. In the embodiments of the present application, the spatial position of the target speaker is used as a clue, aiming to create a nonlinear filter that can be flexibly adjusted to the selected direction. In addition, by introducing a variable enhancement width of different interference angles, the model can extract detailed information of the interference signal and dynamically adjust the enhancement range.

[0058] I. Three-directional spatial selection method

[0059] A. Problem definition

[0060] The present study is aimed at the so-called cocktail party problem: extracting the speech signal of the target speaker from the interference speech. It is assumed that the interference speaker comes from a different direction from the target speaker. Therefore, the problem can be described as follows:

[0061] , Equation (2)

[0062] where, denotes the input signal captured by the th microphone, denotes the target speech signal, denotes the interference speech signal, and denote the azimuth angles of the target and interference sound sources relative to the microphone, respectively. The goal of speech enhancement is to train a deep neural network to map to . The enhanced signal is estimated by:

[0063] , Equation (3)

[0064] where, denotes the network parameters, denotes the predicted output, denotes the two-channel speech signal captured by the microphone.

[0065] B. Three-directional spatial selection

[0066] In the scheme provided in the embodiments of the present application, the target angle and two edge angles are used, and the two edge angles are calculated by subtracting and adding the input width from the target angle, so as to generate three steering vectors for the beamformer. The network then inputs the frequency domain representation of the two original microphone signals, and the beamformer outputs under the target angle and the two edge angles. This enables the model to accurately locate the direction of the target speaker. In addition, by combining the input width, the model can estimate the angular separation between the target and the interference source, thereby achieving more accurate directional enhancement.

[0067] C. Causal directional U-Net model

[0068] Figure 3 A schematic diagram of the CDUNet architecture according to the embodiments of the present application is shown.

[0069] As shown in Figure 3 , the model adopts a convolutional U-shaped network architecture. The beamformer output combines the target direction and width input, capturing spatial region information that is crucial for enhancement. represents the range of the target region to be enhanced, represents the direction of the target speaker. The “nearest microphone selection” operation selects the speech signal from the microphone closest to the target speaker.

[0070] Specifically, in the model, a powerful encoder-decoder architecture with skip connections is used to build a deep neural network. The encoder consists of 3 two-dimensional convolution (Conv2D) blocks to encode the input features into latent representations. Correspondingly, the decoder consists of 3 two-dimensional transposed convolution (ConTrans2D) blocks to decode the latent space features back to the feature space.

[0071] Between the encoder and the decoder, a sequence modeling module is integrated, including a frequency sequence layer and a long short-term memory (LSTM) layer, following the dual-path recurrent neural network (DPRNN) framework.

[0072] In addition, a convolutional block attention module (CBAM) is also introduced, which combines channel and spatial attention mechanisms. In the model, CBAM is applied in the decoder and the skip connection to recalibrate the time-frequency (TF) feature maps, thereby improving the accuracy of target reconstruction.

[0073] Let be the i-th output of an encoder block, where . The channel attention gate for a block is computed as follows:

[0074] , Equation (4)

[0075] where, denotes the Sigmoid activation function, and denote the weights of the shared linear layer. The average pooling and max pooling are applied to to compute . The spatial attention gate is computed as follows:

[0076] , Equation (5)

[0077] where, is a convolutional layer (Conv2D). The spatial attention gate is then applied to to compute .

[0078] The output speech signal is estimated by applying the mask generated by the decoder to the closer original channels. The closer channels are selected according to their proximity to the input target angle, and the closer channels are selected for processing. For example, if the input angle is less than 90°, the first channel is selected as the closer channel, otherwise the second channel is used.

[0079] II. Experimental Setup

[0080] A. Dataset

[0081] All clean speech and interfering speech are randomly sampled from LibriSpeech and an in-house corpus. The training dataset is simulated by the following ways:

[0082] • Geometry: Rooms with different geometries are generated according to the JNF benchmark, as shown in Figure 2 . In each simulated room, the microphone spacing is set to 30 mm. The audio sources are randomly selected, one designated as the target source and the other as the interfering source.

[0083] • Mixing: The original audio captured by the two microphones is synthesized using the simulated room impulse response (RIR). For training, the ground-truth signal comes from the early reverberation component of the clean signal, with a reverberation delay of 150 ms.

[0084] Based on this approach, two datasets were synthesized for the baseline model and the proposed CDUNet:

[0085] • Fixed target dataset: The fixed target dataset was used to compare the CDUNet model with the beamformer and other neural network baselines that do not accept angle as an input feature. This dataset contains 250,000 training samples with the target direction fixed between 85° and 95°. The signal-to-noise ratio (SNR) of the target speech relative to the interfering speaker mixture varies from -5 dB to 10 dB across samples.

[0086] • Variable target dataset: To train neural networks for directional speech enhancement, a variable target dataset was developed. This dataset introduces variation in the target speaker position, which is randomly generated, while the interfering direction is always 15° away from the target direction. The SNR and the number of speeches in this dataset are consistent with the fixed target dataset.

[0087] Figure 4 An effect diagram of the simulation setup for the first fixed target dataset is shown. As shown in Figure 4 , the target direction ranges from 85° to 95°, represented by red stars in the figure, while the interfering direction is 15° away from the target direction, represented by green stars. The room information is uniformly sampled from the range provided in the table.

[0088] B. Model setup

[0089] The window size of 512 and the step size of 256 were used for the input Short-Time Fourier Transform (STFT) and the output inverse STFT. The STFT representation was treated as independent channels and decomposed into corresponding phase and magnitude components, resulting in a 10-channel frequency-domain input for the CDUNet model.

[0090] C. Evaluation

[0091] Evaluation process. Evaluation datasets for fixed target and variable target were constructed:

[0092] • For fixed target evaluation, a corresponding test set was created with the target angle , the interfering angle ranging from 0° to 180°, the test set included 500 speech segments with signal-to-noise ratios (SNRs) of 0 dB and 5 dB.

[0093] • For variable target evaluation, 500 speech segments were generated with SNR of 0 dB, the target direction ranging from 0° to 90° and evaluated their PESQ scores. Such angular setup was adopted because 0°-90° is symmetric to 90°-180° for the two-microphone model.

[0094] Baseline models. Two classes of baseline models were selected:

[0095] • U-Net Free. DAS (Delay and Sum) is a traditional delay-and-sum beamformer model, GSC is a generalized sidelobe canceller model, and JNF (Joint Normalization Function) represents a joint spatial and time-frequency nonlinear filter model that uses a circular array of three microphones, which produced superior results that Table I failed to capture.

[0096] Table I. PESQ scores in fixed regions (trained filters with target angle around 90°). Noisy Sp. denotes noisy speech.

[0097]

[0098] • U-Net Based. The U-Net uses the same U-Net architecture as Figure 3 IPD Unet model introduces the interchannel phase difference (IPD), a well-recognized spatial feature, into the U-Net architecture as input, while the BF Unet supplements the output of the beamformer on top of the U-Net model. Unlike the CDUNet model proposed in this application, the latter has a width input.

[0099] Evaluation metrics. The perceptual evaluation of speech quality (PESQ) was used to evaluate the quality of the speech enhanced by the models. In addition, the enhanced speech was input into a pre-trained ASR model to calculate the word error rate (WER).

[0100] III. Results and analysis

[0101] A. Fixed region and directional training

[0102] Here, the enhancement effect is evaluated when the target speaker position remains constant while the direction of the interference speech changes. All models were trained on a fixed target dataset where the target speaker position is approximately 90°, consistent with the testing conditions. The first row of Table I for each SNR setting shows the PESQ score under different interference angles. All U-Net-based models outperformed the U-Net Free models, demonstrating the superior performance of the U-Net architecture.

[0103] Model results based on U-Net show that including IPD significantly degrades performance, while the beamformer output has a relatively small impact on performance. When trained on a fixed target dataset, the proposed CDUNet model, with only 74.4k parameters, achieves a significant performance improvement compared to the baseline model with the U-Net structure, highlighting its superior performance. Effectiveness in improving performance. Furthermore, although speaker locations are variable, CDUNet trained on a variable target dataset still achieves performance comparable to augmentation results for fixed regions. This achievement is particularly significant because CDUNet has only 74.4k parameters, while JNF has a whopping 1M parameters. Therefore, the proposed CDUNet can not only learn a single spatial filter, but also learn 180 spatial filters in each direction with fewer training samples (1400 per direction instead of 250,000). This efficiency in augmentation learning with fewer samples highlights the model's robustness and adaptability when dealing with complex spatial data.

[0104] B. Different Impact on CDUNet

[0105] To determine the optimal input width for CDUNet, we evaluated the PESQ score under different input angles using a fixed-target training dataset, examining the variation... The impact on speech quality enhancement. The results in Table II show that when... The model performs optimally when the target direction and edge angles are considered. This model performs beamforming based on the target direction and edge angles. The edge angles are calculated by adjusting the input width up and down by a specified amount. The introduction of the width input enables the model to distinguish the spatial distribution of interfering speech, which is achieved by comparing the edge speech signal with the target signal.

[0106]

[0107] Table II. Fixed-target PESQ scores for different input widths.

[0108] It is worth noting that in the dataset, the angular separation between the target and interference directions always exceeds 15°. Therefore, when the width is set below 15°, the model can effectively utilize the input width as a distinguishing boundary to separate the target speech. In contrast, when the width is reduced to 3°, the input width is close to the target direction, resulting in a weakened ability to obtain novel and effective information, which in turn slightly impairs the model performance. When the width exceeds 15°, the input width fails to separate the target direction from the interference direction, leading to a performance decline. In practical applications, the input width can be flexibly adjusted to match the actual interference direction.

[0109] C. Directional speech enhancement

[0110] Table III. PESQ scores under target speaker position variation.

[0111]

[0112] As shown in Table III, the CDUNet model consistently demonstrates its ability to dynamically adapt to the target speaker using the input angle. Regardless of the target angle variation, the model achieves consistent enhancement effects in all directions. Although these baseline models can perform directional speech enhancement within a predefined target region, their performance significantly declines when the target speaker position varies. This limitation requires retraining the model for new target directions, increasing complexity and computational demand. Moreover, this capability highlights the versatility of our model, as it can be effectively deployed in many real-world scenarios without the need for model training for specific scenarios. This feature reflects the strong adaptability of the model and its potential for widespread application.

[0113] D. Enhanced signals for downstream ASR tasks

[0114] The same model and the same test dataset from the first fixed target speaker position experiment were used to evaluate the model's performance. By evaluating the average performance of each test subset, the goal was to quantify the model's enhancement effect in the speech recognition task. This experiment, as a supplement to the fixed target experiment, aimed to verify the model's enhancement effect on downstream tasks. The results in Table IV further demonstrate that the CDUNet model proposed in this application not only achieves high scores in speech quality but also exhibits excellent compatibility in downstream tasks.

[0115] Table IV. Downstream task performance for fixed target speaker position (training target angle is approximately 90°).

[0116]

[0117] Through the embodiments of the present application, a directional multi-channel directional speech enhancement method based on three-guide spatial selection is proposed, three guide vectors are integrated with a U-Net architecture to determine the target direction and the enhancement range. By introducing the width as an input parameter, the model can flexibly adjust the enhancement range according to the application scenario. Compared with the traditional beamformer and various neural network baselines, the CDUNet proposed herein not only provides excellent front-end performance in fixed and variable target directions, but also improves the performance of the back-end ASR.

[0118] Thus, the beamformer is combined with the causal U-shaped network model. The model processes the raw audio from the microphones, uses the output of the beamformer and the target and width parameters to achieve directional enhancement of the target speaker. The three-guide spatial selection method not only effectively improves the speech quality in various scenarios, but also demonstrates excellent performance in the downstream ASR task. In addition, the model can efficiently operate using only two microphones, which is suitable for practical device applications such as hearing aids.

[0119] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of actions, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other orders or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment is focused on, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0120] In some embodiments, the embodiments of the present application provide a non-volatile computer readable storage medium, wherein one or more programs including execution instructions are stored in the storage medium, the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute any directional speech enhancement method of the present application.

[0121] In some embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program stored on a non-volatile computer readable storage medium, the computer program includes program instructions, when the program instructions are executed by a computer, the computer executes any directional speech enhancement method described above.

[0122] In some embodiments, the electronic device provided by the embodiments of the present application further includes at least one processor, and a memory connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the directional speech enhancement method.

[0123] Figure 5 is a schematic diagram of the hardware structure of the electronic device for performing the directional speech enhancement method provided by another embodiment of the present application, as shown in the figure, the device includes: Figure 5

[0124] one or more processors 510 and a memory 520, Figure 5 wherein the processor 510 is taken as an example.

[0125] The device for performing the directional speech enhancement method can further include an input device 530 and an output device 540.

[0126] The processor 510, the memory 520, the input device 530 and the output device 540 can be connected through a bus or other means, Figure 5 wherein the connection through the bus is taken as an example.

[0127] The memory 520 as a non-volatile computer readable storage medium can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the directional speech enhancement method in the embodiments of the present application. The processor 510 performs various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 520, that is, implements the directional speech enhancement method of the above method embodiments.

[0128] The memory 520 can include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function; the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 520 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 520 can optionally include a memory remotely arranged relative to the processor 510, and these remote memories can be connected to the electronic device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0129] The input device 530 can receive input digital or character information, and generate signals related to the user settings and function control of the electronic device. The output device 540 can include a display device such as a display screen.​

[0130] The one or more modules are stored in the memory 520 and, when executed by the one or more processors 510, perform the directional speech enhancement method of any of the method embodiments described above.

[0131] The product described above can perform the method provided by the embodiments of the present application, has the corresponding function modules and beneficial effects of performing the method. Technical details not described in detail in the embodiments can be referred to the method provided by the embodiments of the present application.

[0132] The electronic device of the embodiments of the present application exists in various forms, including but not limited to:

[0133] (1) Mobile communication device: The feature of this kind of device is to have mobile communication function, and to provide voice and data communication as the main target. This kind of terminal includes: smart phone, multimedia phone, functional phone, and low-end phone, etc.

[0134] (2) Ultra-mobile personal computer device: This kind of device belongs to the category of personal computer, has the functions of calculation and processing, and generally also has the characteristics of mobile Internet. This kind of terminal includes: PDA, MID and UMPC device, etc.

[0135] (3) Portable entertainment device: This kind of device can display and play multimedia content. This kind of device includes: audio and video player, handheld game machine, electronic book, and smart toy and portable car navigation device.

[0136] (4) Other onboard electronic devices with data interaction function, such as vehicle-mounted device installed on vehicle.

[0137] The device embodiments described above are only schematic, wherein the units illustrated as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0138] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software plus a general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions essentially or say the part that contributes to the related art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the methods described in each embodiment or some part of the embodiment.

[0139] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same. Although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing examples, or make equivalent replacements for some of the technical features therein. Such modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A directional speech enhancement method, comprising: Based on the target direction, forward edge angle, and reverse edge angle, the corresponding target-guided speech vector, forward edge-guided vector, and reverse edge-guided vector are extracted from the original audio to be enhanced. Beamforming operations are performed on the target guided speech vector, the positive edge guided vector, and the reverse edge guided vector to obtain the corresponding beam-enhanced speech; Each of the beam-enhanced speech segments and the original audio are input into the speech enhancement model to generate target-enhanced speech corresponding to the target direction.

2. The method according to claim 1, wherein, The forward edge angle and the reverse edge angle are equal in width relative to the target direction by the angle of deviation.

3. The method according to claim 2, wherein, The angle width is not less than 3°.

4. The method according to claim 1, wherein, The speech enhancement model comprises a cascaded encoder, denoiser, and decoder; The noise reduction unit includes multiple DPRNN blocks for sequence modeling, each of which includes a frequency sequence layer and a long short-term memory layer.

5. The method according to claim 4, wherein, The encoder comprises multiple encoder blocks, the decoder comprises multiple decoder blocks, and the speech enhancement model comprises multiple CBAM modules; The first CBAM module is used to bridge the output of the first encoder block and the input of the first decoder block.

6. The method according to claim 4, wherein, The original audio is multi-channel audio; The speech enhancement model is used to filter the nearest channel audio targeting the speaker from the multi-channel audio, and generate the target enhanced speech based on the nearest channel audio and the mask signal output by the decoder; The nearest channel audio is the channel audio with the smallest deviation angle between the corresponding pickup direction and the target direction among all channel audios.

7. The method according to claim 4, wherein, The loss function of the speech enhancement model is: In the formula, This represents the loss function of the speech enhancement model. It is a set of STFT points. Indicates the first MR-STFT loss at each STFT point Indicates the first SI-SNR loss at each STFT point, and It is the weighting factor for the loss term; It is the target speech signal, representing the actual speech signal; It is the predicted enhanced speech signal, representing the speech signal output by the model.

8. The method according to claim 4, wherein, The speech enhancement model employs a convolutional U-Net network architecture.

9. A storage medium having a computer program stored thereon, wherein, When executed by a processor, the program implements the steps of the method described in any one of claims 1-8.

10. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-8.

Images