Directional speech enhancement method and device, storage medium and program product

Abstract

The invention provides a directional speech enhancement method and device, a storage medium and a program product, and relates to the technical field of speech processing. Respectively extracting a corresponding target steering voice vector, a forward edge steering vector and a reverse edge steering vector from the original audio to be enhanced; performing beam forming operation on the target steering voice vector, the forward edge steering vector and the reverse edge steering vector to obtain corresponding beam enhanced voice; and inputting each beam enhanced voice and the original audio into a voice enhancement model to generate a target enhanced voice corresponding to the target direction. Therefore, by using a three-steering vector selection strategy, spatial information is effectively utilized, the system can dynamically adapt to enhancement of the target voice and suppress interference voice from other directions at the same time, and the practicability and accuracy of voice enhancement are improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a directional speech enhancement method, device, storage medium and program product. Background Art

[0002] In recent years, speech enhancement (SE) in very low signal-to-noise ratio (SNR) environments has received extensive attention due to its key applications in telecommunications, hearing aids, and automatic speech recognition (ASR) systems. In these environments, the main challenges include background noise, speech interference, and reverberation, which can severely degrade the intelligibility and quality of speech. Among them, the most challenging is human speech interference, because speech recognition systems have difficulty distinguishing the target speech from the interfering speech, resulting in a significant performance degradation.

[0003] In order to effectively distinguish the target speech from other interfering speech, some scholars in the industry have proposed some directional speech enhancement systems, which can enhance the speech of the target speaker according to the specified target direction and suppress the speech signals of interfering speakers from other directions. Specifically, non-neural network speech enhancement is achieved by adopting the delay and sum beamforming method and the generalized sidelobe canceler (GSC) adaptive beamforming method.

[0004] In addition, some technology companies have also proposed speech enhancement systems based on neural networks, such as Google's "BASNet" and "GSENet". However, the target enhancement direction of these neural network-based systems is usually set before training, and can only be applied when the direction of the target speaker remains unchanged. If the speaker's direction changes, the model needs to be retrained, which consumes a lot of resources. Specifically, it is usually necessary to first determine the directions of the target speaker and the interfering speaker, and then train the model to enhance the target direction and suppress the interfering direction. In this way, the trained directional speech enhancement model can enhance speech in a fixed direction. Therefore, when the target speaker changes, the speech enhancement performance of such directional speech enhancement models is poor.

[0005] The industry has not yet proposed a better solution to the above problems. Summary of the invention

[0006] The present application provides a directional speech enhancement method, device, storage medium and program product, which are used to at least solve the problem that the speech enhancement system based on neural network in the current related technology has poor performance when the direction of the target speaker changes.

[0007] In a first aspect, an embodiment of the present application provides a directional speech enhancement method, comprising: extracting corresponding target-oriented speech vectors, forward edge-oriented vectors, and reverse edge-oriented vectors from the original audio to be enhanced according to the target direction, forward edge angle, and reverse edge angle; performing beamforming operations on the target-oriented speech vectors, the forward edge-oriented vectors, and the reverse edge-oriented vectors to obtain corresponding beam-enhanced speech; and inputting the beam-enhanced speech and the original audio into a speech enhancement model to generate target-enhanced speech corresponding to the target direction.

[0008] In a second aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the directional speech enhancement method of any embodiment of the present application.

[0009] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, the steps of the directional speech enhancement method of any embodiment of the present application are implemented.

[0010] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the directional speech enhancement method of any embodiment of the present application.

[0011] The beneficial effects of the embodiments of the present application are: By using a three-directional vector selection strategy, spatial information is effectively utilized, and speech information in different directions can be flexibly extracted and distinguished, ensuring that even if the direction of the target speaker changes, the system can dynamically adapt, enhance the target speech while suppressing interfering speech from other directions, thereby improving the practicality and accuracy of speech enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A flowchart of an example of a directional speech enhancement method according to an embodiment of the present application is shown; Figure 2 A schematic diagram showing a structural connection of an example of a speech enhancement model according to an embodiment of the present application is shown; Figure 3 A schematic diagram of a CDUNet architecture according to an embodiment of the present application is shown; Figure 4 A schematic diagram showing the effect of the simulation setting for the first fixed target data set; Figure 5 It is a schematic structural diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0015] It is understood that humans are able to focus attention in a specific direction using binaural spatial information. Similarly, modern hearing aids, speech recognition systems, and other devices are equipped with multiple microphones. Therefore, multichannel speech enhancement has become increasingly important, and many researchers have begun to explore the use of spatial features for speech separation or speaker extraction. Traditional multichannel speech enhancement techniques, such as delayed summation and beamformers such as Generalized Sidelobe Canceller (GSC), have limitations in performance. Recently, many neural network methods have surpassed traditional methods in terms of quality and intelligibility of speech enhancement, such as EMGSE, VSEGAN, METRICGAN-U, HGCN, TF-GridNet, and FullSubNet+. However, these methods also have some limitations: First, they usually predefine the target speech area without inputting spatial information when used, such as GSENet (guided speech enhancement network) and BASNet (binaural angular separation network). Second, they mainly focus on scenarios with three or more microphones, such as multiple extraction methods, JNF (Joint Normalization Function), and JNF-SSF (Joint Normalization Function with Single-SidedFiltering). Using too many microphones is not practical in on-device speech enhancement systems because resource consumption increases rapidly with the number of microphones, and these systems are limited by memory and computing resources. In addition, the large-scale parameters of neural networks limit their feasibility in practical applications. Finally, previous work mainly focuses on improving speech quality indicators, but often ignores their impact on downstream tasks.

[0016] In current related technologies, more than three microphones are usually used for directionally enhancing speech, which is expensive. In addition, neural network-based systems usually have a large number of parameters and are difficult to use on end-side devices. In addition, when the target speaker changes, the phase of the incoming spectrogram needs to be manually changed to shift the target direction to the target direction during training, which consumes a lot of resources.

[0017] In view of this, it is proposed in the embodiments of the present application that not only the target angle but also the "width" is passed in as a parameter to prompt the model of the interference direction, thereby providing users with a solution that is more adaptable to various environments. The enhancement width can be set during training according to the actual environmental conditions. At the same time, this also requires the model to have stronger adaptability and the ability to switch directions at any time.

[0018] Figure 1 A flowchart of an example of a directional speech enhancement method according to an embodiment of the present application is shown.

[0019] like Figure 1 As shown, in step S110, the corresponding target-oriented speech vector, forward edge-oriented vector and reverse edge-oriented vector are respectively extracted from the original audio to be enhanced according to the target direction, the forward edge angle and the reverse edge angle.

[0020] Here, the target direction may be the direction of the target speaker sensed by a user input or a sensor, for example, the target direction may be accurately determined by a direction estimation algorithm (such as a sound source localization algorithm). The forward edge angle and the reverse edge angle respectively define two boundary directions within a certain angle range related to the target direction in space. It should be understood that the forward edge angle and the reverse edge angle may be pre-set or pre-calibrated parameters.

[0021] In addition, the target-oriented speech vector is extracted based on the speech signal in the target direction, and the vector contains the speech information of the potential target speaker. The forward edge-oriented vector is extracted based on the audio signal of the forward edge angle relative to the target direction, and the vector may contain the signal (such as interference or noise) of this deviation direction. The reverse edge-oriented vector is extracted based on the audio signal of the reverse edge angle relative to the target direction to capture the signal from the other side. By extracting the oriented speech vector in the target direction and the edge angles on both sides of it, the speech information and interference information of the spatial area where the sampled speaker target is located can be effectively covered.

[0022] In step S120, a beamforming operation is performed on the target-oriented speech vector, the forward edge-oriented vector, and the reverse edge-oriented vector to obtain corresponding beam-enhanced speech.

[0023] Here, processing can be performed by a variety of non-restrictive beamforming algorithms. For example, by adopting technologies such as Delay-and-Sum Beamforming or adaptive beamforming (such as generalized sidelobe canceller, GSC), the phase and amplitude of the input signal are adjusted to enhance the speech signal from the target direction, while suppressing interference signals from other directions to generate a corresponding beam-enhanced speech signal. In addition, for different steering vectors, corresponding beamforming parameters such as gain, delay, etc. can be configured separately to optimize the speech signal in the target direction. Therefore, by combining multiple steering speech vectors, more features of the target speech can be captured, so that the enhanced speech signal has higher clarity and intelligibility.

[0024] In step S130, the beam-enhanced speech and the original audio are input into a speech enhancement model to generate target enhanced speech corresponding to the target direction.

[0025] Here, the speech enhancement model can adopt a model based on a deep neural network, such as a convolutional neural network (CNN), a long short-term memory network (LSTM) or a Transformer structure. In this way, through the learning ability of the neural network, the input audio features are analyzed, the speech signal in the target direction is further amplified, and the interference signal in the non-target direction is further reduced, so as to achieve the enhancement effect of optimizing the target speech.

[0026] In some examples of the embodiments of the present application, the angle widths of the forward edge angle and the reverse edge angle relative to the target direction are equal. In this way, the corresponding forward edge angle and reverse edge angle can be automatically determined by simply setting the angle width in the speech enhancement system.

[0027] In some business scenarios, the angle width can be flexibly adjusted and can be set or updated according to the actual interference direction or business needs.

[0028] It can be seen from experiments that the angle width is not less than 3°. When the width is less than 3°, the input width is close to the target direction, resulting in a weakened ability to obtain effective information, and the performance is degraded. Exemplarily, the angle width ranges from 3° to max (boundary angle), where the boundary angle is the estimated maximum angle between the target direction and the interference direction (which can be determined according to actual conditions). More details will be demonstrated and expanded in conjunction with the experimental part below.

[0029] Figure 2 A structural connection diagram of an example of a speech enhancement model according to an embodiment of the present application is shown.

[0030] like Figure 2 As shown, the speech enhancement model 200 includes a cascaded encoder 210, a noise reducer 220 and a decoder 230.

[0031] Here, the input audio signal is converted from the time domain to a representation space (or feature space) more suitable for neural network processing by the encoder 210, and the original beam-enhanced speech and the original audio are processed by the encoder to generate a feature sequence. Exemplarily, the encoder can use a convolutional neural network to extract local time domain features of the audio signal, for example, by processing the audio through a sliding window, capturing frequency information at different time points, and mapping the time domain audio signal to a higher dimensional feature space.

[0032] The denoiser 220 includes a plurality of DPRNN (Dual-Path Recurrent Neural Network) blocks for sequence modeling, each DPRNN block including a frequency sequence layer and a long short-term memory (LSTM) layer. Specifically, the DPRNN block divides the input sequence into shorter blocks and uses two RNN layers to model from two division and recombination directions, respectively for local and global, i.e., the shorter frequency dimension and the longer time dimension.

[0033] Here, the encoded audio feature sequence is subjected to denoising by the denoiser 220, and the DPRNN can simultaneously process information in the time domain and the frequency domain, thereby better removing noise. In each DPRNN block, the frequency dimension of the input feature is modeled by the frequency sequence layer, so that the network can capture the frequency relationship and correlation in the feature, thereby distinguishing the speech signal from the noise signal. The time dimension of the input feature is processed by the LSTM layer, thereby capturing the temporal change pattern of the speech signal. In this way, the model can process audio data with a long time span, and effectively isolate and suppress interference and noise in the time domain. Through the "dual path" structure of the DPRNN, the time domain and frequency domain features are processed alternately, so that the network can capture the dynamic changes of the audio signal in these two dimensions, thereby more effectively performing speech enhancement. Through the processing of multiple DPRNN blocks, the system can work stably in a dynamic and complex acoustic environment, not only suitable for single background noise, but also adapt to multiple mixed noises and multiple speakers, especially when used in application scenarios such as hearing aids and speech recognition, it has strong robustness and adaptability.

[0034] The feature sequence processed by the noise reducer is reconverted into a time-domain audio signal through the decoder 230, thereby generating the final enhanced speech or related speech feature representation. Specifically, the decoder can adopt an architecture symmetrical to the encoder (for example, using a deconvolution layer or an upsampling layer) to achieve time-domain reconstruction of the features. Exemplarily, the decoder may include a deconvolution (Deconv) layer or a transposed convolution layer to restore the features after the noise reduction process to an audio signal. By gradually enlarging the feature space, the decoder can restore the time domain information of the original signal. Based on the DPRNN processing results, the decoder can ensure that the generated enhanced speech has a high signal-to-noise ratio while maintaining the naturalness and coherence of the speech signal.

[0035] Through the embodiment of the present application, based on the cascade structure of the encoder, the denoiser of multiple DPRNN blocks and the decoder, an effective processing flow from the time domain audio signal to the feature space and then back to the time domain is realized. The encoder extracts features, the DPRNN in the denoiser processes the time-frequency information of the audio signal through dual paths, and finally the decoder reconstructs the enhanced speech signal. As a result, efficient and accurate speech enhancement effects can be achieved in complex acoustic environments, making the target speech clearer and more natural, while effectively suppressing noise and improving the performance of speech recognition and communication systems.

[0036] In some examples of the embodiments of the present application, the encoder includes a plurality of encoder blocks, and the decoder includes a plurality of decoder blocks, and the speech enhancement model includes a CBAM module (Convolutional Block Attention Module), and the first CBAM module is used to bridge the output end of the first encoder block and the input end of the first decoder block. Here, the term "first" is not specific and may refer to any selected object in some cases.

[0037] Exemplarily, each encoder block can respectively adopt a two-dimensional convolution block, and each decoder block can respectively adopt a two-dimensional deconvolution block. In each encoder block, the convolution kernel is used to capture local features in the audio data, thereby extracting higher-level features layer by layer. A pooling layer can be introduced between each convolution block for downsampling to compress the input time-frequency features to a lower dimension, which not only reduces the amount of calculation, but also increases the focus on key features. In each decoder block, the features compressed by the encoder are upsampled and restored to ensure that key time-frequency information is not lost during the speech enhancement process. In addition, based on skip connections, the output of the encoder block is connected to the input of the corresponding decoder block through the CBAM module to retain low-level time-frequency feature information, which can effectively improve the reconstruction capability of the decoder.

[0038] Here, the CBAM module includes a channel attention module and a spatial attention module. By recalibrating the feature map output by the encoder, the weight of the feature map is adaptively allocated to ensure that the model pays more attention to the time-frequency area that is beneficial to target enhancement. Based on the cross-connection mechanism of the CBAM module, the input of the decoder is further enhanced, so that the model can use more useful time-frequency information in the reconstruction process, greatly improving the accuracy and clarity of speech enhancement.

[0039] In some examples of the embodiments of the present application, the original audio is multi-channel audio, for example, including audio data collected by two or more microphones (Mic.), each microphone may be arranged in a different position to collect audio signals in different directions and capture sound information from different spatial angles. The collected original audio signal is subjected to a short-time Fourier transform (STFT) to convert the time domain audio signal into time-frequency domain features, and the frequency and phase information of the audio signals of different channels are extracted. The speech enhancement model is used to filter the nearest channel audio for the speaker target from the multi-channel audio, and generate the target enhanced speech based on the nearest channel audio and the mask signal output by the decoder, wherein the nearest channel audio is the channel audio with the smallest angle of deviation between the corresponding pickup direction and the target direction in each channel audio.

[0040] In some embodiments, the direction of the target speaker is determined by a sensor or a direction estimation algorithm (such as a sound source localization technology based on a microphone array), and then the deviation angle between the pickup direction and the target direction is calculated for the audio signal of each channel. According to the calculated deviation angle, the channel audio with the smallest deviation angle is selected as the "closest channel audio", that is, the channel audio is closest to the target direction and thus contains the clearest target speech signal.

[0041] In addition, the decoder extracts the corresponding speech mask signal from the time-frequency features transmitted by the encoder and the noise reducer module, which is used to filter the input audio signal, suppress the interference noise and highlight the target speech signal. Here, the system applies the mask signal output by the decoder to the filtered nearest channel audio to generate a masked time-frequency feature map, in which the interference components are greatly weakened and the time-frequency components of the target speech signal are significantly enhanced. Finally, the system converts the processed feature map back to the time domain signal through the inverse STFT to generate the target enhanced speech signal.

[0042] Through the embodiments of the present application, the audio closest to the target direction is screened out from the multi-channel audio to ensure that the captured channel voice signal is the clearest. The mask signal generated by the decoder can efficiently enhance the target voice, suppress interference noise and background sound, and significantly improve the signal-to-noise ratio of the voice. Then, by fusing the mask signal output by the decoder and the clearest channel voice signal for voice enhancement, it can dynamically adapt to the changing sound source position and environmental noise, and is suitable for various complex multi-talker and noise scenes, with strong adaptability and flexibility.

[0043] It should be noted that in the short-time Fourier transform domain, the target and estimated output are represented by s and Preliminary studies have shown that using the Scale-Invariant Signal-to-Noise Ratio (SI-SNR) loss function significantly enhances the stability of the learning process. However, when the network is trained using only the SI-SNR loss, it is observed that the network has a tendency to over-suppress low-frequency components. To address this issue, an innovative combined loss function is proposed that combines the SI-SNR and Multi-Resolution STFT (MR-STFT) losses.

[0044] More specifically, the loss function of the speech enhancement model is: , Formula (1) In the formula, represents the loss function of the speech enhancement model, is the set of STFT points, Indicates The MR-STFT loss of STFT points is Indicates SI-SNR loss of STFT points, and is the loss term weighting factor; is the target speech signal, which represents the real speech signal; is the predicted enhanced speech signal, which represents the speech signal output by the model.

[0045] Through the embodiments of the present application, the MR-STFT loss and the SI-SNR loss are combined, and the loss function jointly optimizes the speech enhancement in both the frequency domain and the time domain, ensuring that the model can simultaneously improve the spectral characteristics of the speech (through MR-STFT) and the time domain signal-to-noise ratio (through SI-SNR), so that the speech enhancement is not only more delicate in the spectrum, but also can significantly reduce noise, improve the overall signal-to-noise ratio of the enhanced speech, and can maintain a high enhancement effect in different noise scenarios, different frequency characteristics, and a variety of complex environments.

[0046] It should be noted that in multi-speaker scenarios, exploiting spatial features is crucial for enhancing the target speech. However, developing a compact multi-channel speech enhancement system with limited microphone arrays remains challenging, especially under extremely low signal-to-noise ratio conditions.

[0047] In some examples of the embodiments of the present application, a neural-guided speech enhancement method for a dual-microphone array in a high-noise scene is provided, specifically a three-guided space selection method, which is a flexible framework that uses three guide vectors to guide the enhancement and determine the enhancement range. Specifically, the speech enhancement model adopts a convolutional U-Net network architecture. By introducing a causal-directed U-Net (CDUNet) model, the model takes the original multi-channel speech and the desired enhancement width as input. This makes it possible to dynamically adjust the guidance vector according to the target direction and fine-tune the enhancement area according to the angular separation between the target and the interference signal. The model provided in the embodiment of the present application uses only a dual-microphone array and performs well in both speech quality and downstream task performance. It operates in real time with minimal parameters and is very suitable for low-latency, on-device streaming applications.

[0048] Specifically, first, generate more than 100 hours of speech and interference speech with a target direction of 0-180° as a training set for training, so that the model has the ability to enhance in various directions. During the training process, the correct direction of the model is input into the model, and the desired enhancement angle width is input at the same time. The model generates three steering vectors for beamforming based on the target direction and the direction of the target direction plus or minus the width, and then compares these beamforming results with the original spectrum. Figure 1 The UNet model designed with the same input is used. This helps the model learn to enhance the speech in the target direction and distinguish the target speech from the interfering speech according to the input width, which enhances the model's grasp of the real scene. At the same time, the downstream speech recognition task was tested and good test results were obtained. Finally, a dual-microphone directional enhancement model was obtained that can adjust the input width according to the actual situation and does not need to adjust the input data.

[0049] Through the embodiments of the present application, a large amount of existing data can be fully utilized, which greatly improves the robustness of the model. At the same time, this solution can generate three derivative vectors according to the precise angle of the input and the width to be enhanced, and can accurately divide the target range to be enhanced, and the enhanced speech quality score is high. In addition, this solution can flexibly adjust the input width to adapt to various scenarios, with a wide range of applications and high flexibility. In addition, the model is compressed and can be used on the end-side device; only dual microphones can be used to achieve directional enhancement of the model, and few resources are required. In addition, it has also been tested on downstream tasks of speech recognition, and the performance of the backend is good.

[0050] Specifically, a causal-directed U-Net (CDUNet) model is proposed, which combines U-Net with beamforming. In the embodiment of the present application, the spatial position of the target speaker is used as a clue to create a nonlinear filter that can be flexibly adjusted to the selected direction. In addition, by introducing variable enhancement widths for different interference angles, the model is able to extract detailed information of the interference signal and dynamically adjust the enhancement range.

[0051] Ⅰ. Three-Guide Space Selection Method A. Problem Definition This study targets the so-called cocktail party problem: extracting the speech signal of the target speaker from the interfering speech. Assume that the interfering speaker comes from a different direction than the target speaker. Therefore, the problem can be described as follows: , Formula (2) in, Indicated by The input signal captured by the microphones, represents the target speech signal, Indicates interfering speech signal, and Represent the azimuth of the target and interference sound source relative to the microphone. The goal of speech enhancement is to train a deep neural network ,Will Mapping The enhanced signal It is estimated by: , Formula (3) in, represents the network parameters, represents the predicted output, Represents a two-channel speech signal captured by a microphone.

[0052] B. Three-Guide Space Selection In the solution provided in the embodiment of the present application, the target angle and two edge angles are used, and the two edge angles are calculated by subtracting and adding the input width from the target angle to generate three steering vectors for the beamformer. The network then takes as input the frequency domain representation of the two original microphone signals and the beamformer output at the target angle and the two edge angles. This enables the model to accurately locate the direction of the target speaker. In addition, by combining the input width, the model can estimate the angular separation between the target and the interference source, thereby achieving more accurate directional enhancement.

[0053] C. Causal Guided U-Net Model Figure 3A schematic diagram of a CDUNet architecture according to an embodiment of the present application is shown.

[0054] like Figure 3 As shown in Figure 2, the model adopts a convolutional U-network architecture. The beamformer output combines the target direction and width inputs to capture the spatial region information that is critical for enhancement. Indicates the range of the target area to be enhanced. Indicates the direction of the specified target speaker. The "Closest Microphone Selection" operation selects the speech signal from the microphone that is closer to the target speaker.

[0055] Specifically, in the model, a powerful encoder-decoder architecture with skip connections is used to build a deep neural network. The encoder consists of 3 two-dimensional convolution (Conv2D) blocks to encode the input features into potential representations. Correspondingly, the decoder consists of 3 two-dimensional deconvolution (ConTrans2D) blocks to decode the latent space features back to the feature space.

[0056] Between the encoder and decoder, a sequence modeling module is integrated, including a frequency sequence layer and a long short-term memory (LSTM) layer, following the dual-path recurrent neural network (DPRNN) framework.

[0057] In addition, a convolutional block attention module (CBAM) is introduced, which combines channel and spatial attention mechanisms. In the model, CBAM is applied in the decoder and jump connections to recalibrate the time-frequency (TF) feature map, thereby improving the accuracy of target reconstruction.

[0058] make For the The output of the encoder block, where . No. Channel attention gates for blocks The calculation is as follows: , Formula (4) in, represents the Sigmoid activation function, and represents the weights of the shared linear layer. Apply average pooling and max pooling , and then apply the channel attention gate to Calculate . Spatial Attention Gate The calculation is as follows: , Formula (5) in, is a convolutional layer (Conv2D). The spatial attention gate is then applied to Calculate .

[0059] The output speech signal is estimated by applying the mask generated by the decoder to the closer original channels. The closer channels are selected based on their proximity to the input target angle, and the closer channels are selected for processing. For example, if the input angle is less than 90°, the first channel is selected as the closer channel, otherwise the second channel is used.

[0060] II. Experimental Setup A. Dataset All clean and jammed speech are randomly sampled from LibriSpeech and internal corpora. The training dataset is simulated in the following way: • Geometry: Generate rooms with different geometries based on the JNF benchmark, e.g. Figure 2 As shown in Figure 2. In each simulated room, the microphone spacing was set to 30 mm. The audio sources were randomly selected, with one designated as the target source and the other as the interference source.

[0061] • Mixing: The original audio captured by two microphones is synthesized using a simulated Room Impulse Response (RIR). For training, the ground truth signal comes from the early reverberation component of the clean signal with a reverberation delay of 150 ms.

[0062] Based on this approach, two datasets are synthesized for the baseline model and the proposed CDUNet: • Fixed Target Dataset: The Fixed Target dataset is used to compare the CDUNet model with beamformers and other neural network baselines that do not accept angles as input features. The dataset contains 250,000 training samples with the target direction fixed between 85° and 95°. The signal-to-noise ratio (SNR) of the target speech relative to the interfering speaker mixture varies from -5 dB to 10 dB between samples.

[0063] • Variable Target Dataset: A variable target dataset was developed to train neural networks for directional speech enhancement. This dataset introduces variations in the target speaker position, which is randomly generated, while the distractor direction is always 15° away from the target direction. The SNR and number of speech in this dataset are consistent with the fixed target dataset.

[0064] Figure 4 Figure 2 shows the effect of the simulation setting for the first fixed target data set. Figure 4 As shown, the target directions range from 85° to 95°, represented by red stars in the figure, while the distractor directions are 15° away from the target directions, represented by green stars. The room information is uniformly sampled from the range provided in the table.

[0065] B. Model Setup The input Short-Time Fourier Transform (STFT) and the output inverse STFT use a window size of 512 and a step size of 256. The STFT representation is treated as independent channels and decomposed into corresponding phase and amplitude components, ultimately forming a 10-channel frequency domain input for the CDUNet model.

[0066] C. Evaluation Evaluation process. We constructed evaluation datasets for fixed and variable targets: • For fixed target evaluation, a corresponding test set was created with target angles , interference angle range The test set includes 500 speech clips from 0° to 180° with signal-to-noise ratio (SNR) of 0 dB and 5 dB.

[0067] • For the variable target evaluation, 500 speech clips were generated with 0 dB SNR and target direction The range is from 0° to 90°, and its PESQ score is evaluated. This angle setting is adopted because 0°-90° and 90°-180° are symmetrical for the two-microphone model.

[0068] Baseline models. Two types of baseline models were selected: •U-Net Free. DAS (Delay and Sum) is the traditional delay and sum beamformer model, GSC is the generalized sidelobe canceller model, and JNF (Joint Normalization Function) stands for the joint spatial and time-frequency nonlinear filter model, which uses a circular array of three microphones and produces excellent results that are not captured in Table I.

[0069] Table I. PESQ scores in fixed regions (trained filters with target angles around 90°). Indicates the direction of interfering speech, and Noisy Sp. indicates noisy speech.

[0070]

[0071] •U-Net Based. U-Net is based on Figure 3 The IPD Unet model introduces the interchannel phase difference (IPD) between microphones, a well-known spatial feature, as input into the U-Net architecture, while the BF UNet supplements the output of the beamformer on the basis of the U-Net model, which is different from the CDUNet model proposed in the embodiment of the present application, which has a width input.

[0072] Evaluation Metrics. Perceptual Evaluation of Speech Quality (PESQ) was performed on the speech enhancement. In addition, the enhanced speech was input into a pre-trained ASR model to calculate the word error rate (WER).

[0073] III. Results and Analysis A. Fixed area and directional training Here, the enhancement effect is evaluated when the target speaker position remains unchanged while the interfering speech direction changes. All models are trained on the fixed target dataset, where the target speaker position is roughly 90°, consistent with the test conditions. The first row of each SNR setting in Table I shows the PESQ scores for different interfering angles. All U-Net based models outperform the U-Net Free model, showing the superior performance of the U-Net architecture.

[0074] The results of the U-Net based model show that including IPD significantly degrades performance, while the beamformer output has relatively little impact on performance. When trained on a fixed target dataset, the proposed CDUNet model has only 74.4k parameters, but achieves significant performance improvement over the U-Net structure baseline model, which highlights Effectiveness in improving performance. Moreover, despite the variable speaker position, CDUNet trained with a variable target dataset still performs comparable to the fixed region enhancement results. This achievement is particularly remarkable because CDUNet has only 74.4k parameters, while JNF has up to 1M parameters. Therefore, the proposed CDUNet is able to learn not only one spatial filter, but also 180 spatial filters with fewer training samples in each direction (1400 instead of 250000). This efficiency of enhanced learning with fewer samples highlights the robustness and adaptability of the model when dealing with complex spatial data.

[0075] B. Different Impact on CDUNet To determine the optimal input width for CDUNet, we evaluated the PESQ scores at different input angles using a fixed target training dataset, examining the effect of varying The results in Table II show that when The model performs best when . The model performs beamforming based on the target direction and the edge angle. The edge angle is calculated by adjusting the target angle up or down by the specified input width. The introduction of the width input enables the model to distinguish the spatial distribution of interfering speech, which is achieved by comparing the edge speech signal with the target signal.

[0076]

[0077] Table II Fixed-target PESQ scores under different input widths.

[0078] It is worth noting that in the dataset, the angular separation between the target and interference directions is always more than 15°. Therefore, when the width is set to less than 15°, the model can effectively use the input width as a distinguishing boundary to separate the target speech. In contrast, when the width is reduced to 3°, the input width is close to the target direction, resulting in a weakened ability to acquire novel and effective information, which in turn slightly impairs the model performance. When the width exceeds 15°, the input width cannot separate the target direction from the interference direction, resulting in a decrease in performance. In practical applications, the input width can be flexibly adjusted to match the actual interference direction.

[0079] C. Directional Speech Enhancement Table III PESQ scores under target speaker position changes.

[0080]

[0081] As shown in Table III, the CDUNet model remains consistent in its ability to dynamically adapt to the target speaker using the input angle, achieving consistent enhancement in all directions regardless of the target angle. While these baseline models can perform directional speech enhancement within a predefined target region, their performance degrades significantly when the target speaker position changes. This limitation requires retraining the model for new target directions, increasing complexity and computational requirements. Moreover, this capability highlights the versatility of our model as it can be effectively deployed in many real-world scenarios without the need for scenario-specific model training. This feature reflects the strong adaptability of the model and its potential for widespread application.

[0082] D. Signal enhancement for downstream ASR tasks The performance of the model is evaluated using the same model and the same test dataset from the first fixed target speaker position experiment. The average performance of each test subset is evaluated to quantify the enhancement effect of the model in the speech recognition task. This experiment is a supplement to the fixed target experiment and aims to verify the enhancement effect of the model on downstream tasks. The results in Table IV further show that the CDUNet model proposed in the embodiment of the present application not only achieves high scores in speech quality, but also shows excellent compatibility in downstream tasks.

[0083] TABLE IV Downstream task performance with fixed target speaker position (training target angle is approximately 90°).

[0084]

[0085] Through the embodiments of the present application, a directional multi-channel directional speech enhancement method based on three-guide space selection is proposed, and three guide vectors are integrated with the U-Net architecture to determine the target direction and enhancement range. By introducing width as an input parameter, the model can flexibly adjust the enhancement range according to the application scenario. Compared with traditional beamformers and various neural network baselines, the CDUNet proposed here not only provides excellent front-end performance in fixed and variable target directions, but also improves the performance of background ASR.

[0086] Thus, a beamformer is combined with a causal U-network model. The model processes the raw audio from the microphone and uses the output of the beamformer along with the target and width parameters to achieve directional enhancement of the target speaker. The three-guided spatial selection method not only effectively improves speech quality in a variety of scenarios, but also demonstrates excellent performance in downstream ASR tasks. In addition, the model can run efficiently using only two microphones, making it suitable for real-world device applications such as hearing aids.

[0087] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of actions combined, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0088] In some embodiments, an embodiment of the present application provides a non-volatile computer-readable storage medium, in which one or more programs including execution instructions are stored. The execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute any of the above-mentioned directional speech enhancement methods of the present application.

[0089] In some embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer performs any one of the above-mentioned directional speech enhancement methods.

[0090] In some embodiments, the embodiments of the present application also provide an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a directional speech enhancement method.

[0091] Figure 5 is a schematic diagram of the hardware structure of an electronic device for executing a directional speech enhancement method provided by another embodiment of the present application, such as Figure 5 As shown, the device includes: One or more processors 510 and memory 520, Figure 5 A processor 510 is taken as an example.

[0092] The device for executing the directional speech enhancement method may further include: an input device 530 and an output device 540 .

[0093] The processor 510, the memory 520, the input device 530 and the output device 540 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0094] The memory 520 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the directional speech enhancement method in the embodiment of the present application. The processor 510 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 520, that is, the directional speech enhancement method in the above method embodiment is implemented.

[0095] The memory 520 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 520 may optionally include a memory remotely arranged relative to the processor 510, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0096] The input device 530 may receive input digital or character information and generate signals related to user settings and function control of the electronic device. The output device 540 may include a display device such as a display screen.

[0097] The one or more modules are stored in the memory 520, and when executed by the one or more processors 510, the directional speech enhancement method in any of the above method embodiments is performed.

[0098] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0099] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to: (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.

[0100] (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.

[0101] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0102] (4) Other onboard electronic devices with data interaction functions, such as on-board devices installed in vehicles.

[0103] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0104] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A directional speech enhancement method, comprising: According to the target direction, the forward edge angle and the reverse edge angle, the corresponding target-oriented speech vector, the forward edge-oriented vector and the reverse edge-oriented vector are respectively extracted from the original audio to be enhanced; Performing a beamforming operation on the target-oriented speech vector, the forward edge-oriented vector, and the reverse edge-oriented vector to obtain corresponding beam-enhanced speech; The beam-enhanced speech and the original audio are input into a speech enhancement model to generate a target-enhanced speech corresponding to the target direction.

2. The method according to claim 1, wherein: The angle widths of the positive edge angle and the negative edge angle relative to the target direction are equal.

3. The method according to claim 2, wherein: The angle width is not less than 3°.

4. The method according to claim 1, wherein: The speech enhancement model comprises a cascaded encoder, a noise reducer and a decoder; The denoiser comprises a plurality of DPRNN blocks for sequence modeling, each of the DPRNN blocks comprising a frequency sequence layer and a long short-term memory layer.

5. The method according to claim 4, wherein: The encoder comprises a plurality of encoder blocks, the decoder comprises a plurality of decoder blocks, and the speech enhancement model comprises a plurality of CBAM modules; The first CBAM module is used to bridge the output end of the first encoder block and the input end of the first decoder block.

6. The method according to claim 4, wherein: The original audio is multi-channel audio; The speech enhancement model is used to filter the nearest channel audio for the speaker target from the multi-channel audio, and generate the target enhanced speech according to the nearest channel audio and the mask signal output by the decoder; The nearest channel audio is the channel audio having the smallest deviation angle between the corresponding sound pickup direction and the target direction among all channel audios.

7. The method according to claim 4, wherein: The loss function of the speech enhancement model is: In the formula, represents the loss function of the speech enhancement model, is the set of STFT points, Indicates The MR-STFT loss of STFT points is Indicates SI-SNR loss of STFT points, and is the loss term weighting factor; is the target speech signal, which represents the real speech signal; is the predicted enhanced speech signal, which represents the speech signal output by the model.

8. The method according to claim 4, wherein: The speech enhancement model adopts a convolutional U-Net network architecture.

9. A storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

10. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 8.

Images