A personalized speech enhancement system and method based on inter-frame and intra-frame feature fusion

Abstract

The application discloses a kind of personalized speech enhancement system and method based on interframe and intraframe feature fusion, STFT feature extraction is carried out to mixed speech signal and reference speech signal, and time-domain waveform is converted into time-frequency domain signal;Second, the time-frequency characteristics obtained are encoded, and the reference speech speaker feature is input into the speaker encoder to extract the speaker personalized feature;Then, the speaker feature and mixed speech feature are input into the speaker extraction module, which includes multiple fusion modules and interframe intraframe processing layer, the encoded speech feature is enhanced by interframe and intraframe features in combination with speaker features, to eliminate the speech of non-target speaker;Then, the enhanced feature is decoded, and the enhanced speech time-frequency feature is output;Finally, the enhanced time-frequency feature is converted by ISTFT, and the final enhanced speech time-domain waveform is obtained.

Description

Technical Field

[0001] This invention belongs to the field of speech signal processing technology, and in particular relates to a personalized speech enhancement system and method based on inter-frame and intra-frame feature fusion. Background Technology

[0002] Personalized speech enhancement is a key technology in speech signal processing. Its core objective is to extract target speech from noisy or aliased speech, and it is widely used in scenarios such as intelligent voice interaction, remote conferencing, and hearing aids. With the development of microphone array technology, the demand for speech enhancement in multi-speaker scenarios is becoming increasingly prominent. Especially when there is interference from non-target speakers, traditional speech enhancement methods struggle to retain the characteristics of the target speaker, resulting in poor enhancement effects.

[0003] In existing technologies, speech enhancement methods are mainly divided into two categories: one is based on signal processing theory, such as beamforming and adaptive filtering, but its adaptability to complex scenarios is limited and it is difficult to handle nonlinear aliased speech signals; the other is based on deep learning, which uses neural networks to model time-frequency features to achieve enhancement, but most methods do not fully consider the speaker's individual characteristics and are insufficient in the coordinated processing of inter-frame (frame-to-frame relationship in time series) and intra-frame (time-frequency feature distribution within a single frame) features. They either only focus on intra-frame local features and ignore inter-frame temporal dependencies, or they rely too much on complex network structures (such as multi-layer LSTM or Transformer), resulting in high computational complexity and difficulty in deploying on resource-constrained devices. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a personalized speech enhancement system and method based on the fusion of inter-frame and intra-frame features, which can effectively fuse inter-frame and intra-frame features to enhance personalized features and improve the personalized speech enhancement effect.

[0005] The technical solution of this invention is implemented as follows:

[0006] A personalized speech enhancement method based on inter-frame and intra-frame feature fusion includes the following steps:

[0007] S1. Extract features from the mixed speech signal and the reference speech signal respectively, and convert the time-domain waveform into a time-frequency domain signal;

[0008] S2. Encode the extracted and converted mixed speech time-frequency features and reference speech time-frequency features, and output the encoded mixed speech time-frequency features and reference speech time-frequency features.

[0009] S3. Input the encoded reference speech time-frequency features into the speaker encoder, extract the speaker features, and output the speaker prediction results;

[0010] S4. Input the speaker features and mixed speech features into the speaker extraction module, which includes multiple feature fusion modules and inter-frame and intra-frame layers. Based on the speaker features, perform inter-frame and intra-frame feature enhancement on the encoded mixed speech time-frequency features to eliminate the speech of non-target speakers.

[0011] S5. Decode the obtained enhanced features and output the enhanced reference speech time-frequency features;

[0012] S6. The enhanced speech time-frequency features are transformed by inverse short-time Fourier transform, and the time-frequency domain signal is converted into a time-domain waveform to obtain the final model output.

[0013] Furthermore, in step S1, feature extraction and transformation are performed on the mixed speech signal and the reference speech signal respectively using short-time Fourier transform, and the formula is as follows:

[0014]

[0015] Where Y(n,k) represents the time-frequency features of the mixed speech and the reference speech, n represents the time frame, and k represents the frequency point; y(m) represents the input discrete time-domain speech signal; w(nm) represents the sliding window function—the Hanning window; and L represents the frame length parameter.

[0016] Furthermore, in step S2, the extracted and converted mixed speech time-frequency features and reference speech time-frequency features are input into the speech encoder for high-bit embedding encoding. The speech encoder includes a single convolutional layer and a group normalization layer. The speech encoder uses a sequence of convolutional layers, including convolution operations and group normalization operations.

[0017] The parameters of a convolutional layer include the number of input channels, the number of output channels, the kernel size, the stride, and padding; the output dimension is the same as the input dimension, and the number of output channels is the preset embedding dimension.

[0018] Determine the number of groups for group normalization, and normalize the convolution output features using the following formula:

[0019]

[0020] Where, μ g and Let γ and β be the mean and variance of each feature group, γ and β be the learnable parameters, and ε = 1e-5;

[0021] The output is the encoded mixed speech feature X. mix ∈R B×F×T×128 Reference speech features

[0022] X aux ∈R B×F×T×128 .

[0023] Furthermore, in S3, the time-frequency features of the input reference speech are processed by the TFGTCRN network, and after mean pooling, the speaker features are output through a linear layer, and then the speaker prediction result is output through another linear layer; the speaker encoder includes a TFGTCRN network and a sequence of linear layers.

[0024] The TFGTCRN network is an improvement on the GTCRN network, including upsampling and downsampling layers;

[0025] The upsampling layer consists of alternating time-axis decoupled attention (TCA) and frequency-axis decoupled attention (FCA) layers, while the downsampling layer consists of multiple layers of time-frequency axis decoupled attention (TFCA) layers connected together.

[0026] The temporal decoupled attention (TCA), frequency decoupled attention (FCA), and multi-layer temporal-frequency decoupled attention (TFCA) are composed of similar structures, including average pooling layers, one-dimensional convolutional layers, and batch normalization layers; and Sigmoid activation functions are combined with deep convolutional layers according to different modes "time", "freq", and "tf".

[0027] The formula for calculating the time-axis decoupled attention TCA is:

[0028]

[0029] Where T represents the time axis length of the input features after pooling, and O t (t,f) is the attention value of the pooled input feature in the time-axis decoupled attention, W t These are learnable weights;

[0030] The formula for calculating the frequency axis decoupled attention (FCA) is as follows:

[0031]

[0032] Where F represents the frequency axis length of the input feature after pooling, and O f (t,f) is the attention value of the pooled input features in the frequency-axis decoupled attention, W f These are learnable weights;

[0033] The formula for calculating multi-layer time-frequency axis decoupled attention (TFCA) is as follows:

[0034]

[0035] Where T represents the time axis length of the input features after pooling, F represents the frequency axis length of the input features after pooling, and W... t and W f These are the learnable weights corresponding to the time dimension and the frequency dimension, respectively, O tf(t,f) is the attention value of the pooled input feature in the time-frequency axis decoupled attention.

[0036] Furthermore, in S3, the speaker encoder consists of a TFGTCRN network and a linear layer, and the specific process includes:

[0037] The sampling layer on the TFGTCRN network consists of two layers of time-axis decoupled attention (TCA) and two layers of frequency-axis decoupled attention (FCA), which are alternately concatenated to capture the time-varying characteristics of harmonic structures and noise in speech.

[0038] The bottleneck layer in the middle of the TFGTCRN network adopts the Group-DualPath Recurrent Neural Network (G-DPRNN), which integrates the Group Recurrent Neural Network (GRNN) and the Dual-Path Recurrent Neural Network (DPRNN). The GRNN divides the input and hidden state into two groups, each using a recurrent layer with half the parameters. The DPRNN models the spectral pattern and temporal dependency through intra-frame bidirectional and inter-frame unidirectional RNNs, respectively.

[0039] The downsampling layer consists of multiple time-frequency decoupled attention (TFCA) layers cascaded together, with parameters identical to those of the upsampling layer;

[0040] After the TFGTCRN output features are averaged and pooled, they are passed through a linear layer to output speaker features, and then through another linear layer to output speaker prediction results.

[0041] Furthermore, in S4, the feature fusion module expands the speaker features into (B,1,C) dimensions, where B is the batch size and C is the number of feature dimension channels;

[0042] The encoded speech features are flattened into (B, T*F, C) dimensions, where T is the number of time frames and F is the number of frequency points;

[0043] The speaker features are adapted by calculating the attention between speaker features and flattened speech features through a multi-head attention mechanism.

[0044] The adapted speaker features are added to the original speaker features and expanded to the same dimension as the encoded speech features;

[0045] The fused features are mapped back to the preset embedding dimension through convolutional layers.

[0046] Furthermore, the inter-frame and intra-frame layers are composed of inter-frame blocks and intra-frame blocks respectively:

[0047] The fused features are input into inter-frame blocks, passing through a normalization layer, a one-dimensional convolutional layer, and a PReLU activation layer. A sliding window then captures local temporal features along the time dimension. These extracted local temporal features are then input into a bidirectional LSTM network. Finally, a transposed one-dimensional convolution is used to change the feature dimension, as shown in the formula:

[0048] Inter(x)=x+PReLU(Conv1D(Norm(x)))+

[0049] ConvTranspose1D(BLSTM(Unfold(x)))

[0050] Wherein, Norm represents layer normalization, Conv1D represents one-dimensional convolution, PReLU is the activation function, Unfold represents the feature unfolding operation, BLSTM represents a bidirectional recurrent neural network, and ConvTranspose1D is a transposed one-dimensional convolutional layer used to perform inverse operations on the features after one-dimensional convolution.

[0051] Specifically, the parameters of the one-dimensional convolutional layer are: number of input channels = number of output channels, kernel size = 5, stride = 1, number of groups = 8. The "same" padding mode (the same mode pads shorter sequences with zeros to make the two sequences the same length) ensures that the length of the output sequence after the convolution operation is consistent with the length of the input sequence. Then, after passing through a normalization layer, features are extracted through a sliding operation. A convolution kernel of size (4,1) and a stride of (2,1) are used. The features are then fed into a bidirectional LSTM network, where the LSTM input dimension is 256, the hidden layer parameters are set to 256, and the output dimension is 512. Finally, the output dimension of the LSTM is converted from 512 to 64 through a transposed one-dimensional convolution. The transposed convolution kernel size is 4 and the stride is 2.

[0052] Features output from inter-frame blocks are directly passed to intra-frame blocks and processed through a multi-head self-attention mechanism. After feature normalization, the multi-head attention mechanism calculates temporal correlation features, which are then dropout-free and residually connected to the current features. This mechanism randomly discards a portion of neurons during training, enabling the model to learn more robust features and thus improving generalization ability. The calculation formula is as follows:

[0053]

[0054] in, Let W be the query, key, and value projection matrix of the i-th head. O To output the projection matrix, d k For a single head, SoftMax represents the attention weights, and the head represents the feature dimension. iThis indicates single-head attention. Concat concatenates the weights of multiple single-head attention mechanisms. MultiHead(Q,K,V) represents the output features of the multi-head attention mechanism.

[0055] The features are then processed by an LS convolutional module consisting of a normalization layer, a one-dimensional convolutional layer, a SiLU activation layer, and an LS convolutional layer. Finally, the output features of the intra-frame block and the output features of the inter-frame block are residually concatenated to ensure that the model fully utilizes the feature information from both shallow and deep layers. The intra-frame block formula is as follows:

[0056] Intra(x)=x+Dropout(MHSA(LN(x)))+LSConv(SiLU(Conv1D(LN[x])));

[0057] Here, LN performs layer normalization on features, MHSA is a multi-head attention layer, Dropout performs regularization on the output to prevent overfitting, Conv1D represents one-dimensional convolution, SiLU is an activation function, and LSConv is a convolution module, as detailed below.

[0058] Furthermore, the LS convolutional module in the intra-block includes a large kernel sensing module and a small kernel aggregation module;

[0059] The large kernel perception module is used to capture a wide range of spatial context information of the input features and generate aggregate weights. The large kernel perception module first reduces the number of input feature channels to a preset dimension through point convolution, and then passes the kernel with a kernel size of (K). L ,K L Large kernel depthwise convolutions capture K features centered on each feature. L ×K L Neighborhood information is used to generate context-adaptive weights w through point convolution. i ;

[0060] The small-kernel aggregation module performs small-range adaptive aggregation of features based on aggregation weights. It divides the input feature channels into G groups and combines the weights w generated by the large-kernel perception module. i Remodeled into G×K S ×K S small core weight And by using the small kernel weights to assign K to each feature S ×K S Neighborhood features are convolved and aggregated to obtain output features.

[0061] Furthermore, in S5, the enhanced features are decoded by a speech decoder module. The speech decoder module uses a transposed convolutional layer, with the number of input channels being a preset embedding dimension, and the number of output channels, kernel size, stride, and padding being consistent with the kernel size in the speech encoder.

[0062] Specifically, the speech decoder module consists of a single-layer transposed two-dimensional convolutional layer, which transforms the enhanced time-frequency features from (B,C,T,F) to (B,2,T,F) so that the real and imaginary parts can be fused and the inverse short-time Fourier transform (ISTFT) can be performed to obtain the enhanced speech of the target speaker.

[0063] A personalized speech enhancement system based on inter-frame and intra-frame feature fusion applies a personalized speech enhancement method based on inter-frame and intra-frame feature fusion as described above.

[0064] Compared with the prior art, the present invention achieves the following beneficial effects:

[0065] This invention provides a personalized speech enhancement system and method based on inter-frame and intra-frame feature fusion. The system extracts STFT features from a mixed speech signal and a reference speech signal, converting the time-domain waveform into a time-frequency domain signal. Next, the acquired time-frequency features are encoded, and the speaker features of the reference speech are input into a speaker encoder to extract personalized speaker features. Then, the speaker features and mixed speech features are input into a speaker extraction module, which includes multiple fusion modules and inter-frame and intra-frame processing layers. These modules combine speaker features with the encoded speech features to perform inter-frame and intra-frame feature enhancement processing to eliminate non-target speaker speech. The enhanced features are then decoded to output the enhanced speech time-frequency features. Finally, the enhanced time-frequency features are subjected to ISTFT transformation to obtain the final enhanced speech time-domain waveform. The speaker encoder employs an improved TFGTCRN network, incorporating attention mechanisms such as TCA, FCA, and TFCA. The feature fusion module fuses speaker features and speech features through a multi-head attention mechanism. The inter-frame and intra-frame layers consist of inter-frame blocks and intra-frame blocks, processing inter-frame and intra-frame features respectively, thus improving the personalized effect of speech enhancement. Attached Figure Description

[0066] Figure 1 This is a flowchart of a personalized speech enhancement method based on inter-frame and intra-frame feature fusion provided in an embodiment of the present invention;

[0067] Figure 2 This is an overall structural diagram of the time-frequency domain-based inter-frame and intra-frame personalized speech enhancement network provided in this embodiment of the invention;

[0068] Figure 3 This is a structural diagram of the TFGTCRN speaker network, which is a personalized speech enhancement method based on inter-frame and intra-frame feature fusion provided in this embodiment of the invention.

[0069] Figure 4This is an InterIntra module structure diagram of a personalized speech enhancement method based on inter-frame and intra-frame feature fusion provided in this embodiment of the invention.

[0070] Figure 5 This is an LS convolutional structure diagram of a personalized speech enhancement method based on inter-frame and intra-frame feature fusion provided in this embodiment of the invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0072] Example

[0073] like Figures 1 to 5 A personalized speech enhancement method based on inter-frame and intra-frame feature fusion is described below, along with the abbreviations and their meanings:

[0074] STFT Short-Time Fourier Transform;

[0075] ISTFT (Inverse Short-Time Fourier Transform) is the inverse operation of STFT (Short-Time Fourier Transform) and is used to reconstruct a frequency domain signal back to the time domain. Its core principle is to recover the original signal through inverse Fourier transform; however, due to window overlap and phase difference, the reconstructed signal may differ from the original signal.

[0076] LSTM (Long Short-Term Memory) is a special type of recurrent neural network (RNN) designed to address the vanishing and exploding gradient problems faced by traditional RNNs when processing long sequences of data. By introducing a "gating mechanism" and "memory units," it can effectively capture and utilize long-term dependencies in sequences, and is widely used in natural language processing, time series prediction, speech recognition, and other fields.

[0077] BLSTM (Bidirectional Long Short-Term Memory Network) is a model that processes time-series data by combining forward LSTM and backward LSTM. It can capture both forward and backward information of the sequence and is often used in natural language processing, video retrieval and other fields.

[0078] GTCRN (Grouped Temporal Convolutional Recurrent Network) is a lightweight speech denoising network. This design improves upon it to obtain TFGTCRN (Time-Frequency Grouped Temporal Convolutional Recurrent Network).

[0079] The TFGTCRN network is an improvement on the lightweight speech denoising network GTCRN (Grouped Temporal Convolutional Recurrent Network). It replaces the grouped temporal convolution in the GTCRN encoder with decoupled time-connected attention (TCA) and decoupled frequency-connected attention (FCA) on the time axis, and replaces the grouped temporal convolution in the decoder with decoupled time-frequency-connected attention (TFCA). This time-frequency axis attention mechanism can utilize long-range narrowband and cross-frequency speaker information without relying on recursive units.

[0080] Recurrent Neural Networks (RNNs) are a type of neural network specifically designed for processing sequential data (such as text, time series, and speech). Their core feature is the use of a "recurrent structure" to preserve historical information, allowing the model to understand current data using the context of previous inputs, thus solving the problem of traditional neural networks being unable to handle "time dependencies."

[0081] GRNN (Grouped Recurrent Neural Network) is a variant of Recurrent Neural Network (RNN) that introduces a "grouping mechanism". Its core idea is to divide the hidden state or input features of the network into several independent subgroups and perform iterative computation on each subgroup to balance model performance and computational efficiency.

[0082] DPRNN (Dual-Path Recurrent Neural Network) is a neural network structure specifically designed for long sequence data (such as speech and audio). Its core feature is that it captures the long-range dependencies of data in the time and frequency dimensions simultaneously through "dual-path recurrent", and it performs well in tasks such as speech separation.

[0083] G-DPRNN (Group-Dual Path Recurrent Neural Network) is a combination of Group Recurrent Neural Network (GRNN) and Dual-Path Recurrent Neural Network (DPRNN).

[0084] Fusion (feature fusion module) is the core pre-component of the "speaker extraction module". Its core function is to accurately fuse the target speaker features (extracted from the speaker encoder in step S3) with the encoded mixed speech features (output from the speech encoder in step S2), so that the mixed speech features carry the "target speaker personalized identifier", providing targeted guidance for the subsequent "inter-frame temporal enhancement" and "intra-frame temporal enhancement" of the TF-InterIntra module.

[0085] The TF-InterIntra module is the core component of the "Speaker Extraction Module." Its core function is to combine the target speaker's features to perform inter-frame temporal dependency modeling and intra-frame temporal detail optimization on the encoded mixed speech time-frequency features, ultimately achieving a personalized enhancement effect of "eliminating non-target speaker interference and strengthening target speech features." This module receives the output of the "Feature Fusion Module" (mixed speech coding features that have been fused with speaker features), providing high-quality enhanced features for subsequent decoding (step S5), and is a crucial link connecting "feature fusion" and "decoding."

[0086] Specifically, the following steps are included:

[0087] S1. Extract features from the mixed speech signal and the reference speech signal respectively, and convert the time-domain waveform into a time-frequency domain signal;

[0088] In step S1, feature extraction and transformation are performed on the mixed speech signal and the reference speech signal respectively using short-time Fourier transform, and the formula is as follows:

[0089]

[0090] Where Y(n,k) represents the time-frequency features of the mixed speech and the reference speech, n represents the time frame, and k represents the frequency point; y(m) represents the input discrete time-domain speech signal; w(nm) represents the sliding window function—the Hanning window; and L represents the frame length parameter.

[0091] Specifically, STFT feature extraction is performed on the mixed speech signal and the reference speech signal respectively, and the time-domain signal of the two-dimensional feature is converted into the time-domain signal of the four-dimensional feature. The STFT parameters include Hanning window, window length = 512, and frame shift = 256.

[0092] S2. Encode the extracted and converted mixed speech time-frequency features and reference speech time-frequency features, and output the encoded mixed speech time-frequency features and reference speech time-frequency features.

[0093] In step S2, the extracted and converted mixed speech time-frequency features and reference speech time-frequency features are input into the speech encoder for high-bit embedding encoding. The speech encoder includes a single convolutional layer and a group normalization layer. The speech encoder uses a sequence of convolutional layers, including convolution operations and group normalization operations.

[0094] The parameters of the convolutional layer include the number of input channels, the number of output channels, the kernel size, the stride, and the padding; the number of input channels = 2 (corresponding to the real and imaginary parts), the number of output channels = 128 (preset embedding dimension), the kernel size = 1, the stride = 1, and the padding = 1, ensuring that the dimension of the output is consistent with that of the input, and the number of output channels is the preset embedding dimension;

[0095] Determine the number of groups for group normalization. The number of groups for group normalization is 8. Normalize the convolution output features using the following formula:

[0096]

[0097] Where, μ g and Let γ and β be the mean and variance of each feature group, γ and β be the learnable parameters, and ε = 1e-5;

[0098] The output is the encoded mixed speech feature X. mix ∈R B×F×T×128 Reference speech features

[0099] X aux ∈R B×F×T×128 .

[0100] S3. The encoded reference speech time-frequency features X aux Input the speaker encoder, extract speaker features, and output the speaker prediction result;

[0101] In S3, the time-frequency features of the input reference speech are processed by the TFGTCRN network. After mean pooling, the speaker features are output through a linear layer, and then the speaker prediction result is output through another linear layer. The speaker encoder includes the TFGTCRN network and the linear layer sequence.

[0102] The TFGTCRN network is an improvement on the GTCRN network, including upsampling and downsampling layers;

[0103] The upsampling layer consists of alternating time-axis decoupled attention (TCA) and frequency-axis decoupled attention (FCA) layers, while the downsampling layer consists of multiple layers of time-frequency axis decoupled attention (TFCA) layers connected together.

[0104] The temporal decoupled attention (TCA), frequency decoupled attention (FCA), and multi-layer temporal-frequency decoupled attention (TFCA) are composed of similar structures, including average pooling layers, one-dimensional convolutional layers, and batch normalization layers; and Sigmoid activation functions are combined with deep convolutional layers according to different modes "time", "freq", and "tf".

[0105] The formula for calculating the time-axis decoupled attention TCA is:

[0106]

[0107] Where T represents the time axis length of the input features after pooling, and O t (t,f) is the attention value of the pooled input feature in the time-axis decoupled attention, W t These are learnable weights;

[0108] The formula for calculating the frequency axis decoupled attention (FCA) is as follows:

[0109]

[0110] Where F represents the frequency axis length of the input feature after pooling, and O f (t,f) is the attention value of the pooled input features in the frequency-axis decoupled attention, W f These are learnable weights;

[0111] The formula for calculating multi-layer time-frequency axis decoupled attention (TFCA) is as follows:

[0112]

[0113] Where T represents the time axis length of the input features after pooling, F represents the frequency axis length of the input features after pooling, and W... t and W f These are the learnable weights corresponding to the time dimension and the frequency dimension, respectively, O tf (t,f) is the attention value of the pooled input feature in the time-frequency axis decoupled attention.

[0114] In S3, the speaker encoder consists of a TFGTCRN network and a linear layer, and the specific process includes:

[0115] The sampling layer on the TFGTCRN network consists of two layers of time-axis decoupled attention (TCA) and two layers of frequency-axis decoupled attention (FCA), which are alternately concatenated to capture the time-varying characteristics of harmonic structures and noise in speech.

[0116] The bottleneck layer in the middle of the TFGTCRN network adopts the Group-DualPath Recurrent Neural Network (G-DPRNN), which integrates the Group Recurrent Neural Network (GRNN) and the Dual-Path Recurrent Neural Network (DPRNN). The GRNN divides the input and hidden state into two groups, each using a recurrent layer with half the parameters. The DPRNN models the spectral pattern and temporal dependency through intra-frame bidirectional and inter-frame unidirectional RNNs, respectively.

[0117] The downsampling layer consists of three layers of time-frequency decoupled attention TFCA concatenated, using the "tf" mode (a combination of one-dimensional depthwise convolutions with time first and frequency second), and the parameters are the same as those of the upsampling layer.

[0118] The TFGTCRN output features are averaged (along the time and frequency axes), then passed through a linear layer to output speaker features of dimension 64, and then passed through another linear layer to output speaker prediction results.

[0119] S4. Input the speaker features and mixed speech features into the speaker extraction module, which includes multiple feature fusion modules and inter-frame and intra-frame layers. Based on the speaker features, perform inter-frame and intra-frame feature enhancement on the encoded mixed speech time-frequency features to eliminate the speech of non-target speakers.

[0120] In S4, the feature fusion module expands the speaker features into (B,1,C) dimensions, where B is the batch size and C is the number of feature dimension channels.

[0121] The encoded speech features are flattened into (B, T*F, C) dimensions, where T is the number of time frames and F is the number of frequency points;

[0122] The speaker features are adapted by calculating the attention between speaker features and flattened speech features through a multi-head attention mechanism.

[0123] The adapted speaker features are added to the original speaker features and expanded to the same dimension as the encoded speech features;

[0124] The fused features are mapped back to the preset embedding dimension through convolutional layers.

[0125] The inter-frame and intra-frame layers consist of inter-frame blocks and intra-frame blocks respectively:

[0126] The fused features are input into inter-frame blocks, passing through a normalization layer, a one-dimensional convolutional layer, and a PReLU activation layer. A sliding window then captures local temporal features along the time dimension. These extracted local temporal features are then input into a bidirectional LSTM network. Finally, a transposed one-dimensional convolution is used to change the feature dimension, as shown in the formula:

[0127] Inter(x)=x+PReLU(Conv1D(Norm(x)))+ConvTranspose1D(BLSTM(Unfold(x)));

[0128] Wherein, Norm represents layer normalization, Conv1D represents one-dimensional convolution, PReLU is the activation function, Unfold represents the feature unfolding operation, BLSTM represents a bidirectional recurrent neural network, and ConvTranspose1D is a transposed one-dimensional convolutional layer used to perform inverse operations on the features after one-dimensional convolution.

[0129] Specifically, the parameters of the one-dimensional convolutional layer are: number of input channels = number of output channels, kernel size = 5, stride = 1, number of groups = 8. The "same" padding mode (the same mode pads shorter sequences with zeros to make the two sequences the same length) ensures that the length of the output sequence after the convolution operation is consistent with the length of the input sequence. Then, after passing through a normalization layer, features are extracted through a sliding operation. A convolution kernel of size (4,1) and a stride of (2,1) are used. The features are then fed into a bidirectional LSTM network, where the LSTM input dimension is 256, the hidden layer parameters are set to 256, and the output dimension is 512. Finally, the output dimension of the LSTM is converted from 512 to 64 through a transposed one-dimensional convolution. The transposed convolution kernel size is 4 and the stride is 2.

[0130] Features output from inter-frame blocks are directly passed to intra-frame blocks and processed through a multi-head self-attention mechanism. After feature normalization, the multi-head attention mechanism calculates temporal correlation features, which are then dropout-free and residually connected to the current features. This mechanism randomly discards a portion of neurons during training, enabling the model to learn more robust features and thus improving generalization ability. The calculation formula is as follows:

[0131]

[0132] in, Let W be the query, key, and value projection matrix of the i-th head. O To output the projection matrix, d k For a single head, SoftMax represents the attention weights, and the head represents the feature dimension. iThis indicates single-head attention. Concat concatenates the weights of multiple single-head attention mechanisms. MultiHead(Q,K,V) represents the output features of the multi-head attention mechanism.

[0133] The features are then processed by an LS convolutional module consisting of a normalization layer, a one-dimensional convolutional layer, a SiLU activation layer, and an LS convolutional layer. Finally, the output features of the intra-frame block and the output features of the inter-frame block are residually concatenated to ensure that the model fully utilizes the feature information from both shallow and deep layers. The intra-frame block formula is as follows:

[0134] Intra(x)=x+Dropout(MHSA(LN(x)))+

[0135] LSConv(SiLU(Conv1D(LN[x])));

[0136] Here, LN performs layer normalization on features, MHSA is a multi-head attention layer, Dropout performs regularization on the output to prevent overfitting, Conv1D represents one-dimensional convolution, SiLU is an activation function, and LSConv is a convolution module, as detailed below.

[0137] The LS convolutional module in an intra-block includes a large kernel sensing module and a small kernel aggregation module;

[0138] The large kernel perception module is used to capture a wide range of spatial context information of the input features and generate aggregate weights. The large kernel perception module first reduces the number of input feature channels to a preset dimension through point convolution, and then passes the kernel with a kernel size of (K). L ,K L Large kernel depthwise convolutions capture K features centered on each feature. L ×K L Neighborhood information is used to generate context-adaptive weights w through point convolution. i ;

[0139] The small-kernel aggregation module performs small-range adaptive aggregation of features based on aggregation weights. It divides the input feature channels into G groups and combines the weights w generated by the large-kernel perception module. i Remodeled into G×K S ×K S small core weight And by using the small kernel weights to assign K to each feature S ×K S Neighborhood features are convolved and aggregated to obtain output features.

[0140] Specifically, a time-frequency domain-based TF-InterIntra module is constructed, which integrates speaker features E with mixed speech coding features X. mixAfter fusion, the input to the TF-InterIntra module mainly involves the following steps:

[0141] First, the input features are passed to the inter-frame blocks. To avoid gradient explosion caused by long-term speech signals, the features are passed through a network layer consisting of group normalization, one-dimensional convolution, and PReLU activation function. Next, the features are unfolded, using a sliding window with kernel size (4,1) and stride (2,1) to capture local temporal features in the time dimension while maintaining the integrity of features in the frequency dimension to avoid cross-frequency interference. Then, the extracted features are input to a Bi-directional Long Short-Term Memory (BLSTM) network layer. By using BLSTM to model historical and future information, the ability to perceive speech context is enhanced, the target speaker features are better identified, and non-target interference is suppressed. Finally, a transposed one-dimensional convolution is used to change the feature dimension, as shown in the following formula:

[0142] Inter(x)=x+PReLU(Conv1D(Norm(x)))+ConvTranspose1D(BLSTM(Unfold(x)));

[0143] The features output from the inter-frame block are directly passed to the intra-frame block. The features are first refined and enhanced with personalized information by a multi-head self-attention mechanism for the time-frequency features within a single frame:

[0144]

[0145] The output features are then dropped and fed into the LS convolutional module. A dual reconstruction mechanism combining a large kernel for global perception and a small kernel for local aggregation effectively improves the discriminative power of intra-frame features. The calculation process is as follows:

[0146]

[0147] Where σ represents the Sigmoid function, S g This represents the global context features of the g-th group from the big kernel perception module. Finally, the calculation formula for intra-blocks is given:

[0148] Intra(x)=x+Dropout(MHSA(LN(x)))+LSConv(SiLU(Conv1D(LN[x])));

[0149] S5. Decode the obtained enhanced features and output the enhanced reference speech time-frequency features;

[0150] In step S5, the enhanced features are decoded by a speech decoder module. The speech decoder module uses a transposed convolutional layer, with the number of input channels being a preset embedding dimension, and the number of output channels, kernel size, stride, and padding being consistent with the kernel size in the speech encoder.

[0151] Specifically, the speech decoder module consists of a single-layer transposed two-dimensional convolutional layer, which transforms the enhanced time-frequency features from (B,C,T,F) to (B,2,T,F) so that the real and imaginary parts can be fused and the inverse short-time Fourier transform (ISTFT) can be performed to obtain the enhanced speech of the target speaker.

[0152] Specifically, the features are decoded, and the enhanced features X are... enh The input is fed into a speech decoder consisting of a single layer of transposed 2D convolutions, which restores the features to the time-frequency features Y. enh ∈R B×2×F×T (B represents the batch size, 2 represents the real and imaginary parts of the complex number, F represents the number of frequency points, and T represents the number of time frames).

[0153] S6. The enhanced speech time-frequency features are transformed by inverse short-time Fourier transform, and the time-frequency domain signal is converted into a time-domain waveform to obtain the final model output.

[0154] Specifically, the inverse short-time Fourier transform (ISTFT) is finally used to transform the time-frequency features Y. enh Reconstruct the time domain signal.

[0155] A personalized speech enhancement system based on inter-frame and intra-frame feature fusion is provided, which applies the personalized speech enhancement method based on inter-frame and intra-frame feature fusion as described above.

[0156] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the present invention should also fall within the protection scope of the claims of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.

Claims

1. A personalized speech enhancement method based on inter-frame and intra-frame feature fusion, characterized in that, The method comprises the following steps: S1, feature extraction is performed on the mixed voice signal and the reference voice signal respectively, and time domain waveform is converted into time-frequency domain signal; S2, the extracted and converted mixed voice time-frequency feature and the reference voice time-frequency feature are encoded, and the encoded mixed voice time-frequency feature and the reference voice time-frequency feature are output; S3, the encoded reference voice time-frequency feature is input into a speaker encoder, speaker features are extracted, and a speaker prediction result is output; S4, the speaker features and the mixed voice features are input into a speaker extraction module, which comprises a plurality of feature fusion modules and inter-frame and intra-frame layers, the encoded mixed voice time-frequency features are enhanced in inter-frame and intra-frame according to the speaker features, and the voice of the non-target speaker is eliminated; S5, the obtained enhanced features are decoded, and the enhanced reference voice time-frequency features are output; S6, the enhanced voice time-frequency features are transformed through inverse short-time Fourier transform, the time-frequency domain signal is converted into time domain waveform, and the final model output is obtained.

2. The personalized speech enhancement method based on fusion of inter-frame and intra-frame features according to claim 1, characterized in that, In the S1, the mixed voice signal and the reference voice signal are respectively extracted and converted through short-time Fourier transform, and the formula is: Wherein, Y(n, k) represents the time-frequency features of the mixed voice and the reference voice, n represents the time frame, and k represents the frequency point; y(m) represents the input discrete time domain voice signal; w(n-m) represents a sliding window function, that is, a Hanning window; and L represents a frame length parameter.

3. The personalized speech enhancement method based on fusion of inter-frame and intra-frame features according to claim 1, characterized in that, In the S2, the extracted and converted mixed voice time-frequency features and the reference voice time-frequency features are input into a voice encoder for high-bit embedding coding; the voice encoder comprises a single convolutional layer and a group normalization layer, the voice encoder adopts a convolutional layer sequence, which comprises convolutional operation and group normalization operation, The convolutional layer parameters comprise input channel number, output channel number, convolution kernel size, step and padding; the output dimension is consistent with the input, and the output channel number is a preset embedding dimension; The group normalization group number is determined, and the convolutional output features are normalized, and the formula is: where μ g and are the mean and variance of each set of features, γ and β are learnable parameters, and ε = 1e-5; Output the encoded mixed speech feature X mix ∈R B×F×T×128 , the reference speech feature X aux ∈R B×F×T×128 .

4. The personalized speech enhancement method based on fusion of inter-frame and intra-frame features according to claim 3, characterized in that, In the S3, the input reference voice time-frequency features are processed by using the TFGTCRN, the speaker features are output through a linear layer after mean pooling, and the speaker prediction result is output through another linear layer; the speaker encoder comprises a TFGTCRN network and a linear layer sequence; The TFGTCRN network comprises an up-sampling layer and a down-sampling layer; The up-sampling layer is composed of time axis decoupling attention TCA and frequency axis decoupling attention FCA layers which are alternately connected in series, and the down-sampling layer is composed of a plurality of time-frequency axis decoupling attention TFCA layers which are connected in series; The time axis decoupling attention TCA, the frequency axis decoupling attention FCA and the plurality of time-frequency axis decoupling attention TFCA layers are composed of similar structures, comprising an average pooling layer, a one-dimensional convolutional layer and a batch normalization layer; a Sigmoid activation function is combined according to different modes of deep convolutional layers; The calculation formula of the time axis decoupling attention TCA is: wherein T represents the length of the time axis of the pooled input feature, O t (t,f) is the attention value of the pooled input feature in the time axis decoupled attention, W t is a learnable weight; The calculation formula of the frequency axis decoupling attention FCA is: wherein F denotes the length of the frequency axis of the pooled input feature, O f (t,f) is the attention value of the pooled input feature in the frequency axis decoupled attention, W f is a learnable weight; The calculation formula of the plurality of time-frequency axis decoupling attention TFCA layers is: wherein T represents the length of the time axis of the pooled input feature, F represents the length of the frequency axis of the pooled input feature, W t and W f are respectively the learnable weights of the time dimension and the frequency dimension, O tf (t,f) is the attention value of the pooled input feature in the time-frequency axis decoupled attention.

5. The personalized speech enhancement method based on fusion of inter-frame and intra-frame features according to claim 4, characterized in that, In the S3, the speaker encoder is composed of the TFGTCRN network and the linear layer, and the specific process comprises: The TFGTCRN network upsampling layer is composed of two layers of time axis decoupling attention TCA and two layers of frequency axis decoupling attention FCA, which capture the harmonic structure in the speech and the time-varying characteristics of the noise by alternating series connection; The TFGTCRN network adopts a group double-path recurrent neural network G-DPRNN, which integrates a group recurrent neural network GRNN and a double-path recurrent neural network DPRNN; the GRNN divides the input and hidden state into two groups, each of which uses a recurrent layer with halved parameters, and the DPRNN models the spectral pattern and temporal dependence by using intra-frame bidirectional and inter-frame unidirectional RNNs, respectively; The downsampling layer is composed of multiple layers of time-frequency decoupling attention TFCA in series, and the parameters are consistent with those of the upsampling layer; After the TFGTCRN output features are averaged and pooled, they are output through a linear layer to obtain speaker features, and then through another linear layer to output speaker prediction results.

6. The personalized speech enhancement method based on fusion of inter-frame and intra-frame features according to claim 5, characterized in that, In the S4, the feature fusion module expands the speaker features to a (B, 1, C) dimension, where B is the batch size and C is the number of feature channels; The encoded speech features are flattened to a (B, T*F, C) dimension, where T is the number of time frames and F is the number of frequency points; The speaker features are calculated through a multi-head attention mechanism, and the adapted speaker features are obtained; The adapted speaker features are added to the original speaker features and expanded to the same dimension as the encoded speech features; The fused features are mapped back to the preset embedding dimension through a convolutional layer.

7. The personalized speech enhancement method based on fusion of inter-frame and intra-frame features according to claim 6, characterized in that, The inter-frame and intra-frame layers are composed of inter-frame blocks and intra-frame blocks, respectively: The mixed speech time-frequency features input into the inter-frame block through the feature fusion module, and the features are normalized, one-dimensional convolution, and PReLU activation, then captured through a sliding window to capture the local temporal features in the time dimension, then input into the bidirectional LSTM network, and finally use the transpose one-dimensional convolution to change the feature dimension, the formula is: Inter(x) = x + PReLU(Conv1D(Norm(x))) + ConvTranspose1D(BLSTM(Unfold(x))); Where Norm represents layer normalization, Conv1D represents one-dimensional convolution, PReLU is an activation function; Unfold represents an unfolding operation on the features, BLSTM represents a bidirectional recurrent neural network, and ConvTranspose1D is a transpose one-dimensional convolution layer, which is used to perform inverse operations on the one-dimensional convolution features; The features output by the inter-frame block are directly passed to the intra-frame block and processed through a multi-head self-attention mechanism. After normalization, the correlation features in the time dimension are calculated through a multi-head attention mechanism, and after dropout, the current features are connected in residual connection. It randomly discards a part of the neurons during training to make the model learn more robust features, thereby improving the generalization ability, and the calculation formula is: ​ wherein, are the query, key, value projection matrices for the i-th head, respectively, W O is the output projection matrix, d k is the feature dimension for a single head, SoftMax denotes the attention weights, head i denotes single-head attention, Concat concatenates multiple single-head attention weights, and MultiHead(Q, K, V) denotes the output features of the multi-head attention mechanism. Then the features are input into the LS convolution module composed of a normalization layer, a one-dimensional convolution layer, a SiLU activation layer and an LS convolution layer for processing. Finally, the output features of the intra-frame block and the output features of the inter-frame block are connected in residual to ensure that the model fully utilizes the feature information of the shallow and deep layers. The formula of the intra-frame block is: Intra(x)=x+Dropout(MHSA(LN(x)))+LSConv(SiLU(Conv1D(LN[x]))); where LN is layer normalization of the features, MHSA is a multi-head attention layer, Dropout is regularization of the output to prevent model overfitting; Conv1D represents one-dimensional convolution, SiLU is an activation function, and LSConv is a convolution module. The LS convolution module in the intra-frame block includes a large kernel perception module and a small kernel aggregation module.

8. The personalized speech enhancement method based on fusion of inter-frame and intra-frame features according to claim 7, characterized in that, In S5, the enhanced features are decoded by a speech decoder module. The speech decoder module adopts a transpose convolution layer, the input channel number is a preset embedding dimension, and the output channel number, the convolution kernel size, the step and the padding are consistent with the convolution kernel size in the speech encoder. The large kernel perception module is used for capturing large-range spatial context information of input features and generating aggregated weights, the large kernel perception module sequentially reduces the input feature channel number to a preset dimension through point convolution, then captures K L ×K L neighborhood information centered on each feature through large kernel deep convolution with a kernel size of (K L ×K L ), and finally generates context adaptive weights w i ; The small kernel aggregation module performs small-range self-adaptive aggregation on the features based on aggregation weights, the small kernel aggregation module divides the input feature channel into G groups, and the weights w generated by the large kernel perception module are divided into G groups i Reshape to GxK S ×K S Small kernel weights And the K S ×K S Neighborhood features corresponding to each feature are convolved and aggregated by the small kernel weights to obtain output features.

9. The personalized speech enhancement method based on fusion of inter-frame and intra-frame features according to claim 8, characterized in that, The application of a personalized speech enhancement method based on inter-frame and intra-frame feature fusion as claimed in any one of claims 1 to 9.

10. A personalized speech enhancement system based on fusion of inter-frame and intra-frame features, characterized in that, ​

Images