Bandwidth expansion method for narrowband voice signal

Abstract

The invention discloses a bandwidth expansion method for a narrowband voice signal, which belongs to the technical field of voice processing, and comprises the following steps: acquiring the narrowband voice signal; constructing a voice bandwidth expansion model based on boundary perception and fundamental frequency feature fusion; training the voice bandwidth expansion model to obtain a trained voice bandwidth expansion model; and inputting the narrowband voice signal into the trained voice bandwidth expansion model to realize the broadband expansion of the narrowband voice signal. According to the method, adjustment factors generated by a TFiLM module are introduced into a codec architecture, the factors act on different blocks of input features, the amplitude of original features is adjusted, changes of time sequence data are dynamically adapted in the reconstruction process, and the perception ability of a model to specific time domain features is enhanced. A time domain gating module in the decoder stage generates a gating mask in the time dimension, dynamically and selectively retains or shields certain time steps of the input features, thereby effectively extracting high frequency information related to low frequencies.

Description

Technical Field

[0001] This invention belongs to the field of signal processing technology and relates to a method for bandwidth expansion of narrowband speech signals. Background Technology

[0002] Early bandwidth extension methods were largely based on traditional statistical models, such as Gaussian Mixture Models (GMMs) and Hidden Markov Models (HMMs). The core idea of ​​these methods is to recover the missing high-frequency components by estimating the spectral envelope of the speech signal and establishing a narrowband-to-wideband mapping. However, these methods often suffer from "oversmoothing," failing to accurately recover the details of the high-frequency components, resulting in a blurry and distorted sound in the reconstructed speech, especially in the recovery of high-frequency content. With advancements in signal processing technology, some methods employ interpolation or windowing techniques to estimate and reconstruct the high-frequency components of narrowband signals. These methods typically recover high frequencies by performing spectral interpolation and phase recovery on the low-frequency components. However, these methods have drawbacks: the recovered high-frequency components may introduce artifacts, and they struggle to effectively handle the detailed information in dynamic speech signals. These methods are relatively simple in processing speech signals and cannot guarantee the naturalness and clarity of the high-frequency components, especially in complex speech environments. In recent years, bandwidth extension (BWE) methods based on neural networks have made significant progress, especially with the application of autoencoders and generative adversarial networks (GANs), which have significantly improved the recovery of high-frequency components. However, these deep learning methods still have certain limitations. On the one hand, network models typically require a large amount of training data and computational resources, making the training process time-consuming. On the other hand, although these methods can recover most high-frequency content, artifacts or distortions may still occur in certain situations, especially in low-quality input signals and noisy environments, particularly in the recovery of high-frequency details. Therefore, despite the good recovery results achieved by deep learning methods, challenges remain regarding detail fidelity and robustness.

[0003] In analog or early digital transmission systems, high-frequency signals are more susceptible to noise, attenuation, and interference (such as line noise or electromagnetic interference), leading to a significant increase in transmission costs. Therefore, system design often prioritizes sacrificing high-frequency components to ensure overall reliability and stability. Even today, with broadband technology available, mobile phone systems automatically fall back to narrowband mode in environments with weak signals, such as high-speed rail, subways, or remote areas, resulting in degraded voice quality. Furthermore, in video calls (such as WeChat), if one party's network is unstable, the voice signal is compressed into narrowband, resulting in the loss of high-frequency information and a muffled, unclear, and unnatural sound. In everyday telephone calls, users may switch from the traditional PSTN (Public Switched Telephone Network) to VoIP or encounter narrowband compatibility modes during international long-distance calls. In these situations, while addressing the issues of high-frequency information loss and spectrum discontinuity, low latency must also be met—algorithms with excessive computational demands or overly complex models are prone to stuttering on low-end devices or when the network fluctuates. These problems directly impact work and study efficiency; therefore, methods for extending the bandwidth of narrowband voice signals must be optimized to address these issues and achieve efficient and reliable voice quality improvement. Summary of the Invention

[0004] To solve the above problems, the technical solution adopted by the present invention is: a method for bandwidth extension of narrowband voice signals, comprising the following steps: Acquire narrowband speech signals; Construct a speech bandwidth extension model based on boundary awareness and fundamental frequency feature fusion; The speech bandwidth expansion model is trained to obtain a trained speech bandwidth expansion model; By inputting the narrowband speech signal into a trained speech bandwidth extension model, the bandwidth extension of the narrowband speech signal can be achieved.

[0005] Furthermore: the speech bandwidth extension model based on boundary awareness and fundamental frequency feature fusion includes: First encoder: used to mark and compress the temporal features of narrowband speech signals; The first TFiLM module is used to divide the labeled and compressed temporal features output by the first encoder into multiple blocks through pooling downsampling. Each block will be processed by an LSTM layer to calculate the adjustment factor and reshape the temporal features back to the original dimension. The second encoder is used to calculate the activation features and confidence level of the narrowband speech signal in the frequency domain and output the features. Multi-head cross-attention mechanism module: It is used to calculate the attention weight based on the similarity between the query and the key for the time-domain features and frequency-domain features output by the first TFiLM module, and to perform weighted summation of the V values ​​to obtain a weighted feature representation that integrates information from the fundamental frequency features and time-domain features; The association module is used to add the features output by the first TFiLM module and the fused features output by the multi-head cross-attention mechanism module to obtain feature information containing both. The merged features are activated by Sigmoid to generate an interaction mask. The important parts of the attention features are extracted from the mask and added to the basic features to obtain the preliminary enhanced features. Boundary awareness module: used to assess the features based on the initial enhanced features output by the association module, the temporal features reconstructed back to the original dimension output by the first TFiLM module, and the labeled and compressed features output by the first encoder. Capture the spatiotemporal correlation between local content and the overall scene, extract the weights of branches to integrate information, and perform weighted fusion of features containing fundamental frequency information and texture features; Decoder module: Used to extract key features related to low and high frequencies from the fused fundamental frequency information output by the boundary perception module, and then splice it with the original audio signal and adjust the dimensions to output the final bandwidth-extended audio.

[0006] Further: The first encoder includes: The first encoder layer is used to compress the input narrowband speech signal once; The second TFiLM module is used to perform key feature annotation on the compressed narrowband speech signal output by the first encoder layer; Second encoder layer: used to perform secondary compression on the key features of the annotations output by the second TFiLM module; Bottleneck layer: Used to compress the features output after the second encoder layer has been compressed twice and then output them again.

[0007] Furthermore: the boundary awareness module includes a difference extraction branch: used to capture the spatiotemporal correlation between local content and the overall scene; Fusion branch: Used to integrate information by utilizing the weights of the difference extraction branch to analyze the spatiotemporal correlation features between the local content output by the difference extraction branch and the overall scene. Texture aggregation module: Based on the fused difference features output by the fusion branch and the enhanced features output by the association module, channel compression is performed, and a joint attention mechanism is used to perform weighted fusion of features containing fundamental frequency information and texture features.

[0008] Further: the decoder includes: First decoder layer: used to restore the output features of the texture fusion module and the features concatenated by the first encoder module; The third TFiLM module is used to annotate the important features restored by the first decoder layer; The second decoder layer is used to restore the labeled features output by the third TFiLM module; then it is concatenated with the original audio signal and the final bandwidth-extended audio is output through one-dimensional convolution to adjust the dimension.

[0009] Furthermore, the first decoder layer and the second decoder layer have the same structure. The first decoder layer includes a time-domain gating module, which is used to generate a gating mask according to the time dimension, dynamically and selectively retain or mask channel features, and filter out key features related to low frequency and high frequency.

[0010] Furthermore, the expression for the adjustment factor is as follows:

[0011] in: Generate adjustment factors for each block of the LSTM in the TFiLM module. The TFiLM module in the Wave encoder is the post-processing module for the first three encoder layers, and its output features are the features of the third convolutional layer of the encoder. .

[0012] Furthermore: the loss function of the speech bandwidth expansion model is constructed by adding the audio loss function and the multi-resolution loss function and then subtracting the consistency loss function; The multi-resolution loss function is the sum of the loss functions with an FFT window length of 512, an FFT window length of 1024, and an FFT window length of 2048.

[0013] This invention provides a bandwidth extension method for narrowband speech signals. This method models the high-frequency components of narrowband speech by fully considering the influence of the signal's acoustic and temporal features. The method introduces adjustment factors generated by the TFiLM module into the codec architecture. These factors act on different blocks of input features, adjusting the amplitude of the original features and dynamically adapting to changes in temporal data during reconstruction, thus enhancing the model's ability to perceive specific temporal features. The introduced fundamental frequency information is effectively utilized through a multi-head cross-attention mechanism, leveraging the natural pitch and rhythmic features of the speech contained within the fundamental frequency information. This strengthens the interaction and information fusion between features, highlighting dynamic features such as fundamental frequency and energy abrupt changes while suppressing unimportant redundant components, thereby improving the accuracy of feature representation and the ability to recover details.

[0014] Furthermore, the boundary awareness module designed in this invention retains positive and negative semantic information while generating compact difference representations through linear remixing across channels. This preserves the difference features and integrates shared features, effectively avoiding the ambiguity caused by oversmoothing and providing subsequent modules with accurate ability to capture differences before and after enhancement.

[0015] In the decoder stage, the temporal gating module generates a gating mask in the time dimension, dynamically and selectively preserving or masking certain time steps of the input features, thereby effectively extracting high-frequency information related to low frequencies. The texture feature aggregation module fuses the output texture features with the original features. This fusion method enhances the diversity of features, making the reconstructed speech signal richer and more detailed.

[0016] Finally, the multi-level loss functions designed in this invention include boundary consistency loss, multi-resolution spectral loss, and time-domain audio loss. These loss functions work together to guide the network in effective parameter optimization, thereby obtaining better network parameters and enhancement effects. Experimental results show that, compared with other methods, this invention significantly outperforms other methods in speech signal enhancement, fully demonstrating the effectiveness of the proposed method. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the method; Figure 2 This is a model structure diagram of the method; Figure 3 The structure of the modules included in the model is shown in Figure 1. (a) is the structure of the decoder layer, (b) is the structure of the decoder layer, (c) is the texture aggregation module, (d) is the association module, and (e) is the temporal gating module. Figure 4 This is the audio time-domain waveform before bandwidth expansion; Figure 5 This is the audio spectrum before bandwidth expansion; Figure 6 This is the audio time-domain waveform after bandwidth expansion; Figure 7 This is the audio spectrum diagram after bandwidth expansion. Detailed Implementation

[0019] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Figure 1 This is a flowchart of the method; A method for bandwidth extension of narrowband speech signals includes the following steps: S1: Acquire narrowband voice signal; S2: Construct a speech bandwidth extension model based on boundary awareness and fundamental frequency feature fusion; S3: Train the speech bandwidth expansion model to obtain a trained speech bandwidth expansion model; S4: Input the narrowband speech signal into the trained speech bandwidth extension model to achieve broadband extension of the narrowband speech signal.

[0022] Steps S1 / S2 / S3 / S4 are executed sequentially; Figure 2 This is a model structure diagram of the method; Figure 3 The structure of the modules included in the model is shown in Figure 1. (a) is the structure of the decoder layer, (b) is the structure of the decoder layer, (c) is the texture aggregation module, (d) is the association module, and (e) is the temporal gating module. The speech bandwidth extension model based on boundary awareness and fundamental frequency feature fusion includes: First encoder: used to mark and compress the temporal features of narrowband speech signals; The first TFiLM module is used to divide the labeled and compressed temporal features output by the first encoder into multiple blocks through pooling downsampling. Each block will be processed by an LSTM layer to calculate the adjustment factor and reshape the temporal features back to the original dimension. The second encoder is used to calculate the activation features and confidence level of the narrowband speech signal in the frequency domain and output the features. Multi-head cross-attention mechanism module: It is used to calculate the attention weight based on the similarity between the query and the key for the time-domain features and frequency-domain features output by the first TFiLM module, and to perform weighted summation of the V values ​​to obtain a weighted feature representation that integrates information from the fundamental frequency features and time-domain features; The association module is used to add the features output by the first TFiLM module and the fused features output by the multi-head cross-attention mechanism module to obtain feature information containing both. The merged features are activated by Sigmoid to generate an interaction mask. The important parts of the attention features are extracted from the mask and added to the basic features to obtain the preliminary enhanced features. Boundary awareness module: used to assess the features based on the initial enhanced features output by the association module, the temporal features reconstructed back to the original dimension output by the first TFiLM module, and the labeled and compressed features output by the first encoder. Capture the spatiotemporal correlation between local content and the overall scene, extract the weights of branches to integrate information, and perform weighted fusion of features containing fundamental frequency information and texture features; Decoder module: Used to extract key features related to low and high frequencies from the fused fundamental frequency information output by the boundary perception module, and then splice it with the original audio signal and adjust the dimensions to output the final bandwidth-extended audio.

[0023] The first encoder includes: The first encoder layer is used to compress the input narrowband speech signal once; The second TFiLM module is used to perform key feature annotation on the compressed narrowband speech signal output by the first encoder layer; Second encoder layer: used to perform secondary compression on the key features of the annotations output by the second TFiLM module; Bottleneck layer: Used to compress the features output after the second encoder layer has been compressed twice and then output them again.

[0024] The boundary awareness module includes a difference extraction branch: used to capture the spatiotemporal correlation between local content and the overall scene; The difference extraction branch includes a first difference calculation module and a second difference calculation module: the expression of the second difference calculation module is as follows:

[0025] Where A represents SimAM1d processing, used to extract boundary change intensity, narrowband single-channel speech signal. , express Boundary saliency map, express The boundary saliency map.

[0026] Fusion branch: Used to integrate information by utilizing the weights of the difference extraction branch to analyze the spatiotemporal correlation features between the local content output by the difference extraction branch and the overall scene. Texture aggregation module: Based on the fused difference features output by the fusion branch and the enhanced features output by the association module, channel compression is performed, and a joint attention mechanism is used to perform weighted fusion of features containing fundamental frequency information and texture features.

[0027] The decoder includes: First decoder layer: used to restore the output features of the texture fusion module and the features concatenated by the first encoder module; The third TFiLM module is used to annotate the important features restored by the first decoder layer; The second decoder layer is used to restore the labeled features output by the third TFiLM module; then it is concatenated with the original audio signal and the final bandwidth-extended audio is output through one-dimensional convolution to adjust the dimension.

[0028] The first decoder layer and the second decoder layer have the same structure. The first decoder layer includes a time-domain gating module, which is used to generate a gating mask according to the time dimension, dynamically and selectively retain or mask channel features, and filter out key features related to low frequency and high frequency.

[0029] The second encoder module includes a CREPE module: extracting fundamental frequency features and calculating the activation features and confidence level of the signal.

[0030] The expression for the adjustment factor is as follows:

[0031] in: The intermediate time-domain features of the output of the first three encoder layers, The adjustment factor generated by the TFiLM module in the Wave encoder, intermediate time-domain features Features were obtained after processing with adjustment factors .

[0032] The loss function of the speech bandwidth extension model is constructed by adding the audio loss function and the multi-resolution loss function and subtracting the consistency loss function; The multi-resolution loss function is the sum of the loss functions with an FFT window length of 512, an FFT window length of 1024, and an FFT window length of 2048.

[0033] Example 1: A method for bandwidth extension of narrowband speech signals includes the following steps: S1: Acquire narrowband voice signal; S2: Construct a speech bandwidth extension model based on boundary awareness and fundamental frequency feature fusion; S3: Train the speech bandwidth expansion model to obtain a trained speech bandwidth expansion model; S4: Input the narrowband speech signal into the trained speech bandwidth extension model to achieve broadband extension of the narrowband speech signal.

[0034] Steps S1 / S2 / S3 / S4 are executed sequentially; The speech bandwidth extension model based on boundary awareness and fundamental frequency feature fusion includes: First encoder: used to mark and compress the temporal features of narrowband speech signals; The first TFiLM module is used to divide the labeled and compressed temporal features output by the first encoder into multiple blocks through pooling downsampling. Each block will be processed by an LSTM layer to calculate the adjustment factor and reshape the temporal features back to the original dimension. The second encoder is used to calculate the activation features and confidence level of the narrowband speech signal in the frequency domain and output the features. Multi-head cross-attention mechanism module: It is used to calculate the attention weight based on the similarity between the query and the key for the time-domain features and frequency-domain features output by the first TFiLM module, and to perform weighted summation of the V values ​​to obtain a weighted feature representation that integrates information from the fundamental frequency features and time-domain features; The association module is used to add the features output by the first TFiLM module and the fused features output by the multi-head cross-attention mechanism module to obtain feature information containing both. The merged features are activated by Sigmoid to generate an interaction mask. The important parts of the attention features are extracted from the mask and added to the basic features to obtain the preliminary enhanced features. Boundary awareness module: used to assess the features based on the initial enhanced features output by the association module, the temporal features reconstructed back to the original dimension output by the first TFiLM module, and the labeled and compressed features output by the first encoder. Capture the spatiotemporal correlation between local content and the overall scene, extract the weights of branches to integrate information, and perform weighted fusion of features containing fundamental frequency information and texture features; Decoder module: Used to extract key features related to low and high frequencies from the fused fundamental frequency information output by the boundary perception module, and then splice it with the original audio signal and adjust the dimensions to output the final bandwidth-extended audio.

[0035] The first encoder is a Wave encoder, and the second encoder is an F0 encoder. The Wave encoder extracts temporal features through a multi-layer convolutional layer and a temporal feature linear modulation (TFiLM) module. The TFiLM module receives the temporal features output by the encoder. In the TFiLM module, the pooling downsampling operation divides the features into multiple blocks. Each block is processed by an LSTM layer to calculate the adjustment factor. The temporal features that are reconstructed back to the original dimension are used as keys (K) and values ​​(V).

[0036] CREPE (Convolutional Representation for Pitch Estimation) is a pitch detection model that can directly estimate pitch from the original waveform. Here, CREPE is used to extract fundamental frequency features, such as pitch information, and to calculate the activation features and confidence level of the signal. The fundamental frequency features are used as input to the query (Q), and together with the temporal features, are fed into a multi-head cross-attention mechanism. Attention weights are calculated based on the similarity between the query (Q) and the key (K). The values ​​(V) are then weighted and summed to obtain a weighted feature representation that integrates information from both the fundamental frequency and temporal features.

[0037] In the association module, the basic features extracted in the temporal domain are added to the attention features output by the multi-head attention module to obtain feature information containing both. The merged features are then activated by a sigmoid function to generate an interaction mask. This mask is used to extract the important parts of the attention features, which are then added to and merged with the basic features to obtain preliminary enhanced features. The enhanced features and temporal features are then input into the boundary awareness module (CBM) for texture extraction and fusion.

[0038] The CBM module comprises two branches: a difference extraction branch and a fusion branch. The difference extraction branch captures the spatiotemporal correlation between local content and the overall scene, while the fusion branch integrates information using the weights from the difference extraction branch. The output texture features and the original temporal features undergo channel compression in the texture feature aggregation module, and a joint attention mechanism is used to perform weighted fusion of features containing fundamental frequency information and texture features.

[0039] The fused features are input into a time-domain gating module, which generates a gating mask based on the time dimension. This mask dynamically and selectively retains or masks channel features, filtering out key features related to low and high frequencies. Finally, after processing by a multi-layer decoder, the decoder output is concatenated with the original audio signal and its dimensions are adjusted through one-dimensional convolution to output the final bandwidth-extended audio.

[0040] This invention employs a two-stage training process. The first stage is the pre-training stage, in which only the encoder, decoder, and multi-resolution STFT loss participate in the training.

[0041] The Fast Fourier Transform (FFT) with an n_fft window length of 2048 aligns harmonics and timbre, while the FFT with an n_fft window length of 512 aligns transient frequency rises and falls. During backpropagation, the three resolutions jointly constrain the encoder / decoder and the gradient of the intermediate model, thereby achieving more accurate reconstruction.

[0042] First, the narrowband speech input encoder is processed through convolution, TFiLM module scaling modulation, normalization, and nonlinear activation to extract temporal features. The calculation process is as follows: (1) Among them, narrowband single-channel voice signal Temporal features of encoder output , To reconstruct broadband audio after high-frequency information.

[0043] The third-layer TFiLM module of the encoder and decoder collaborates with the intermediate TFiLM module. First, it performs pooling downsampling on the input temporal features and divides the features into multiple distinct blocks. Each block is then fed into an LSTM layer, where an adjustment factor is generated for each block. This adjustment factor is multiplied by the corresponding temporal feature to obtain the temporal features reconstructed back to their original dimensions. These features ultimately serve as keys (K) and values ​​(V). The fundamental frequency features are used as query (Q) input, participating in the subsequent multi-head cross-attention mechanism, which can be represented as: (2) The features output after the third convolutional layer of the encoder is processed by the TFiLM module are... To generate adjustment factors for each block in the LSTM of TFiLM, These are the features of the output of the third convolutional layer of the encoder.

[0044] The temporal gating module in the decoder is located in each layer of the decoder. It generates a sparse temporal mask by Gumbel-Softmax gating. The mask is then regularized, linearly interpolated, upsampled back to its original length, and multiplied with the input of a single-layer decoder. This process removes irrelevant or redundant temporal components, resulting in more refined temporal features. .

[0045] (3) The temporal features are reconstructed layer by layer by the decoder, and the multi-resolution STFT loss is calculated. loss: (4) here and These are the power spectrum and logarithmic spectrum under different n_fft values, respectively; to avoid the problem of infinity during logarithmic operations. .

[0046] (5) Calculate audio loss using the L1 norm; For the reconstructed broadband audio, For target broadband audio. This is a hyperparameter that controls the weights between waveform loss and spectral loss. In this stage, it is set to 200, and the final training loss for this stage is: (6) After the speech bandwidth extension model has basic speech reconstruction capabilities, it enters the second stage of joint training: inputting narrowband speech time-domain waveforms, and extracting fundamental frequency (F0) features, including activation features and confidence, through the CREPE module.

[0047] (7) Simultaneously, the original waveform is input into the pre-trained encoder. The temporal features output by the encoder are downsampled in the TFiLM module and generated by LSTM to produce adjustment factors for each feature block. The adjustment factors are multiplied with the corresponding features and reconstructed back to the original dimensions. In multi-head cross-attention, these adjustment factors serve as Key (K) and Value (V), enhancing the model's ability to capture speech elements such as pitch and prosody. Meanwhile, the fundamental frequency features extracted by CREPE serve as the input to Query (Q).

[0048] (8) in: , , Let Q be the fundamental frequency feature, K be the temporal feature, and V be the temporal feature. The features are obtained by retrieving the pitch-related context on the time axis. The global adjustment factor is extracted from the output of the Wave encoder using intermediate TFiLM. , The time-domain characteristics are adjusted by the adjustment factor. To fuse the fundamental frequency information and time domain information, the encoder outputs the time domain features. .

[0049] Boundary Aware Module (CBM) The features output by cross-attention and the encoder output are jointly input into the CBM module. The CBM module consists of a fusion branch and a difference extraction branch. The difference extraction branch is responsible for capturing the spatiotemporal context information between the features of the two stages, focusing on extracting the correlation between local content and the overall scene. The fusion branch uses the weight allocation in the difference extraction branch to constrain the integration of information. The two gains are processed through a ReLU activation function to obtain two complementary heatmaps, which are then fused at the channel level through one-dimensional convolution to obtain... and , This indicates the stronger part after enhancement. This indicates the weaker parts after enhancement, while SimAM1d calculates attention weights by using the statistics (mean and variance) of the input features to highlight the positions with more information in the time dimension.

[0050] (9) In the texture feature aggregation module, and The two features are compressed separately by a one-dimensional convolutional layer. The compressed features are then concatenated to form a merged feature tensor. Next, a joint attention branch consisting of multiple convolutional layers is used to calculate the relationship between the two features, outputting an attention weight matrix for weighted fusion of the two features.

[0051] (10) Decoding stage The decoder has the same structure as the pre-trained version, with the fused features input to the decoder. Temporal gate module: Decoder multi-layer structure: (11) In addition to increasing the model size, the second stage of training also added boundary consistency loss to the loss function: (12) (13) (14) in: It is the time-domain feature of the encoder output. yes Intermediate features after processing by intermediate block TFiLM, Texture features extracted by the texture aggregation module. The difference features between time-domain features and intermediate features The difference between temporal features and texture features, A binary boundary mask is used to mark the locations where features should change. A threshold is set... Give the discriminant line that shows a significant change. The larger the value, the sparser the boundary; here it is set to 0.1. In the loss calculation, two loss functions were used: L2 loss, which calculates the mean square error between the prediction difference and the target mask, and drives the model to produce greater changes at the boundary. (15) BCE loss: Calculates the binary cross-entropy loss between the predicted difference and the target mask, helping the model determine which regions need enhancement and which regions do not.

[0052] (16) Finally, by weighting and combining these two losses: (17) in, and It is control and The hyperparameters for the loss weights are all set to 1 here. (18) in: , , The values ​​represent the spectral loss for FFT window lengths of 512, 1024, and 2048, respectively. For boundary consistency loss, This is due to audio loss.

[0053] Both training phases utilize an encoder-decoder network structure, with each encoder and decoder containing six complex convolutional layers. The number of channels in each layer is set to (24, 48, 72, 96, 120, 144), respectively. The TFiLM module is placed in the third layer of the encoder, and an eight-head attention network is used for multi-head attention.

[0054] All input narrowband speech was 8kHz, and all output speech was 16kHz. The learning rate was set to 1e-5, and the Adam optimizer was used. The epoch count was 50, and the batch size was 64. The model was trained on an Ubuntu 20.04.6 server, using Python 3.10 and the PyTorch framework.

[0055] Figure 4 This is the audio time-domain waveform before bandwidth expansion; Figure 5 This is the audio spectrum before bandwidth expansion; Figure 6 This is the audio time-domain waveform after bandwidth expansion; Figure 7 This is the audio spectrum diagram after bandwidth expansion.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for bandwidth extension of narrowband voice signals, characterized in that: Includes the following steps: Acquire narrowband speech signals; Construct a speech bandwidth extension model based on boundary awareness and fundamental frequency feature fusion; The speech bandwidth expansion model is trained to obtain a trained speech bandwidth expansion model; By inputting the narrowband speech signal into a trained speech bandwidth extension model, the bandwidth extension of the narrowband speech signal can be achieved.

2. The bandwidth extension method for narrowband voice signals according to claim 1, characterized in that: The speech bandwidth extension model based on boundary awareness and fundamental frequency feature fusion includes: First encoder: used to mark and compress the temporal features of narrowband speech signals; The first TFiLM module is used to divide the labeled and compressed temporal features output by the first encoder into multiple blocks through pooling downsampling. Each block will be processed by an LSTM layer to calculate the adjustment factor and reshape the temporal features back to the original dimension. The second encoder is used to calculate the activation features and confidence level of the narrowband speech signal in the frequency domain and output the features. Multi-head cross-attention mechanism module: It is used to calculate the attention weight based on the similarity between the query and the key for the time-domain features and frequency-domain features output by the first TFiLM module, and to perform weighted summation of the V values ​​to obtain a weighted feature representation that integrates information from the fundamental frequency features and time-domain features; The association module is used to add the features output by the first TFiLM module and the fused features output by the multi-head cross-attention mechanism module to obtain feature information containing both. The merged features are activated by Sigmoid to generate an interaction mask. The important parts of the attention features are extracted from this mask and added to the basic features to obtain the preliminary enhanced features. Boundary awareness module: used to assess the features based on the initial enhanced features output by the association module, the temporal features reconstructed back to the original dimension output by the first TFiLM module, and the labeled and compressed features output by the first encoder. Capture the spatiotemporal correlation between local content and the overall scene, extract the weights of branches to integrate information, and perform weighted fusion of features containing fundamental frequency information and texture features; Decoder module: Used to extract key features related to low and high frequencies from the fused fundamental frequency information output by the boundary perception module, and then splice it with the original audio signal and adjust the dimensions to output the final bandwidth-extended audio.

3. The bandwidth extension method for narrowband voice signals according to claim 2, characterized in that: The first encoder includes: The first encoder layer is used to compress the input narrowband speech signal once; The second TFiLM module is used to perform key feature annotation on the compressed narrowband speech signal output by the first encoder layer; Second encoder layer: used to perform secondary compression on the key features of the annotations output by the second TFiLM module; Bottleneck layer: Used to compress the features output after the second encoder layer has been compressed twice and then output them again.

4. The bandwidth extension method for narrowband voice signals according to claim 2, characterized in that: The boundary awareness module includes a difference extraction branch: used to capture the spatiotemporal correlation between local content and the overall scene; Fusion branch: Used to integrate information by utilizing the weights of the difference extraction branch to analyze the spatiotemporal correlation features between the local content output by the difference extraction branch and the overall scene. Texture aggregation module: Based on the fused difference features output by the fusion branch and the enhanced features output by the association module, channel compression is performed, and a joint attention mechanism is used to perform weighted fusion of features containing fundamental frequency information and texture features.

5. The bandwidth extension method for narrowband voice signals according to claim 2, characterized in that: The decoder includes: First decoder layer: used to restore the output features of the texture fusion module and the features concatenated by the first encoder module; The third TFiLM module is used to annotate the important features restored by the first decoder layer; The second decoder layer is used to restore the labeled features output by the third TFiLM module; then it is concatenated with the original audio signal and the final bandwidth-extended audio is output through one-dimensional convolution to adjust the dimension.

6. The bandwidth extension method for narrowband voice signals according to claim 5, characterized in that: The first decoder layer and the second decoder layer have the same structure. The first decoder layer includes a time-domain gating module, which is used to generate a gating mask according to the time dimension, dynamically and selectively retain or mask channel features, and filter out key features related to low frequency and high frequency.

7. The bandwidth extension method for narrowband voice signals according to claim 2, characterized in that: The expression for the adjustment factor is as follows: in: Generate adjustment factors for each block of the LSTM in the TFiLM module. The TFiLM module in the Wave encoder is the post-processing module for the first three encoder layers, and its output features are the features of the third convolutional layer of the encoder. .

8. The bandwidth extension method for narrowband voice signals according to claim 2, characterized in that: The loss function of the speech bandwidth extension model is constructed by adding the audio loss function and the multi-resolution loss function and subtracting the consistency loss function; The multi-resolution loss function is the sum of the loss functions with an FFT window length of 512, an FFT window length of 1024, and an FFT window length of 2048.

Images