Lightweight voice band extension method and device for edge device, terminal and medium

Abstract

The edge device-oriented lightweight voice band expansion method, device, terminal and medium provided by the application belong to the technical field of voice signal processing, and the method comprises the following steps: obtaining a logarithmic domain narrowband audio signal amplitude spectrum and a logarithmic domain mixed amplitude spectrum; constructing a white noise amplitude spectrum; inputting the logarithmic domain narrowband audio signal amplitude spectrum, the logarithmic domain mixed amplitude spectrum and the white noise amplitude spectrum into a trained voice bandwidth expansion model to generate a first high-frequency component corresponding to a consonant in the narrowband audio signal and a second high-frequency component corresponding to a vowel in the narrowband audio signal, and then obtaining a predicted amplitude spectrum; expanding phase information of the narrowband audio signal, generating a wideband audio signal according to the predicted amplitude spectrum and the expanded phase information of the narrowband audio signal, and outputting the wideband audio signal. The voice bandwidth expansion model is used to reconstruct the consonant, so that the reconstructed consonant component has higher fidelity, and the intelligibility of the reconstructed voice is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech signal processing, and particularly relates to a lightweight speech frequency band expansion method and device for edge devices, a terminal and a medium. BACKGROUND

[0002] Speech bandwidth expansion (BWE) technology aims to reconstruct the missing high-frequency components (>= 2kHz) in narrowband signals (usually band <= 3kHz), thereby improving the perceived intelligibility of speech, and is a key means for edge devices (such as mobile phones, Internet of Things terminals, etc.) to optimize speech experience under limited bandwidth conditions. However, the existing mainstream methods generally rely on half-wave rectification (HWR) technology to recover high-frequency information, and the reconstruction mechanism is based on the periodic harmonic structure of vowels to extrapolate high frequencies. This mechanism is essentially incompatible with the non-periodic, noise-like spectral characteristics of consonants, resulting in distortion in the reconstruction of high-frequency components of consonants, which in turn affects the intelligibility of the reconstructed speech.

[0003] Therefore, the prior art has defects and needs to be improved and developed. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a lightweight speech frequency band expansion method and device for edge devices, a terminal and a storage medium, aiming to solve the problem of distortion in the reconstruction of high-frequency components of consonants and the resulting impact on the intelligibility of the overall speech.

[0005] The technical solution adopted by the present application to solve the technical problem is as follows:

[0006] In a first aspect, the present application provides a lightweight speech frequency band expansion method for edge devices, comprising:

[0007] Obtaining a narrowband audio signal from an edge device and preprocessing it to obtain a log-domain narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum;

[0008] Constructing a white noise amplitude spectrum;

[0009] Inputting the log-domain narrowband audio signal amplitude spectrum, the log-domain mixed amplitude spectrum and the white noise amplitude spectrum into a trained speech bandwidth expansion model, using the speech bandwidth expansion model to generate a first high-frequency component corresponding to a consonant in the narrowband audio signal and a second high-frequency component corresponding to a vowel in the narrowband audio signal, and obtaining a predicted amplitude spectrum based on the first high-frequency component and the second high-frequency component;

[0010] The phase information of the narrowband audio signal is expanded, and a wideband audio signal is generated and output according to a predicted amplitude spectrum and the expanded phase information of the narrowband audio signal.

[0011] In an embodiment, the narrowband audio signal is obtained from an edge device and preprocessed to obtain a log-domain narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum, including:

[0012] The narrowband audio signal is obtained from an edge device;

[0013] The sampling rate of the narrowband audio signal is upsampled to a preset target sampling rate to obtain an upsampled time-domain waveform;

[0014] The upsampled time-domain waveform is subjected to a half-wave rectification operation to obtain a rectified time-domain waveform;

[0015] The time-domain waveform and the rectified time-domain waveform are subjected to short-time Fourier transform using a Hanning window, respectively, to obtain a narrowband audio signal amplitude spectrum and a rectified narrowband audio signal amplitude spectrum;

[0016] The narrowband audio signal amplitude spectrum and the rectified narrowband audio signal amplitude spectrum are superimposed to obtain a mixed amplitude spectrum;

[0017] The narrowband audio signal amplitude spectrum and the mixed amplitude spectrum are subjected to logarithmic transformation, respectively, to obtain a log-domain narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum.

[0018] In an embodiment, the speech bandwidth expansion model includes a feature extraction module, a double-weighted Gaussian mixture module, a fusion module, and a frequency band guided masking module.

[0019] In an embodiment, the log-domain narrowband audio signal amplitude spectrum, the log-domain mixed amplitude spectrum, and the white noise amplitude spectrum are input into a trained speech bandwidth expansion model, the speech bandwidth expansion model is used to generate a first high-frequency component corresponding to a consonant in the narrowband audio signal and a second high-frequency component corresponding to a vowel in the narrowband audio signal, respectively, and a predicted amplitude spectrum is obtained based on the first high-frequency component and the second high-frequency component, including:

[0020] The log-domain mixed amplitude spectrum is input into the feature extraction module to obtain a time sequence feature representing the characteristics of the narrowband audio signal after processing;

[0021] The time sequence feature, the log-domain mixed amplitude spectrum, and the white noise amplitude spectrum are input into the double-weighted Gaussian mixture module, two sets of control parameters are generated based on the time sequence feature to drive the first weighted Gaussian mixture module and the second weighted Gaussian mixture module of the double-weighted Gaussian mixture module to operate, respectively, to correspondingly generate the first high-frequency component and the second high-frequency component.

[0022] inputting the first high frequency component and the second high frequency component into the fusion module to fuse the first high frequency component and the second high frequency component, and outputting an initial prediction amplitude spectrum;

[0023] inputting the log domain narrowband audio signal amplitude spectrum and the initial prediction amplitude spectrum into the frequency band guided masking module to optimize and obtain a prediction amplitude spectrum.

[0024] In an embodiment, inputting the first high frequency component and the second high frequency component into the fusion module to fuse the first high frequency component and the second high frequency component, and outputting an initial prediction amplitude spectrum, comprises:

[0025] inputting the first high frequency component and the second high frequency component into the fusion module, splicing the first high frequency component and the second high frequency component by using the fusion module to obtain spliced features;

[0026] processing the spliced features to generate weight coefficients;

[0027] performing weighted calculation on the first high frequency component and the second high frequency component based on the weight coefficients to obtain an initial prediction amplitude spectrum.

[0028] In an embodiment, expanding the phase information of the narrowband audio signal, generating and outputting a wideband audio signal according to the prediction amplitude spectrum and the expanded phase information of the narrowband audio signal, comprises:

[0029] flip-expanding the phase information of the narrowband audio signal;

[0030] combining the prediction amplitude spectrum and the expanded phase information of the narrowband audio signal, performing inverse short-time Fourier transform by using a Hanning window to generate a wideband audio signal;

[0031] outputting the wideband audio signal.

[0032] In an embodiment, the training step of the speech bandwidth expansion model comprises:

[0033] obtaining a training data pair, the training data pair being composed of a narrowband audio signal sample and a corresponding real wideband audio sample;

[0034] preprocessing the narrowband audio signal sample to obtain a log domain narrowband audio signal amplitude spectrum and a log domain mixed amplitude spectrum for training;

[0035] constructing a white noise amplitude spectrum for training;

[0036] constructing a training framework comprising a generator and a discriminator, the generator being a speech bandwidth expansion model to be trained, and the discriminator being used to distinguish between generated wideband audio samples and real wideband audio samples.

[0037] inputting the log domain narrowband audio signal amplitude spectrum, the log domain mixed amplitude spectrum and the white noise amplitude spectrum into a generator to obtain a predicted training amplitude spectrum;

[0038] performing inverse short-time Fourier transform on the predicted training amplitude spectrum and phase information of the real wideband audio sample to generate a wideband audio sample and output the wideband audio sample;

[0039] calculating a training loss between the wideband audio sample and the real wideband audio sample;

[0040] optimizing parameters of the generator and the discriminator based on the training loss until the loss converges to obtain a trained voice bandwidth expansion model.

[0041] In a second aspect, an embodiment of the present application further provides a lightweight voice frequency band expansion device for an edge device, the device comprising:

[0042] a first preprocessing module configured to obtain a narrowband audio signal from the edge device and perform preprocessing to obtain a log domain narrowband audio signal amplitude spectrum and a log domain mixed amplitude spectrum;

[0043] a second preprocessing module configured to construct a white noise amplitude spectrum;

[0044] an amplitude spectrum prediction module configured to input the log domain narrowband audio signal amplitude spectrum, the log domain mixed amplitude spectrum and the white noise amplitude spectrum into the trained voice bandwidth expansion model, generate a first high-frequency component corresponding to a consonant in the narrowband audio signal and a second high-frequency component corresponding to a vowel in the narrowband audio signal by using the voice bandwidth expansion model, and obtain a predicted amplitude spectrum based on the first high-frequency component and the second high-frequency component;

[0045] a waveform reconstruction module configured to expand phase information of the narrowband audio signal, generate a wideband audio signal based on the predicted amplitude spectrum and the expanded phase information of the narrowband audio signal, and output the wideband audio signal.

[0046] In a third aspect, an embodiment of the present application further provides a terminal, comprising a memory, a processor and a lightweight voice frequency band expansion program for an edge device stored in the memory and executable on the processor, wherein the lightweight voice frequency band expansion program for an edge device, when executed by the processor, implements steps of the lightweight voice frequency band expansion method for an edge device.

[0047] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium storing an edge device-oriented lightweight speech bandwidth expansion program, the edge device-oriented lightweight speech bandwidth expansion program being executable to implement the steps of the edge device-oriented lightweight speech bandwidth expansion method as described above.

[0048] The present application has the following beneficial effects: The present application obtains a narrowband audio signal from an edge device and pre-processes the narrowband audio signal to obtain a log-domain narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum; a white noise amplitude spectrum is constructed; the log-domain narrowband audio signal amplitude spectrum, the log-domain mixed amplitude spectrum and the white noise amplitude spectrum are input into a trained speech bandwidth expansion model, the speech bandwidth expansion model is used to respectively generate a first high-frequency component corresponding to a consonant in the narrowband audio signal and a second high-frequency component corresponding to a vowel in the narrowband audio signal, and a predicted amplitude spectrum is obtained based on the first high-frequency component and the second high-frequency component; the phase information of the narrowband audio signal is expanded, a wideband audio signal is generated and output according to the predicted amplitude spectrum and the expanded phase information of the narrowband audio signal. The present application uses the speech bandwidth expansion model to specifically reconstruct the consonant, which can ensure that the reconstructed consonant component has higher fidelity, and thus ensures the intelligibility of the reconstructed speech. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a flowchart of a preferred embodiment of the edge device-oriented lightweight speech bandwidth expansion method in the present application.

[0050] Figure 2 is a flowchart of generating control parameters and weights in the present application.

[0051] Figure 3 is a flowchart of generating a wideband audio signal in the present application.

[0052] Figure 4 is a structural schematic diagram of a preferred embodiment of the edge device-oriented lightweight speech bandwidth expansion device in the present application.

[0053] Figure 5 is a principle block diagram of a terminal in the present application. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions and advantages of the present application clearer and more explicit, the present application is further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0055] The speech bandwidth extension (BWE) technology aims to reconstruct the missing high-frequency components (≥2kHz) in the narrowband signal (usually the frequency band ≤3kHz), thereby improving the perceptual intelligibility of the speech, and is a key means for optimizing the speech experience of edge devices (such as mobile phones, Internet of Things terminals, etc.) under limited bandwidth conditions. However, the existing mainstream methods generally rely on the half-wave rectification (HWR) technology to recover the high-frequency information, and the reconstruction mechanism is based on the periodic harmonic structure of vowels to extrapolate the high frequencies. This mechanism is essentially incompatible with the non-periodic, noise-like spectral characteristics of consonants, resulting in distortion in the reconstruction of high-frequency components of consonants, which in turn affects the intelligibility of the reconstructed speech.

[0056] In view of the above defects of the prior art, the present application provides a lightweight speech bandwidth extension method and device for edge devices, a terminal and a medium, the method comprising: obtaining a log-domain narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum; constructing a white noise amplitude spectrum; inputting the log-domain narrowband audio signal amplitude spectrum, the log-domain mixed amplitude spectrum and the white noise amplitude spectrum into a trained speech bandwidth extension model, using the speech bandwidth extension model to generate a first high-frequency component corresponding to a consonant in the narrowband audio signal and a second high-frequency component corresponding to a vowel in the narrowband audio signal, and obtaining a predicted amplitude spectrum based on the first high-frequency component and the second high-frequency component; extending the phase information of the narrowband audio signal, generating a wideband audio signal according to the predicted amplitude spectrum and the extended phase information of the narrowband audio signal, and outputting the wideband audio signal. The present application uses the speech bandwidth extension model to specifically reconstruct the consonants, which can ensure that the reconstructed consonant components have higher fidelity, thereby ensuring the intelligibility of the reconstructed speech.

[0057] For a better understanding of the present application, reference will be made to the following detailed description of the embodiments of the present application in conjunction with the accompanying drawings, in which: Figure 1 The lightweight speech bandwidth extension method for edge devices according to an embodiment of the present application comprises the following steps:

[0058] Step S100: obtaining a narrowband audio signal from an edge device and pre-processing to obtain a log-domain narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum.

[0059] Specifically, the edge device can be a mobile phone or an Internet of Things terminal device. In the process of obtaining a narrowband audio signal from an edge device and pre-processing to obtain a narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum, the narrowband audio signal is first obtained from the edge device, and then the sampling rate of the narrowband audio signal is upsampled to a preset target sampling rate to obtain an upsampled time-domain waveform The target sampling rate can be 22050. The half-wave rectification operation is performed on the upsampled time-domain waveform to obtain a rectified time-domain waveform It can be understood that = The time-domain waveform and the rectified time-domain waveform are respectively subjected to short-time Fourier transform by using a Hanning window to obtain narrow-band audio signal amplitude spectrum and rectified narrow-band audio signal amplitude spectrum When the short-time Fourier transform (STFT) is performed by using the Hanning window, the frame length is 1024 sample points, the frame shift is 256 sample points, and the number of Fast Fourier Transform (FFT) points is 1024. The obtained narrow-band audio signal amplitude spectrum and rectified narrow-band audio signal amplitude spectrum are both 513-dimensional. The narrow-band audio signal amplitude spectrum and the rectified narrow-band audio signal amplitude spectrum are superimposed to obtain a mixed amplitude spectrum. This process can be represented as Finally, the narrow-band audio signal amplitude spectrum and the mixed amplitude spectrum are respectively subjected to logarithmic transformation to obtain a logarithmic-domain narrow-band audio signal amplitude spectrum and a logarithmic-domain mixed amplitude spectrum. The process of obtaining the logarithmic-domain mixed amplitude spectrum can be represented as:

[0060] ;

[0061] wherein, is used to avoid numerical instability in logarithmic calculation.

[0062] The amplitude spectrum of the original narrow-band audio signal has a significant dynamic range difference, and the energy of the low and medium frequencies is much higher than that of the high frequencies. Direct input of the speech bandwidth expansion model will cause the high-frequency details to be covered, which not only reduces the reconstruction accuracy of the high frequencies, but also causes training instability due to excessive numerical fluctuations, further restricting the convergence effect of the speech bandwidth expansion model. The wide dynamic range of the original amplitude spectrum can be compressed to a narrow range in the logarithmic domain by logarithmic transformation, which reduces the covering of low-frequency energy on high-frequency details, reduces numerical fluctuations, and improves the model training convergence speed and stability.

[0063] Referring to Figure 1 The lightweight speech frequency band expansion method for edge devices according to the embodiment of the present application further includes the following steps:

[0064] Step S200, constructing a white noise amplitude spectrum.

[0065] Specifically, a white noise amplitude spectrum with a mean value of -1 is constructed as .

[0066] Referring to Figure 1 The lightweight speech frequency band expansion method for edge devices according to the embodiment of the present application further includes the following steps:

[0067] The step S300 comprises: inputting the logarithmic domain narrowband audio signal amplitude spectrum, the logarithmic domain mixed amplitude spectrum and the white noise amplitude spectrum into a trained speech bandwidth extension model, respectively generating a first high frequency component corresponding to a consonant in the narrowband audio signal and a second high frequency component corresponding to a vowel in the narrowband audio signal by using the speech bandwidth extension model, and obtaining a predicted amplitude spectrum based on the first high frequency component and the second high frequency component.

[0068] Specifically, the existing HWB-Net (High-Performance and Efficient Hybrid Waveform Bandwidth Extension Method) is a commonly used speech bandwidth extension model, which has an advantage of being convenient for edge device deployment, with a parameter quantity of 194K and a multiply-add operation number per second of 12M. However, it is not optimized for the non-periodicity and noise-like characteristics of consonants (such as / f / , / s / and the like). In order to overcome this defect, the present application designs a speech bandwidth extension model (HWB-PLUS) which is improved based on the existing HWB-Net model and performs targeted processing on vowels and consonants respectively through a double-path reconstruction mechanism. The speech bandwidth extension model comprises a feature extraction module, a double-weighted Gaussian mixture module, a fusion module and a frequency band guided masking module.

[0069] In one implementation, the logarithmic domain narrowband audio signal amplitude spectrum, the logarithmic domain mixed amplitude spectrum and the white noise amplitude spectrum are input into a trained speech bandwidth extension model, a first high frequency component corresponding to a consonant in the narrowband audio signal and a second high frequency component corresponding to a vowel in the narrowband audio signal are respectively generated by using the speech bandwidth extension model, and a predicted amplitude spectrum is obtained based on the first high frequency component and the second high frequency component, comprising:

[0070] The logarithmic domain mixed amplitude spectrum is input into the feature extraction module to obtain time sequence features representing characteristics of the narrowband audio signal after processing;

[0071] The time sequence features, the logarithmic domain mixed amplitude spectrum and the white noise amplitude spectrum are input into the double-weighted Gaussian mixture module, two groups of control parameters are generated based on the time sequence features, and the first weighted Gaussian mixture module and the second weighted Gaussian mixture module of the double-weighted Gaussian mixture module are respectively driven to operate to correspondingly generate the first high frequency component and the second high frequency component;

[0072] The first high frequency component and the second high frequency component are input into the fusion module for fusion to output an initial predicted amplitude spectrum;

[0073] The logarithmic domain narrowband audio signal amplitude spectrum and the initial predicted amplitude spectrum are input into the frequency band guided masking module for optimization to obtain a predicted amplitude spectrum.

[0074] Specifically, the feature extraction module includes a linear frequency to equivalent rectangular bandwidth conversion (Linear2ERB) module and an encoder module. Since the resolution of the human ear is different for different frequencies, the resolution for the low frequency band is higher, and the resolution for the high frequency band is lower, the linear frequency to equivalent rectangular bandwidth conversion module uses its internal triangular ERB (Equivalent Rectangular Bandwidth) filter to process the received log domain hybrid amplitude spectrum, maps the log domain hybrid amplitude spectrum to the ERB (Equivalent Rectangular Bandwidth) based perceptual frequency axis, and obtains an equivalent rectangular bandwidth frequency band spectrum, thereby providing a more human ear perceptual feature input for the subsequent module. The equivalent rectangular bandwidth frequency band spectrum is 128-dimensional, and is more in line with the human auditory characteristics.

[0075] The encoder module includes four one-dimensional convolution layers and a grouped gated recurrent unit. The one-dimensional convolution layer is a core feature encoding module in the speech bandwidth expansion model, which is used to extract local features and compress dimensions of the equivalent rectangular bandwidth frequency band spectrum, to provide compact and meaningful spectral features for the subsequent decoding module, while supporting real-time streaming inference. The channel number configuration of the four one-dimensional convolution layers is 128, 128, 64, and 64, respectively. The kernel size of each convolution layer is 3, and the ReLU activation function is used. After the local features are extracted by the four one-dimensional convolution layers, the grouped gated recurrent unit is used to capture the inter-frame temporal dependence of the spectral features (such as the continuity of the speech fundamental frequency, the dynamic change of the formant, etc.), while taking into account the model lightweight and streaming inference requirements, to provide more temporally consistent feature support for the subsequent amplitude completion and phase optimization. The grouped gated recurrent unit includes two layers of gated recurrent units, and the input and hidden layer dimensions are both 64. The grouped gated recurrent unit finally outputs the time sequence feature , where T is the time frame, F = 64, is a real set.

[0076] The dual weighted Gaussian mixture module (DualWGMM) includes a first weighted Gaussian mixture module and a second weighted Gaussian mixture module. The first weighted Gaussian mixture module (ConsWGMM) is used for high-frequency component reconstruction of the consonants in the narrowband audio signal, and the second weighted Gaussian mixture module (VowelWGMM) is used for high-frequency component reconstruction of the vowels in the narrowband audio signal. The base signal of the first weighted Gaussian mixture module is Its non-periodic, noise-like statistical characteristics perfectly match high-frequency consonants (such as / f / , / s / ), overcoming the shortcomings of existing technologies in adapting to consonants. The first weighted Gaussian mixing module targets the 2000-10000Hz high-frequency band (the core region of consonant energy) and uses 32 Gaussian components for fine modeling to avoid consonant reconstruction distortion. The base signal of the second weighted Gaussian mixing module is... It retains the enhancement effect of half-wave rectification on vowel harmonics in existing technologies, focuses on the high-frequency harmonic extension of mid- and low-frequency vowels, and maintains the continuity of vowel reconstruction in this way, avoiding the decline in vowel quality due to model splitting.

[0077] Considering the wideband distribution characteristics of the high-frequency consonant band, the mean of the first weighted Gaussian mixture module The standard deviation is obtained by averaging 32 values ​​uniformly sampled within the linear frequency range of 2000-10000Hz. It is fixed at 10. This balances the flexibility and stability of the first weighted Gaussian mixture module.

[0078] Furthermore, the parameters of the weighted Gaussian mixture model in existing HWB-Net models are mostly set empirically, without considering the nonlinear perceptual characteristics of the human auditory system. Human sensitivity to high frequencies follows a Mel-scale law (i.e., sensitivity decreases with low and mid-frequency response and decreases with high-frequency response), but empirical parameters do not focus on the key frequency bands of the Mel-scale. This causes the model to waste computational resources on non-critical frequencies such as background noise and inaudible sounds, failing to efficiently optimize perception-related high-frequency details and limiting the quality of high-frequency reconstruction. To address the lack of acoustic rationality in parameter initialization, this invention designs the initial mean and standard deviation of the second weighted Gaussian mixture module based on the Mel-scale, ensuring that the parameters focus on the key perceptual frequency bands. Specifically, the initial mean of the second weighted Gaussian mixture module... Set the center frequency of the k-th filter in the Mel-scale triangular filter bank, directly aligning it with the mid-to-low frequency range (i.e., the vowel core distribution area) that is sensitive to human hearing. Set the initial standard deviation... The formula for calculating the bandwidth of a Mel filter is as follows: .in, For the first The bandwidth of each Mel filter is determined in this way to ensure that the spectral coverage of each Gaussian component is consistent with that of the Mel filter, thus avoiding wasting resources on non-critical frequencies.

[0079] After the time series features, the logarithmic domain mixed amplitude spectrum, and the logarithmic domain amplitude spectrum are input into the double-weighted Gaussian mixture module, the first weighted Gaussian mixture module and the second weighted Gaussian mixture module respectively process the time series features to generate corresponding control parameters.

[0080] The first weighted Gaussian mixture module contains two linear layers in parallel, belonging to two branches respectively. In the branch for calculating the final standard deviation, the linear layer of this branch processes the time sequence feature After that, the softplus function is processed to obtain the standard deviation offset of the first t frame of the first k Gaussian component . The softplus function is a smooth approximation function used to simulate the nonlinear characteristics of the ReLU (Rectified Linear Unit) function while maintaining differentiability. Then, based on the initial standard deviation of the first weighted Gaussian mixture module, the final standard deviation of the first weighted Gaussian mixture module is calculated. The formula involved here is:

[0081] ;

[0082] In the branch for calculating the prediction weight, the linear layer of this branch processes the time sequence feature After that, the Sigmoid activation function is processed to obtain the prediction weight of the first t frame of the first k Gaussian component . The Sigmoid activation function is a nonlinear function that can map any real number input to the interval (0, 1). Then, using , , as the control parameters of the first weighted Gaussian mixture module, the first weighted Gaussian mixture module generates the first high-frequency component . The formula involved in this process is as follows:

[0083] ;

[0084] ;

[0085] wherein is a Gaussian mixture function simulating the high-frequency distribution of consonants. is the amplitude spectrum of white noise.

[0086] The second weighted Gaussian mixture module contains two linear layers in parallel, belonging to two branches respectively. In the branch for calculating the final standard deviation, the linear layer of this branch processes the time sequence feature After that, the softplus function is processed to obtain the standard deviation offset of the first t frame of the first k Gaussian component . Then, based on the initial standard deviation of the second weighted Gaussian mixture module, the final standard deviation of the second weighted Gaussian mixture module is calculated. The formula involved here is:

[0087] ;

[0088] In the branch that calculates the prediction weights, the linear layer of this branch processes temporal features. After processing with the Sigmoid activation function, the first... t Frame number k Prediction weights of Gaussian components Finally, using , and As control parameters for the second weighted Gaussian mixing module, it drives the second weighted Gaussian mixing module to generate the second high-frequency component. The formulas involved in this process are as follows:

[0089] ;

[0090] ;

[0091] in, To simulate the high-frequency distribution of vowels, a Gaussian mixture function. For frequency variables, This indicates point-by-point multiplication. The logarithmic domain mixed amplitude spectrum of a narrowband audio signal. This represents the probability density function of a Gaussian distribution.

[0092] In one implementation, the first high-frequency component and the second high-frequency component are input into the fusion module for fusion to output an initial predicted amplitude spectrum, including:

[0093] The first high-frequency component and the second high-frequency component are input into the fusion module, and the first high-frequency component and the second high-frequency component are spliced ​​together by the fusion module to obtain the splicing feature;

[0094] The splicing features are processed to generate weight coefficients;

[0095] The first high-frequency component and the second high-frequency component are weighted based on the weighting coefficients to obtain the initial predicted amplitude spectrum.

[0096] Specifically, according to Dimensional splicing of the first high-frequency component Second high frequency component The spliced ​​features are obtained. The linear layer within the fusion module transforms these spliced ​​features, mapping them to a scalar. This scalar is then processed by a sigmoid activation function to generate frame-level weight coefficients ranging from [0,1]. If the current frame mainly consists of vowels, then Approximately 1; if it is a consonant, then close to 0. Then, the initial predicted magnitude spectrum is calculated according to the following formula: In this way, the natural transition between vowels and consonants is ensured. The flowchart for generating the control parameters and the generation weights in the present application is shown in Figure 2

[0097] Please refer to Figure 1 The edge device-oriented lightweight voice frequency band expansion method according to the embodiment of the present application further includes the following steps:

[0098] Step S400, the phase information of the narrowband audio signal is expanded, and a wideband audio signal is generated and output according to the predicted magnitude spectrum and the expanded phase information of the narrowband audio signal.

[0099] Specifically, the phase information of the narrowband audio signal is flipped and expanded; the predicted magnitude spectrum is combined with the expanded phase information of the narrowband audio signal, inverse short-time Fourier transform is performed using a Hanning window to generate a wideband audio signal; and the wideband audio signal is output. This process can be represented by the following formula: ; wherein, is the wideband audio signal, is the inverse short-time Fourier transform, is the predicted magnitude spectrum, e is the base of the natural logarithm, and j is the imaginary unit, is the phase information of the narrowband audio signal. The flowchart for generating the wideband audio signal in the present application can be shown in Figure 3

[0100] In an implementation manner, the training step of the voice bandwidth expansion model includes:

[0101] Obtaining a training data pair composed of a narrowband audio signal sample and its corresponding real wideband audio sample;

[0102] Pretreating the narrowband audio signal sample to obtain a training log-domain narrowband audio signal magnitude spectrum and a log-domain mixed magnitude spectrum;

[0103] Constructing a training white noise magnitude spectrum;

[0104] Constructing a training framework containing a generator and a discriminator, wherein the generator is the voice bandwidth expansion model to be trained, and the discriminator is used to distinguish between the generated wideband audio sample and the real wideband audio sample;

[0105] Inputting the training log-domain narrowband audio signal magnitude spectrum, the log-domain mixed magnitude spectrum, and the white noise magnitude spectrum into the generator to obtain a predicted training magnitude spectrum;

[0106] ​​performing inverse short-time Fourier transform on the predicted training amplitude spectrum and phase information of the real wideband audio sample to generate a wideband audio sample and output;

[0107] calculating a training loss between the wideband audio sample and the real wideband audio sample;

[0108] optimizing parameters of the generator and the discriminator based on the training loss until the loss converges, to obtain a trained voice bandwidth expansion model.

[0109] Specifically, the formula of the loss function in the application is:

[0110]

[0111] wherein, . is a waveform loss, is a multi-resolution short-time Fourier transform loss, is an adversarial loss, is a feature matching loss.

[0112] The calculation formula of the waveform loss is:

[0113] ;

[0114] wherein, T is the total number of narrowband audio signal samples, and t is the serial number of the frame. is the predicted t-th frame of the wideband audio signal during training, is the t-th frame of the real wideband audio sample.

[0115] The calculation formula of the multi-resolution short-time Fourier transform loss is:

[0116] ;

[0117] wherein, is 1, denotes the spectral convergence loss under the i-th group of resolutions, denotes the log amplitude spectrum loss under the i-th group of resolutions. The value range of i is 1-3, corresponding to three groups of different time-frequency resolution short-time Fourier transform configurations. The first group is: the number of fast Fourier transform points is 512, the frame shift is 50, and the window length is 240; the second group is: the number of fast Fourier transform points is 1024, the frame shift is 120, and the window length is 600; the third group is: the number of fast Fourier transform points is 2048, the frame shift is 240, and the window length is 1200.

[0118] The adversarial loss includes a discriminator loss and a generator loss Discriminator loss is calculated as: Generator loss is calculated as: wherein represents the output of the discriminator for a real wideband audio sample, represents the output of the discriminator for a narrowband audio signal sample, represents the L2 distance, is a real wideband audio sample, is a narrowband audio signal sample.

[0119] Feature matching loss is calculated as: wherein L is the number of discriminator layers, represents the features of the layer of the discriminator.

[0120] The present application adopts the following indicators to compare the method of the present application (HWB-Plus) and the HWB-Net method (High-Performance and Efficient Hybrid Waveform Bandwidth Extension Method), the bandlimited sinc interpolation method, and the BAE-Lite (bandwidth adaptive extension neural network) method in the prior art: the log spectral distance (Log Spectral Distance, LSD) is used to quantify the difference between the reconstructed spectrum and the real spectrum, and the smaller the value is, the better. The deep noise suppression speech quality score (Deep Noise Suppression Mean Opinion Score, DNSMOS) is used to evaluate the overall auditory quality of the speech, and the score conforms to the P.808 standard, and the larger the value is, the better, and the numerical range is: [0, 5]. The perceptual evaluation of speech quality (Perceptual Evaluation of Speech Quality, PESQ) method is used to evaluate the speech quality, and the larger the value is, the better, and the numerical range is: [-0.5, 4.5]. The virtual speech quality objective listener (Virtual Speech Quality Objective Listener, VISQOL) is used to evaluate the objective quality of the speech, and the larger the value is, the better, and the numerical range is: [0, 5]. The non-intrusive speech quality assessment (Non-Intrusive Speech Quality Assessment, NISQA) is used to evaluate the objective quality of the speech, and the larger the value is, the better, and the numerical range is: [0, 5]. The parameter quantity (Para.) and multiply-accumulate operations per second (Multiply-Accumulate Operations per Second, MACs) are used to measure the deployment efficiency.

[0121] The performance index comparison of the four methods is shown in Table 1:

[0122] Table 1

[0123]

[0124] As can be seen from Table 1, the HWB-PLUS method of the present application is superior to BAE-Lite and HWB-Net in terms of perception indicators under the premise that the parameters and calculation amount are basically consistent with those of HWB-Net, proving that the target of "lightweight invariable and performance improvement" is achieved; the consonant distortion is significantly reduced, and the speech is more natural. Although the log spectral distance is slightly higher than that of BAE-Lite, the perceptual quality is more in line with the human hearing demand, and conforms to the actual application scene of the edge speech band expansion.

[0125] In addition, in order to verify the effects of the double-weighted Gaussian mixture module, the method based on the initialization parameters of the mel scale, and the application of the logarithmic transformation in the present application, the indicators are observed after removing the three respectively, and the results are shown in Table 2.

[0126] Table 2

[0127]

[0128] As can be seen from Table 2, the deep neural network speech quality score and the virtual speech quality objective listener respectively decrease by 0.19 and 0.19 after removing the double-weighted Gaussian mixture module, proving that the split vowel / consonant modeling is crucial to improving the perceptual quality. The log spectral distance rises to 1.07 after removing the operation based on the initialization parameters of the mel scale, indicating that the parameter acoustic rationality directly affects the spectral matching accuracy. The deep neural network speech quality score decreases to 3.31 after removing the operation of applying the logarithmic transformation, verifying the effects of the logarithmic transformation on high-frequency detail preservation and training stability.

[0129] In summary, the present application obtains a log domain narrowband audio signal amplitude spectrum and a log domain mixed amplitude spectrum by obtaining a narrowband audio signal from an edge device and preprocessing the narrowband audio signal; constructs a white noise amplitude spectrum; inputs the log domain narrowband audio signal amplitude spectrum, the log domain mixed amplitude spectrum, and the white noise amplitude spectrum into a trained speech bandwidth expansion model, generates a first high-frequency component corresponding to a consonant in the narrowband audio signal and a second high-frequency component corresponding to a vowel in the narrowband audio signal by using the speech bandwidth expansion model, and obtains a predicted amplitude spectrum based on the first high-frequency component and the second high-frequency component; expands the phase information of the narrowband audio signal, generates a wideband audio signal according to the predicted amplitude spectrum and the expanded phase information of the narrowband audio signal, and outputs the wideband audio signal. The speech bandwidth expansion model is used to reconstruct the consonant, which can ensure that the reconstructed consonant component has higher fidelity, and thus ensures the intelligibility of the reconstructed speech.

[0130] In one embodiment, as shown in FIG. 1, Figure 4 Based on the above-mentioned lightweight speech band expansion method for edge devices, the present application also correspondingly provides a lightweight speech band expansion device for edge devices, which comprises:

[0131] The first preprocessing module 100 is configured to acquire a narrowband audio signal from an edge device and perform preprocessing to obtain a log-domain narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum.

[0132] The second preprocessing module 200 is configured to construct a white noise amplitude spectrum.

[0133] The amplitude spectrum prediction module 300 is configured to input the log-domain narrowband audio signal amplitude spectrum, the log-domain mixed amplitude spectrum and the white noise amplitude spectrum into a trained voice bandwidth expansion model, generate a first high-frequency component corresponding to a consonant in the narrowband audio signal and a second high-frequency component corresponding to a vowel in the narrowband audio signal by using the voice bandwidth expansion model, and obtain a predicted amplitude spectrum based on the first high-frequency component and the second high-frequency component.

[0134] The waveform reconstruction module 400 is configured to expand phase information of the narrowband audio signal, generate a wideband audio signal according to the predicted amplitude spectrum and the expanded phase information of the narrowband audio signal, and output the wideband audio signal.

[0135] In an embodiment, the first preprocessing module comprises:

[0136] The audio acquisition unit is configured to acquire a narrowband audio signal from an edge device.

[0137] The up-sampling unit is configured to up-sample a sampling rate of the narrowband audio signal to a preset target sampling rate to obtain an up-sampled time-domain waveform.

[0138] The half-wave rectification unit is configured to perform a half-wave rectification operation on the up-sampled time-domain waveform to obtain a rectified time-domain waveform.

[0139] The first short-time Fourier transform unit is configured to superimpose the narrowband audio signal amplitude spectrum and the rectified narrowband audio signal amplitude spectrum to obtain a mixed amplitude spectrum.

[0140] The logarithmic transformation unit is configured to apply a logarithmic transformation to the narrowband audio signal amplitude spectrum and the mixed amplitude spectrum to obtain a log-domain narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum.

[0141] In an embodiment, the amplitude spectrum prediction module comprises:

[0142] The time sequence feature generation unit is configured to input the log-domain mixed amplitude spectrum into the feature extraction module to obtain a time sequence feature representing a narrowband audio signal feature.

[0143] a high-frequency component generation unit, configured to input the time sequence feature, the white noise amplitude spectrum, and the log-domain amplitude spectrum into the double-weighted Gaussian mixture module, generate two groups of control parameters based on the time sequence feature, and drive a first weighted Gaussian mixture module and a second weighted Gaussian mixture module of the double-weighted Gaussian mixture module to operate respectively to correspondingly generate a first high-frequency component and the second high-frequency component;

[0144] a fusion unit, configured to input the first high-frequency component and the second high-frequency component into the fusion module for fusion, and output an initial predicted amplitude spectrum;

[0145] an optimization unit, configured to input the log-domain narrowband audio signal amplitude spectrum and the initial predicted amplitude spectrum into the frequency band guided masking module for optimization to obtain a predicted amplitude spectrum.

[0146] In an embodiment, the apparatus further includes:

[0147] a splicing unit, configured to input the first high-frequency component and the second high-frequency component into the fusion module, and splice the first high-frequency component and the second high-frequency component by using the fusion module to obtain a spliced feature;

[0148] a weight generation unit, configured to process the spliced feature to generate a weight coefficient;

[0149] an initial predicted amplitude spectrum generation unit, configured to perform weighted calculation on the first high-frequency component and the second high-frequency component based on the weight coefficient to obtain an initial predicted amplitude spectrum.

[0150] In an embodiment, the frequency band extension module includes:

[0151] a phase extension unit, configured to perform flip extension on phase information of the narrowband audio signal;

[0152] a first inverse short-time Fourier transform unit, configured to combine the predicted amplitude spectrum and the extended phase information of the narrowband audio signal, perform inverse short-time Fourier transform by using a Hanning window, and generate a wideband audio signal;

[0153] an output unit, configured to output the wideband audio signal.

[0154] In an embodiment, the apparatus further includes:

[0155] a training data pair acquisition unit, configured to acquire a training data pair, the training data pair being composed of a narrowband audio signal sample and a corresponding real wideband audio sample;

[0156] a first training preprocessing unit, configured to pre-process the narrowband audio signal sample to obtain a log-domain narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum for training.

[0157] a second training pre-processing unit, configured to construct a white noise amplitude spectrum for training;

[0158] a training framework construction unit, configured to construct a training framework comprising a generator and a discriminator, the generator being a speech bandwidth extension model to be trained, and the discriminator being configured to distinguish between a generated wideband audio sample and a real wideband audio sample;

[0159] a training unit, configured to input the log-domain narrowband audio signal amplitude spectrum, the log-domain mixed amplitude spectrum and the white noise amplitude spectrum for training into the generator to obtain a predicted training amplitude spectrum;

[0160] a second inverse short-time Fourier transform unit, configured to perform inverse short-time Fourier transform on the predicted training amplitude spectrum and phase information of the real wideband audio sample to generate a wideband audio sample and output the wideband audio sample;

[0161] a loss calculation unit, configured to calculate a training loss between the wideband audio sample and the real wideband audio sample;

[0162] a parameter optimization unit, configured to optimize parameters of the generator and the discriminator based on the training loss until the loss converges, to obtain a trained speech bandwidth extension model.

[0163] Based on the above-mentioned embodiments, the application further provides a terminal, a structure diagram of which can be shown in Figure 5 The terminal comprises a processor, a memory, a network interface and a display screen connected through a device bus. The processor of the terminal is configured to provide computing and control capabilities. The memory of the terminal comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating device and a lightweight speech frequency band extension program for edge devices. The internal memory provides an environment for the operation of the operating device and the lightweight speech frequency band extension program for edge devices in the non-volatile storage medium. The network interface of the terminal is configured to communicate with external terminals through network connection. The lightweight speech frequency band extension program for edge devices, when executed by the processor, implements the steps of any of the lightweight speech frequency band extension methods for edge devices. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0164] Those skilled in the art can understand that Figure 5 The structure diagram shown in the above-mentioned embodiments is only a schematic diagram of part of the structure related to the application scheme, and does not constitute a limitation on the terminal to which the application scheme is applied. Specifically, the terminal can comprise more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0165] In one embodiment, a terminal is provided, which comprises a memory, a processor, and an edge device-oriented lightweight voiceband expansion program stored in the memory and executable on the processor, which, when executed by the processor, implements any of the steps of the edge device-oriented lightweight voiceband expansion method provided by the embodiments of the present application.

[0166] The embodiments of the present application also provide a computer readable storage medium, which stores an edge device-oriented lightweight voiceband expansion program, which, when executed by a processor, implements any of the steps of the edge device-oriented lightweight voiceband expansion method provided by the embodiments of the present application.

[0167] It should be understood that the sequence of the steps in the above embodiments does not mean the order of execution, and the execution order of the processes should be determined according to their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0168] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or software. In addition, the specific names of the functional units and modules are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the device can be referred to the corresponding process in the foregoing method embodiments, which will not be described here.

[0169] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0170] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be realized in electronic hardware, or in a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0171] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / terminal device and method can be implemented in other manners. For example, the embodiments of the apparatus / terminal device described above are merely schematic; for example, the division of the modules or units is only a logical function division; there can be another division manner in actual implementation; for example, a plurality of units or components can be combined or integrated into another apparatus, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings between the units, or the coactions relationship between the units, can be implemented by electrical method, mechanical method or other forms.

[0172] The above embodiments are merely used to describe the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that; the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent; and these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the corresponding embodiments, and should be included in the protection scope of the present application.

Claims

1. An edge device-oriented lightweight voice band expansion method, characterized by, The method comprises: obtaining a narrowband audio signal from an edge device and preprocessing to obtain a log-domain narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum; constructing a white noise amplitude spectrum; inputting the log-domain narrowband audio signal amplitude spectrum, the log-domain mixed amplitude spectrum and the white noise amplitude spectrum into a trained voice bandwidth expansion model, using the voice bandwidth expansion model to respectively generate a first high-frequency component corresponding to a consonant in the narrowband audio signal and a second high-frequency component corresponding to a vowel in the narrowband audio signal, and obtaining a predicted amplitude spectrum based on the first high-frequency component and the second high-frequency component; extending the phase information of the narrowband audio signal, generating a wideband audio signal according to the predicted amplitude spectrum and the extended phase information of the narrowband audio signal, and outputting the wideband audio signal; obtaining a narrowband audio signal from an edge device and preprocessing to obtain a log-domain narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum, comprising: obtaining a narrowband audio signal from an edge device; upsampling the sampling rate of the narrowband audio signal to a preset target sampling rate to obtain an upsampled time-domain waveform; performing a half-wave rectification operation on the upsampled time-domain waveform to obtain a rectified time-domain waveform; performing short-time Fourier transform on the time-domain waveform and the rectified time-domain waveform using a Hanning window respectively to obtain a narrowband audio signal amplitude spectrum and a rectified narrowband audio signal amplitude spectrum; superimposing the narrowband audio signal amplitude spectrum and the rectified narrowband audio signal amplitude spectrum to obtain a mixed amplitude spectrum; applying a logarithmic transformation to the narrowband audio signal amplitude spectrum and the mixed amplitude spectrum respectively to obtain a log-domain narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum.

2. The edge device-oriented lightweight voice band expansion method according to claim 1, characterized in that, The voice bandwidth expansion model comprises a feature extraction module, a double-weighted Gaussian mixture module, a fusion module and a frequency band guided masking module.

3. The edge device-oriented lightweight voice band expansion method according to claim 2, characterized in that, inputting the log-domain narrowband audio signal amplitude spectrum, the log-domain mixed amplitude spectrum and the white noise amplitude spectrum into a trained voice bandwidth expansion model, using the voice bandwidth expansion model to respectively generate a first high-frequency component corresponding to a consonant in the narrowband audio signal and a second high-frequency component corresponding to a vowel in the narrowband audio signal, and obtaining a predicted amplitude spectrum based on the first high-frequency component and the second high-frequency component, comprising: inputting the log-domain mixed amplitude spectrum into the feature extraction module to obtain time sequence features representing the characteristics of the narrowband audio signal after processing; inputting the time sequence features, the log-domain mixed amplitude spectrum and the white noise amplitude spectrum into the double-weighted Gaussian mixture module to generate two sets of control parameters based on the time sequence features, respectively driving the first weighted Gaussian mixture module and the second weighted Gaussian mixture module of the double-weighted Gaussian mixture module to operate to correspondingly generate the first high-frequency component and the second high-frequency component; inputting the first high-frequency component and the second high-frequency component into the fusion module for fusion to output an initial predicted amplitude spectrum; inputting the log-domain narrowband audio signal amplitude spectrum and the initial predicted amplitude spectrum into the frequency band guided masking module for optimization to obtain a predicted amplitude spectrum.

4. The edge device-oriented lightweight voice band expansion method according to claim 3, characterized in that, The first high frequency component and the second high frequency component are input into the fusion module for fusion, and an initial prediction amplitude spectrum is output, including: The first high frequency component and the second high frequency component are input into the fusion module, and the first high frequency component and the second high frequency component are spliced by the fusion module to obtain spliced features; The spliced features are processed to generate weight coefficients; Based on the weight coefficients, the first high frequency component and the second high frequency component are weighted to obtain an initial prediction amplitude spectrum.

5. The edge device-oriented lightweight voice band expansion method according to claim 1, characterized in that, The phase information of the narrowband audio signal is expanded, and based on the predicted amplitude spectrum and the expanded phase information of the narrowband audio signal, a wideband audio signal is generated and output, including: The phase information of the narrowband audio signal is flipped and expanded; The predicted amplitude spectrum and the expanded phase information of the narrowband audio signal are combined, and inverse short-time Fourier transform is performed using a Hanning window to generate a wideband audio signal; The wideband audio signal is output.

6. The edge device-oriented lightweight voice band expansion method according to claim 1, characterized in that, The training steps of the speech bandwidth expansion model include: Obtain a training data pair, which is composed of a narrowband audio signal sample and its corresponding real wideband audio sample; Preprocess the narrowband audio signal sample to obtain a training log-domain narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum; Construct a training white noise amplitude spectrum; Construct a training framework including a generator and a discriminator, the generator being the speech bandwidth expansion model to be trained, and the discriminator being used to distinguish between generated wideband audio samples and real wideband audio samples; Input the training log-domain narrowband audio signal amplitude spectrum, the log-domain mixed amplitude spectrum, and the white noise amplitude spectrum into the generator to obtain a predicted training amplitude spectrum; Perform inverse short-time Fourier transform on the predicted training amplitude spectrum and the phase information of the real wideband audio sample to generate and output a wideband audio sample; Calculate the training loss between the wideband audio sample and the real wideband audio sample; Optimize the parameters of the generator and the discriminator based on the training loss until the loss converges, obtaining a trained speech bandwidth expansion model.

7. A lightweight voice band extension device for edge devices, characterized in that, including: A first preprocessing module is configured to obtain a narrowband audio signal from an edge device and perform preprocessing to obtain a log-domain narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum; Obtain a narrowband audio signal from an edge device and perform preprocessing to obtain a log-domain narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum, including: Obtain a narrowband audio signal from an edge device; Upsample the sampling rate of the narrowband audio signal to a preset target sampling rate to obtain an upsampled time-domain waveform; Perform a half-wave rectification operation on the upsampled time-domain waveform to obtain a rectified time-domain waveform; Perform short-time Fourier transform on the time-domain waveform and the rectified time-domain waveform using a Hanning window to obtain a narrowband audio signal amplitude spectrum and a rectified narrowband audio signal amplitude spectrum; Superimpose the narrowband audio signal amplitude spectrum and the rectified narrowband audio signal amplitude spectrum to obtain a mixed amplitude spectrum; Apply a logarithmic transformation to the narrowband audio signal amplitude spectrum and the mixed amplitude spectrum to obtain a log-domain narrowband audio signal amplitude spectrum and a log-domain mixed amplitude spectrum; The second preprocessing module is configured to construct a white noise amplitude spectrum. The amplitude spectrum prediction module is configured to input the log-domain narrowband audio signal amplitude spectrum, the log-domain mixed amplitude spectrum and the white noise amplitude spectrum into a trained voice bandwidth extension model, generate a first high-frequency component corresponding to a consonant in the narrowband audio signal and a second high-frequency component corresponding to a vowel in the narrowband audio signal by using the voice bandwidth extension model, and obtain a predicted amplitude spectrum based on the first high-frequency component and the second high-frequency component. The waveform reconstruction module is configured to expand phase information of the narrowband audio signal, generate a wideband audio signal according to the predicted amplitude spectrum and the expanded phase information of the narrowband audio signal, and output the wideband audio signal.

8. A terminal, characterized by comprising: The terminal comprises a memory, a processor and an edge device-oriented lightweight voice frequency band extension program stored on the memory and executable on the processor, and the edge device-oriented lightweight voice frequency band extension program, when executed by the processor, implements the steps of the edge device-oriented lightweight voice frequency band extension method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores an edge device-oriented lightweight voice frequency band extension program, and the edge device-oriented lightweight voice frequency band extension program, when executed by the processor, implements the steps of the edge device-oriented lightweight voice frequency band extension method according to any one of claims 1-6.

Images