Audio processing method and model training method, device, electronic equipment and product

Abstract

The present disclosure relates to an audio processing method, a model training method, an apparatus, an electronic device and a product. The method comprises: obtaining an amplitude spectrum of an audio to be processed; performing encoding processing on the amplitude spectrum by an audio processing model to obtain a first feature vector; performing quantization processing on the first feature vector by the audio processing model to obtain a second feature vector; and performing decoding processing on the second feature vector by the audio processing model to obtain a first complex spectrum; wherein the audio quality of a target audio generated based on the first complex spectrum is higher than the audio quality of the audio to be processed. Through the method of the present disclosure, a target audio with higher audio quality, which is clearer and more natural than the audio to be processed, can be obtained.

Description

Technical Field

[0001] This disclosure relates to the field of audio processing technology, and in particular to an audio processing method, model training method, apparatus, electronic device and product. Background Technology

[0002] Currently, to improve the quality of acquired audio, the bit rate is typically increased during audio acquisition. However, storing high-bitrate audio consumes significant storage resources, and transmitting high-bitrate audio not only occupies network resources but also reduces transmission speed. Currently, before storing or transmitting audio, the original audio can be compressed to reduce the bit rate. When the audio is needed, the compressed audio is then restored, reducing the storage space required for audio storage and the amount of data transmitted. Therefore, there is an urgent need for a method that can restore compressed audio with good sound quality. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides an audio processing method, model training method, apparatus, electronic device and product, which can obtain target audio that is clearer, more natural and of higher audio quality than the audio to be processed.

[0004] According to a first aspect of the present disclosure, an audio processing method is provided, the method comprising: acquiring an amplitude spectrum of an audio to be processed; encoding the amplitude spectrum using an audio processing model to obtain a first feature vector; quantizing the first feature vector using the audio processing model to obtain a second feature vector; and decoding the second feature vector using the audio processing model to obtain a first complex spectrum; wherein the audio quality of the target audio generated based on the first complex spectrum is higher than the audio quality of the audio to be processed.

[0005] In some embodiments, the audio processing model includes a vector quantization module; quantizing a first feature vector through the audio processing model to obtain a second feature vector includes: performing residual vector quantization on the first feature vector through the vector quantization module to obtain the second feature vector.

[0006] In some embodiments, the vector quantization module includes N discrete vector sets, where N is an integer greater than 1; wherein, the vector quantization module performs residual vector quantization processing on the first feature vector to obtain the second feature vector, including: determining the i-th target vector that is close to the i-th input vector from the i-th discrete vector set using an approximate nearest neighbor search algorithm; calculating the residual between the i-th input vector and the i-th target vector to obtain the i-th residual; wherein i is an integer from 1 to N, when i is 1, the i-th input vector is the first feature vector, and when i is greater than 1, the i-th input vector is the (i-1)-th residual; and determining the second feature vector based on the N residuals.

[0007] In some embodiments, the audio processing model further includes an encoding module and a Long Short-Term Memory (LSTM) module; wherein, encoding the amplitude spectrum through the audio processing model to obtain a first feature vector includes: extracting features from the amplitude spectrum through the encoding module to obtain a third feature vector; and processing the third feature vector in the time dimension through an LSTM module to obtain the first feature vector.

[0008] In some embodiments, the audio processing model includes M encoding modules and M Long Short-Term Memory (LSTM) modules, where M is an integer greater than 1. The audio processing model encodes the amplitude spectrum to obtain a first feature vector, including: using the j-th encoding module to extract features from the amplitude spectrum or the (j-1)-th fourth feature vector to obtain the j-th fifth feature vector; where j is an integer from 1 to M; where when j is 1, the j-th encoding module is used to extract features from the amplitude spectrum; when j is greater than 1, the j-th encoding module is used to extract features from the (j-1)-th fourth feature vector; and using the j-th LSTM module to process the j-th fifth feature vector in the time dimension to obtain the j-th fourth feature vector; where the M-th fourth feature vector is the first feature vector.

[0009] In some embodiments, the bit rate of the audio to be processed is less than or equal to a first threshold.

[0010] According to a second aspect of the present disclosure, a model training method is provided, the method comprising: acquiring a first sample audio and a second sample audio, wherein the second sample audio is audio after compression of the first sample audio; inputting the amplitude spectrum of the second sample audio into an audio processing model to be trained to obtain a second complex spectrum; generating a third sample audio based on the second complex spectrum; calculating a loss value based on the difference between the first sample audio and the third sample audio; and iteratively training the audio processing model to be trained based on the loss value to obtain a trained audio processing model.

[0011] In some embodiments, a loss value is calculated based on the difference between the third sample audio and the first sample audio, including at least one of the following: calculating a loss value based on the phase difference between the third sample audio and the first sample audio; determining a loss value based on the temporal loss between the third sample audio and the first sample audio; and determining a loss value based on the Fourier loss between the third sample audio and the first sample audio.

[0012] According to a third aspect of the present disclosure, an audio processing apparatus is provided, the apparatus comprising: a first acquisition module configured to acquire an amplitude spectrum of an audio to be processed; an extraction module configured to encode the amplitude spectrum using an audio processing model to obtain a first feature vector; a first processing module configured to quantize the first feature vector using the audio processing model to obtain a second feature vector; and a second processing module configured to decode the second feature vector using the audio processing model to obtain a first complex spectrum; wherein the audio quality of the target audio generated based on the first complex spectrum is higher than the audio quality of the audio to be processed.

[0013] In some embodiments, the audio processing model includes a vector quantization module; a first processing module is configured to perform residual vector quantization on a first feature vector through the vector quantization module to obtain a second feature vector.

[0014] In some embodiments, the vector quantization module includes N discrete vector sets, where N is an integer greater than 1; wherein, the first processing module is configured to: determine the i-th target vector that is close to the i-th input vector from the i-th discrete vector set using an approximate nearest neighbor search algorithm; calculate the residual between the i-th input vector and the i-th target vector to obtain the i-th residual; wherein i is an integer from 1 to N, when i is 1, the i-th input vector is the first feature vector, and when i is greater than 1, the i-th input vector is the (i-1)-th residual; and determine the second feature vector based on the N residuals.

[0015] In some embodiments, the audio processing model further includes an encoding module and a Long Short-Term Memory (LSTM) module; wherein the extraction module is configured to: extract features from the amplitude spectrum through the encoding module to obtain a third feature vector; and process the third feature vector in the time dimension through an LSTM module to obtain a first feature vector.

[0016] In some embodiments, the audio processing model includes M encoding modules and M Long Short-Term Memory (LSTM) modules, where M is an integer greater than 1; wherein, the extraction module is configured to: extract features from the amplitude spectrum or the (j-1)th fourth feature vector through the j-th encoding module to obtain the j-th fifth feature vector; wherein j is an integer from 1 to M; wherein, when j is 1, the j-th encoding module is used to extract features from the amplitude spectrum; when j is greater than 1, the j-th encoding module is used to extract features from the (j-1)th fourth feature vector; and process the j-th fifth feature vector in the time dimension through the j-th LSTM module to obtain the j-th fourth feature vector; wherein, the M-th fourth feature vector is the first feature vector.

[0017] In some embodiments, the bit rate of the audio to be processed is less than or equal to a first threshold.

[0018] According to a fourth aspect of the present disclosure, a model training apparatus is provided, the apparatus comprising: a second acquisition module configured to acquire a first sample audio and a second sample audio, wherein the second sample audio is audio after compression of the first sample audio; an input module configured to input the amplitude spectrum of the second sample audio into an audio processing model to be trained to obtain a second complex spectrum; a generation module configured to generate a third sample audio based on the second complex spectrum; a calculation module configured to calculate a loss value based on the difference between the first sample audio and the third sample audio; and a training module configured to iteratively train the audio processing model to be trained based on the loss value to obtain a trained audio processing model.

[0019] In some embodiments, the calculation module is configured to: calculate a loss value based on the phase difference between the third sample audio and the first sample audio; determine a loss value based on the temporal loss between the third sample audio and the first sample audio; and determine a loss value based on the Fourier loss between the third sample audio and the first sample audio.

[0020] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing computer programs or instructions; wherein the processor executes the computer programs or instructions to implement the steps of the methods described in the first and / or second aspects above.

[0021] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, storing executable instructions or a computer program, which, when executed by a processor, implements the steps of the methods described in the first and / or second aspects.

[0022] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the methods described in the first and / or second aspects.

[0023] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0024] In this disclosure, after obtaining the amplitude spectrum of the audio to be processed, the amplitude spectrum can be input into an audio processing model for feature extraction. The extracted first feature vector is then quantized to obtain a discretized second feature vector. Quantization of the first feature vector allows the resulting discrete second feature vector to contain more harmonic textures, spectral features, and temporal characteristics of the audio to be processed. Based on this, decoding the discrete second feature vector yields a target audio that is clearer, more natural, and of higher quality than the original audio.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0027] Figure 1 This is a flowchart illustrating an audio processing method according to an exemplary embodiment.

[0028] Figure 2 This is a schematic diagram illustrating the structure of an audio processing model according to an exemplary embodiment.

[0029] Figure 3 This is a flowchart illustrating an audio processing method according to another exemplary embodiment.

[0030] Figure 4 This is a flowchart illustrating a method for obtaining a second feature vector according to an exemplary embodiment.

[0031] Figure 5 This is a flowchart illustrating a model training method according to an exemplary embodiment.

[0032] Figure 6 This is a block diagram illustrating an audio processing apparatus according to an exemplary embodiment.

[0033] Figure 7 This is a block diagram illustrating a model training apparatus according to an exemplary embodiment.

[0034] Figure 8 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0035] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0036] Currently, to improve the quality of acquired audio, the bit rate is typically increased during audio acquisition. However, storing high-bitrate audio consumes significant storage resources, and transmitting high-bitrate audio not only occupies network resources but also reduces transmission speed. Currently, before storing or transmitting audio, the original audio can be compressed to reduce the bit rate. When the audio is needed, the compressed audio is then restored, reducing the storage space required for audio storage and the amount of data transmitted. Therefore, there is an urgent need for a method that can restore compressed audio with good sound quality.

[0037] To address the aforementioned problems, this disclosure provides an audio processing method. This method can be executed by an electronic device. Exemplarily, the electronic device executing the audio processing method provided in this disclosure can be a mobile terminal or a fixed terminal. The mobile terminal may include smartphones, tablets, laptops, digital cameras, and wearable electronic devices, etc. The fixed terminal may include desktop computers, all-in-one computers, etc.

[0038] Figure 1 This is a schematic flowchart illustrating an audio processing method according to an exemplary embodiment. In the following text, an electronic device will be used as the execution subject, with reference to... Figure 1 The steps shown illustrate the audio processing method provided in this disclosure. This audio processing method mainly includes the following steps:

[0039] In step 101, the amplitude spectrum of the audio to be processed is obtained.

[0040] Understandably, after receiving an audio file to be processed, an electronic device can obtain the amplitude spectrum of that audio file.

[0041] In some embodiments, the audio to be processed can be compressed audio, compressed but damaged audio, etc.

[0042] For example, the audio to be processed acquired by the electronic device can come from any of the following scenarios: compressed audio acquired and received in a video call scenario, compressed audio acquired and received in a voice call scenario, compressed audio received from other devices, compressed audio downloaded from the network, compressed audio acquired by the audio acquisition component on the electronic device, etc.

[0043] In some embodiments, voice call scenarios include, but are not limited to, cellular network call scenarios and satellite network call scenarios.

[0044] In some embodiments, the audio to be processed can be audio with a time-domain signal sampling rate less than a second threshold. For example, the second threshold can be any value between 7 kHz and 10 kHz. For example, the second threshold can be 7 kHz, 8 kHz, 9 kHz, 10 kHz, etc.

[0045] In some embodiments, after receiving the audio to be processed, the electronic device may perform a Fourier transform on the audio to be processed to obtain the amplitude spectrum of the audio to be processed.

[0046] For example, the Fourier transform includes the short-time Fourier transform.

[0047] In some embodiments, the electronic device can segment the audio to be processed into multiple audio frames based on the window function used when performing a short-time Fourier transform on the audio to be processed. Then, the electronic device can perform an inverse Fourier transform on each audio frame to obtain the amplitude spectrum of each audio frame. Based on this, the electronic device can obtain multiple amplitude spectra of the audio to be processed.

[0048] In some embodiments, the type of window function, the length of the window function, the number of sampling points of the Fourier transform, the window shift size, etc., used by the electronic device when performing short-time Fourier transform on the audio to be processed can be set as needed, and the embodiments disclosed herein do not limit this.

[0049] For example, window functions used to perform short-time Fourier transform on the audio to be processed include, but are not limited to: Hanning window, rectangular window, Hamming window, etc.

[0050] For example, the length of the window function used to perform the short-time Fourier transform on the audio to be processed can be any value from 100 to 500 sampling points. For example, the length of the window function can be 100 sampling points, 200 sampling points, 250 sampling points, 300 sampling points, 320 sampling points, 400 sampling points, 450 sampling points, 500 sampling points, etc.

[0051] In some embodiments, the number of sampling points for the Fourier transform used to perform a short-time Fourier transform on the audio to be processed can be set according to the length of the window function used. For example, if the length of the window function is 320 sampling points, the number of sampling points for the Fourier transform can also be 320 sampling points.

[0052] For example, the window shift during the short-time Fourier transform of the audio to be processed can be determined based on the length of the window function. For example, the window shift can be 1 / 2 or 1 / 3 of the window function. For instance, if the window shift is 1 / 2 of the window function and the window function length is 320 samples, the window shift could be 160 samples.

[0053] In step 102: The amplitude spectrum is encoded using an audio processing model to obtain the first feature vector.

[0054] Understandably, after obtaining the amplitude spectrum of the audio to be processed through step 102 above, the electronic device can input the amplitude spectrum of the audio to be processed into the audio processing model. Based on this, the electronic device can encode the amplitude spectrum through the audio processing model to obtain the first feature vector.

[0055] In some embodiments, the audio processing model may include, but is not limited to, a trained convolutional neural network model, a deep neural network model, etc.

[0056] For example, audio processing models may include: models based on VQ-VAE (Vector Quantized-Variational AutoEncoder) framework, network models based on Discrete Variational Autoencoder (dVAE) model, etc.

[0057] In some embodiments, the first feature vector can be a feature vector of the audio to be processed in the latent space.

[0058] For example, the latent space can be understood as encoding or mapping the amplitude spectrum to a low-dimensional space to represent the latent features and intrinsic structure of the amplitude spectrum.

[0059] In some embodiments, the first feature vector may be a continuous vector.

[0060] In some embodiments, since there are multiple amplitude spectra, the electronic device can sequentially input multiple amplitude spectra into the audio processing model according to their temporal order to obtain a first feature vector corresponding to each amplitude spectrum, thereby obtaining multiple first feature vectors.

[0061] In step 103, the first feature vector is quantized using an audio processing model to obtain the second feature vector.

[0062] Understandably, after obtaining the first feature vector through step 102 above, the electronic device can further quantize the first feature vector through an audio processing model to obtain a discrete second feature vector.

[0063] In some embodiments, the second feature vector may be an embedding vector.

[0064] In some embodiments, the algorithms used by the audio processing model to quantize the first feature vector include, but are not limited to: Residual Vector Quantization (RVQ), Additive Vector Quantization (AVQ), Product Vector Quantization (PVQ), etc.

[0065] In some embodiments, when there are multiple first feature vectors, the electronic device can sequentially quantize each of the multiple first feature vectors using an audio processing model according to the order in which the multiple first feature vectors are acquired, so as to obtain multiple second feature vectors.

[0066] In step 104, the second feature vector is decoded using an audio processing model to obtain the first complex spectrum.

[0067] Understandably, after obtaining the discrete second eigenvector based on step 103 above, the electronic device can continue to decode the second eigenvector using an audio processing model. At this point, the electronic device can obtain the first complex spectrum.

[0068] In some embodiments, the electronic device may perform an inverse Fourier transform on the first complex spectrum to obtain the target audio.

[0069] In some embodiments, the inverse transformation of the Fourier transform includes the inverse transformation of the short-time Fourier transform.

[0070] In some embodiments, the electronic device can decode the second eigenvectors corresponding to multiple amplitude spectra using the decoding module included in the audio processing model to obtain multiple first complex spectra. Then, by performing an inverse short-time Fourier transform on the multiple first complex spectra, the target audio can be obtained.

[0071] In some embodiments, the parameters such as the window function and the length of the window function used by the electronic device when performing the inverse Fourier transform on the first complex spectrum can be set according to the parameters used when obtaining the amplitude spectrum of the audio to be processed through the short-time Fourier transform. This disclosure does not limit this.

[0072] In some embodiments, the target audio has a higher audio quality than the audio to be processed.

[0073] For example, the audio quality of the target audio is higher than that of the audio to be processed, including but not limited to one of the following: the clarity of the target audio is higher than that of the audio to be processed; the signal-to-noise ratio of the target audio is greater than that of the audio to be processed; the total harmonic distortion (THD) of the target audio is less than that of the audio to be processed; the spectral flatness (THD) of the target audio is less than that of the audio to be processed.

[0074] For example, the decoding module included in the audio processing model can be a decoder.

[0075] In some embodiments, the decoding module may include a decoding module and a Long Short-Term Memory (LSTM) module. Based on this, an LSTM module processes the second eigenvector in the time dimension, and a decoding module decodes the processed second eigenvector to obtain the first complex spectrum.

[0076] In some embodiments, the decoding module may include M decoding modules and M Long Short-Term Memory (LSTM) modules, where M is an integer greater than 1. The M decoding modules and M LSTM modules are spaced apart. Based on this, the output of the j-th second time-series unit can be used as the input of the j-th decoding unit, and the output of the j-th decoding unit can be used as the input of the (j+1)-th second time-series unit, where j can be an integer from 1 to M-1. The input of the first STM module is the second feature vector, and the output of the M-th decoding unit is the first complex spectrum.

[0077] In some embodiments, each STM module may employ an LSTM network, a gated recurrent unit (GRU), or the like.

[0078] In some embodiments, each decoding module may include deconvolution layers, batch normalization layers, and activation function layers. For example, the activation functions used in the activation function layers include, but are not limited to, the Rectified Linear Unit (ReLU) function and the Leaky Rectified Linear Unit (LeakyReLU) function.

[0079] In some embodiments, reference Figure 2 , Figure 2 This is a schematic diagram illustrating the structure of an audio processing model according to an exemplary embodiment. For example... Figure 2 As shown, the audio processing model may include an encoder 201, a quantizer (vector quantization module) 202, and a decoder 203. The decoder 203 includes three STM modules and three deconvolutional layers (…). Figure 2 TrConv2d in the middle), batch normalization layer ( Figure 2 The decoding module consists of the BN1 layer and the activation function layer (ReLU1). Based on this, the electronic device can use the first STM module ( Figure 2 The first LSTM module, the first decoding module, and the second STM module (in the original text) Figure 2 The LSTM2 module, the second decoding module, and the third STM module (in the image) Figure 2 The LSTM3 in the third decoding module and the second feature vector output by the quantizer 202 are processed sequentially. Figure 2 The zq in the image is processed to obtain the complex spectrum of the target audio. Figure 2 (y in the middle).

[0080] In some embodiments, the channel input / output configurations of the deconvolution layers included in the first, second, and third decoding modules are (1024, 512), (512, 128), and (128, 1), respectively. Specifically, the first decoding module has 1024 input channels and 512 output channels; the second decoding module has 512 input channels and 128 output channels; and the third decoding module has 128 input channels and 1 output channel.

[0081] In this embodiment, after obtaining the amplitude spectrum of the audio to be processed, the amplitude spectrum can be input into an audio processing model for feature extraction. The extracted first feature vector is then quantized to obtain a discretized second feature vector. Quantization of the first feature vector allows the resulting discrete second feature vector to contain more harmonic textures, spectral features, and temporal characteristics of the audio to be processed. Based on this, decoding the discrete second feature vector yields a target audio that is clearer, more natural, and of higher quality than the original audio.

[0082] In some embodiments, the audio processing model includes a vector quantization module. Step 103 includes: performing residual vector quantization on the first feature vector using the vector quantization module to obtain a second feature vector.

[0083] Understandably, an audio processing model can include a vector quantization module. Based on this, after obtaining a first feature vector, the electronic device can input the first feature vector into the vector quantization module of the audio processing model to perform residual vector quantization processing on the first feature vector. At this point, the electronic device can obtain a discretized second feature vector.

[0084] In some embodiments, the quantization algorithm used by the vector quantization module can be a residual vector quantization algorithm. Based on this, after the electronic device extracts the first feature vector, it can also perform residual vector quantization processing on the first feature vector using a residual vector quantization algorithm to obtain a discrete second feature vector.

[0085] For example, refer to Figure 3 , Figure 3 This is a flowchart illustrating an audio processing method according to another exemplary embodiment. For example... Figure 3As shown, after obtaining a damaged audio segment, the electronic device can use this damaged audio as the audio to be processed 10 and perform a short-time Fourier transform on the audio to be processed 10 to obtain the amplitude spectrum 101 of the audio to be processed 10. Then, the electronic device can input the amplitude spectrum 101 of the audio to be processed 10 into the audio processing model 20. Based on this, the encoder 201 included in the audio processing model 20 can encode the amplitude spectrum 101 to encode it into a low-dimensional continuous latent space feature (first feature vector). Then, the electronic device can input the continuous latent space feature output by the encoder 201 into the quantizer 202 to discretize the continuous latent space feature into the embedding space, thereby obtaining a discretized second feature vector. After obtaining the second feature vector, the electronic device can input the second feature vector into the decoder 203 to decode the second feature vector, thereby obtaining a first complex spectrum 102. Then, by performing an inverse short-time Fourier transform on the first complex spectrum, the target audio 30 can be obtained.

[0086] In some embodiments, when the vector quantization module uses a residual vector quantization algorithm, the electronic device can set the order and convergence conditions of the residual vector quantization algorithm as needed. The convergence condition can be understood as the condition that must be met to stop residual calculation before the set order is reached.

[0087] For example, an electronic device may set the order of the residual vector quantization algorithm to 6.

[0088] In this embodiment of the disclosure, firstly, the electronic device encodes the amplitude spectrum of the audio to be processed into the latent space through the encoder in the audio processing model, which can achieve compression of the amplitude spectrum, thereby reducing the amount of data that the audio processing model needs to process and improving the processing efficiency of the audio to be processed.

[0089] Secondly, the electronic device discretizes the continuous latent space features (first feature vector) output by the encoding module into the embedding space through the vector quantization module, which can enhance the audio processing model's ability to express details in the audio to be processed, and also improve the operational stability of the audio processing model.

[0090] Furthermore, the electronic device performs residual vector quantization on the first feature vector through a vector quantization module, enabling multi-level quantization of the first feature vector. This improves the quantization accuracy of the first feature vector, allowing for more refined modeling of audio details and enhancing the audio quality of the target audio. Additionally, since the discretized second feature vector is sparser than the continuous first feature vector, the model can maintain higher generalization ability during operation and is less prone to overfitting.

[0091] Finally, the audio processing model used in this embodiment is an end-to-end encoder-decoder structure. The model has a small number of parameters (about 5M parameters), which can be deployed in terminal devices with poor operating capabilities, thereby improving the application scope of the audio processing method provided in this embodiment.

[0092] In some embodiments, the vector quantization module includes N discrete vector sets, where N is an integer greater than 1. The vector quantization module performs residual vector quantization on the first feature vector to obtain the second feature vector, including: determining a target vector similar to the i-th input vector from the i-th discrete vector set using an approximate nearest neighbor search algorithm; calculating the residual between the i-th output vector and the i-th target vector to obtain the i-th residual, where i is an integer from 1 to N; when i is 1, the i-th input vector is the first feature vector; when i is greater than 1, the i-th input vector is the (i-1)-th residual; and determining the second feature vector based on the N residuals.

[0093] Understandable, for reference Figure 4 The vector quantization module can include N discrete vector sets, where N is an integer greater than 1. Furthermore, each of the N discrete vector sets (codebooks) includes multiple discrete feature vectors. Based on this, after obtaining the first feature vector z, the electronic device can use an Approximate Nearest Neighbor Search (ANNS) algorithm to select from the first discrete vector set (…). Figure 4 From the discrete vector set 1), determine the vector that is similar to the first eigenvector z, and take the vector determined from the first discrete vector set as the first target vector z. 11 The electronic device obtains the first target vector z. 11 Then, the first feature vector z and the first target vector z can be calculated. 11 The residuals between are used to obtain the first residual z1. Then, the electronic device can use the first residual z1 as the second input vector and, through an approximate nearest neighbor search algorithm, find the second discrete vector set (...). Figure 4 From the discrete vector set 2), determine the second target vector z that is similar to the first residual. 22 Then, the electronic device can calculate the first residual z1 and the second target vector z. 22 The residuals between them are used to obtain the second residual z2. And so on, the electronic device can also obtain the second residual z2 from the Nth discrete vector set ( Figure 4 From the discrete vector set N, determine the (N-1)th target vector z that is closest to the (N-1)th residual. N-1N-1 Then, the electronic device can calculate the (N-1)th residual z. N-1 and the (N-1)th target vector zN-1N-1 The residuals between them are used to obtain the Nth residual z. N After obtaining N residuals, the electronic device can calculate the second eigenvector based on the N residuals.

[0094] In some embodiments, the size of N can be set as needed, and this disclosure does not limit this.

[0095] For example, N can be any integer from 5 to 10. For instance, N can be 5, 6, 8, 10, etc.

[0096] In some embodiments, the approximate nearest neighbor search algorithm includes, but is not limited to, Locality Sensitive Hashing (LSH) algorithm, Graph-based Methods (GbM), etc.

[0097] In some embodiments, the electronic device can also determine whether the vector quantization module meets the convergence condition based on the latest obtained residual after obtaining each residual. If the electronic device determines that the vector quantization module meets the convergence condition, it can stop calculating the residual and calculate the second eigenvector based on the obtained residual.

[0098] For example, the electronic device can calculate the square of the latest obtained residual, and then compare the square of the latest obtained residual with the early stopping error parameter. If the electronic device determines that the square of the latest obtained residual is less than the early stopping error parameter, it can determine that the vector quantization module satisfies the convergence condition.

[0099] In some embodiments, the electronic device may calculate the difference between the i-th input vector and the i-th target vector, and use the obtained difference as the i-th residual.

[0100] In some embodiments, the electronic device may use the sum of all obtained residuals as a second feature vector. For example, the electronic device may use the sum of N obtained residuals as a second feature vector.

[0101] In this embodiment of the disclosure, the electronic device uses an approximate nearest neighbor search algorithm to determine a target vector that is close to the i-th input vector from the discrete vector set corresponding to the i-th quantization unit. Compared with the method of using an exposure search algorithm to determine a target vector that is close to the i-th input vector from the discrete vector set corresponding to the i-th quantization unit, the search amount can be reduced and the vector quantization speed of the first feature vector can be accelerated.

[0102] In some embodiments, the encoding module includes an encoding module and an LSTM module. Step 102 includes: extracting features from the amplitude spectrum using the encoding module to obtain a third feature vector; and processing the third feature vector in the time dimension using the LSTM module to obtain a first feature vector.

[0103] Understandably, audio processing models can also include encoding modules and LSTM modules. Based on this, after the electronic device inputs the amplitude spectrum of the audio to be processed into the audio processing model, it can extract features from the amplitude spectrum through the encoding module to obtain a third feature vector. Then, the electronic device can input the obtained third feature vector into the LSTM module, which processes the third feature vector in the time dimension to obtain the first feature vector.

[0104] In some embodiments, Figure 2 The encoder in the code includes an encoding module and an LSTM module.

[0105] In some embodiments, the encoding module may include, but is not limited to, convolutional layers, batch normalization layers, and activation function layers.

[0106] In some embodiments, the third feature vector can be a continuous feature of the latent space.

[0107] In some embodiments, since the audio to be processed has multiple amplitude spectra, the electronic device can sequentially extract features from each amplitude spectrum using an encoding module consisting of convolutional layers, batch normalization layers, and activation function layers, according to the acquisition time of the multiple amplitude spectra, to obtain a first feature vector. Then, the electronic device can use an LSTM module to process the first feature vector output by the encoding module to obtain a second feature vector.

[0108] In some embodiments, when the electronic device processes the first feature vector other than the first first feature vector through the LSTM module, it also refers to the dependency relationship between the current first feature vector and the previous first feature vector.

[0109] In some embodiments, when the LSTM module is a Long Short-Term Memory (LSTM) network, the number of hidden nodes used by the LSTM network can be set as needed. For example, the number of hidden nodes can be from 1000 to 1030. For instance, the number of hidden nodes can be 1000, 1010, 1020, 1024, 1030, etc.

[0110] In some embodiments, the audio processing model includes M encoding modules and M LSTM modules, where M is an integer greater than 1. The audio processing model encodes the amplitude spectrum to obtain a first feature vector, including: using the j-th encoding module to extract features from the amplitude spectrum or the (j-1)-th fourth feature vector to obtain the j-th fifth feature vector. Here, j is an integer from 1 to M; when j is 1, the j-th encoding module is used to extract features from the amplitude spectrum. When j is greater than 1, the j-th encoding module is used to extract features from the (j-1)-th fourth feature vector. The j-th LSTM module processes the j-th fifth feature vector in the time dimension to obtain the j-th fourth feature vector; where the M-th fourth feature vector is the first feature vector.

[0111] Understandably, the audio processing model can also include M encoding modules and M LSTM modules. Based on this, after inputting the amplitude spectrum of the audio to be processed into the audio processing model, the electronic device can extract features from the amplitude spectrum through the first encoding module to obtain the first fifth feature vector. Then, the electronic device can input the first fifth feature vector into the first LSTM module to process it in the time dimension to obtain the first fourth feature vector. After obtaining the first fourth feature vector output by the first LSTM module, the electronic device can extract features from it through the second encoding module to obtain the second fifth feature vector. Then, the electronic device can extract the second fifth feature vector in the time dimension through the second LSTM module. And so on. The electronic device can obtain the Mth fourth feature vector and M fifth feature vectors. At this point, the electronic device can use the Mth fourth feature vector as the first feature vector.

[0112] In some embodiments, the number of encoding modules and the number of LSTM modules included in the encoding module can be determined as needed, and this disclosure does not limit this.

[0113] For example, the number of encoding modules and the number of LSTM modules can both be integers such as 3, 4, 5, etc.

[0114] Continue to refer to Figure 2 ,like Figure 2 As shown, encoder 201 includes three LSTM modules, as well as convolutional layers ( Figure 2 Conv2d in the middle), batch normalization layer ( Figure 2 BN2) and activation function layer ( Figure 2 The encoder 201 consists of three decoding modules, namely ReLU2. Based on this, the electronic device can utilize the first encoding module, the first LSTM module (...) in the encoder 201... Figure 2The second LSTM module (LSTM4), the second encoding module, and the second LSTM module (LSTM4). Figure 2 The LSTM5, the third encoding module, and the third LSTM module (LSTM6) sequentially process the input amplitude spectrum. Figure 2 The x in the vector is processed to obtain the first feature vector. Figure 2 (z in the middle).

[0115] In some embodiments, the channel input / output configurations of the convolutional layers included in the first, second, and third encoding modules are (1, 128), (128, 512), and (512, 1024), respectively. Specifically, the first encoding module has 1 input channel and 128 output channels; the second encoding module has 128 input channels and 512 output channels; and the third encoding module has 512 input channels and 1024 output channels.

[0116] In this embodiment of the disclosure, by extracting multi-level features in depth through multiple encoding modules and multiple LSTM modules, the first feature vector can include more features of the audio to be processed, thereby improving the generation effect of the target audio.

[0117] In some embodiments, the bit rate of the audio to be processed is less than or equal to a first threshold.

[0118] It is understood that the bit rate of the audio processed by the method provided in the embodiments of this disclosure may be less than or equal to a first threshold.

[0119] In some embodiments, the first threshold can be any bit rate between 0.31kbps and 2.5kbps.

[0120] The following section describes a specific embodiment of the audio processing method provided in the above embodiments, as follows:

[0121] as follows Figure 2 As shown, the audio processing model can be an encoder and decoder structure based on the VQ-VAE framework, and combines a convolutional neural network (Conv2D) and an LSTM network.

[0122] The input x of the audio processing model is the amplitude spectrum of the audio to be processed with a bit rate lower than a first threshold. The domain signal sampling rate of the audio to be processed is 8000Hz. The amplitude spectrum of the audio to be processed is obtained by performing a Short Time Fourier Transform (SFT) on the audio. The configuration of the SFT is as follows: window length 320 points, number of Fourier points 320, window shift 160, and window function is Hanning window.

[0123] The main task of the encoder in the audio processing module is to compress the amplitude spectrum x of the audio to be processed into a low-dimensional latent space to obtain the first feature vector z. The encoder includes a module composed of Conv2d, BN2, and ReLU2, as well as an LSTM module. Conv2d represents the convolution operator, BN2 represents batch normalization, and ReLU2 is the activation function. The module composed of Conv2d, BN2, and ReLU2 means that the three are chained together to form an operator module, which is used to extract features from the amplitude spectrum x of the input audio. The LSTM module is used to model the temporal features, and the number of hidden nodes in the LSTM is 1024. The stacking of multiple modules composed of Conv2d, BN2, and ReLU2 and multiple LSTM modules can be hierarchical to extract multi-level features of the input at a deeper level. The module composed of Conv2d, BN2, and ReLU2 and the LSTM module form a Convolutional Recurrent Network (CRN), and the encoder consists of three such CRNs.

[0124] The quantizer is used to quantize the first feature vector z output by the encoder. It discretizes the first feature vector in the continuous space into a second feature vector. This step is crucial in audio processing models, providing discretization modeling capabilities. It captures discrete information in the amplitude spectrum of the audio to be processed at bit rates below a first threshold, reducing the influence of interference harmonics and allowing the audio processing model to focus more on important speech harmonic textures. The input to the quantizer is the encoded latent space representation (first feature vector), and the output is the latent space representation obtained by quantizing the first feature vector (second feature vector). In the vector quantization module of the VQ-VAE model, the quantization method is nearest neighbor search. This method lacks more refined modeling capabilities, while the cascaded quantization method of residual vector quantization significantly improves expressive power and greatly reduces the embedding space size. Therefore, in this embodiment, residual vector quantization is used to quantize the first feature vector, and an approximate nearest neighbor search algorithm is employed during the quantization process.

[0125] The decoder is the module that recovers the signal output. It receives the second eigenvector output by the quantizer and decodes it into a first complex spectrum. The first complex spectrum, after undergoing an inverse short-time Fourier transform, yields the target audio y. The configuration of the inverse short-time Fourier transform is the same as that of the short-time Fourier transform. The decoder is composed of multiple modules consisting of TrConv2d, BN1, and ReLU1, stacked with multiple LSTM modules. TrConv2d is the deconvolution operator, BN1 is batch normalization, and ReLU1 is the linear rectified activation function; the channel input / output configurations for the deconvolution are (1024, 512), (512, 128), and (128, 1).

[0126] The advantages of the audio processing model used in this embodiment are mainly as follows: First, the audio processing model is a single model without stages, making it simple to design and operate. Furthermore, it is a shallow neural network with low computational cost. Second, the audio processing model based on the VQ-VAE framework can effectively represent continuous spatial features. In particular, by employing residual vector quantization on the first feature vector, it can model the details of the audio to be processed more precisely. Simultaneously, the discretized second feature vector is sparser than the continuous feature vector representation, maintaining good generalization performance and reducing the likelihood of overfitting during the training process. Finally, the use of an approximate nearest neighbor search algorithm accelerates vector quantization, reduces the impact of autoregression, and decreases the embedding space size.

[0127] For example, in the case of satellite communication, the electronic device can perform a short-time Fourier transform on the received speech audio to obtain the amplitude spectrum x of the speech audio. Then, the electronic device can input the amplitude spectrum x into an audio processing model for processing. Specifically, the amplitude spectrum is encoded, residual vector quantized, and decoded to obtain a repaired first complex spectrum. Then, an inverse short-time Fourier transform is performed on the first complex spectrum to obtain the repaired target audio y.

[0128] In this embodiment of the disclosure, the electronic device processes the audio to be processed using the audio processing method provided in the above embodiments, which can improve the P808 score and / or MOS score of the target audio by about 0.5 relative to the audio to be processed.

[0129] This disclosure also provides a model training method. This model training method is used to train the audio processing model in the above embodiments. Figure 5 This is a schematic flowchart illustrating a model training method according to an exemplary embodiment. In the following text, it will be combined with... Figure 5 The steps shown illustrate the model training method provided in this disclosure. The method includes:

[0130] In step 501, the first sample audio and the second sample audio are obtained.

[0131] In some embodiments, the second sample audio is the compressed audio of the first sample audio.

[0132] In some embodiments, the first sample audio may be the original audio captured by an audio acquisition device. Alternatively, the first sample audio may be audio from the original audio that has undergone noise reduction processing.

[0133] In some embodiments, the first sample audio may also be audio obtained from the open-source dataset DNS Challenge2024. Each second sample audio has a sampling rate of 8kHz.

[0134] In some embodiments, the total duration of all first sample audio and all second sample audio can be 927 hours.

[0135] In some embodiments, the second sample audio may be audio that has been compressed from the first sample audio to a bit rate less than or equal to a first threshold.

[0136] In some embodiments, the electronic device may compress the first sample audio using an encoding algorithm in an encoder-decoder (Codec) algorithm to obtain a second sample audio with a bit rate lower than a first threshold.

[0137] In step 502, the amplitude spectrum of the second sample audio is input into the audio processing model to be trained to obtain the second complex spectrum.

[0138] Understandably, after obtaining the first and second sample audio samples through step 502, the electronic device can input the second sample audio sample into the audio processing model to be trained, so that the model can process the second sample audio sample. At this time, the electronic device can obtain the second complex spectrum output by the audio processing model to be trained.

[0139] It should be noted that the process of acquiring the second complex spectrum by the electronic device can refer to the above-described process of acquiring the first complex spectrum, and will not be repeated here in the embodiments of this disclosure.

[0140] In addition, the process of processing the second sample audio with the audio processing model to be trained to obtain the second complex spectrum can be referred to the above-described process of processing the audio to be processed with the audio processing model to obtain the first complex spectrum. The embodiments of this disclosure will not be repeated here.

[0141] In step 503, a third sample audio is generated based on the second complex spectrum.

[0142] In some embodiments, after obtaining the second complex spectrum, the electronic device can obtain the third sample audio by performing an inverse Fourier transform on the second complex spectrum.

[0143] In some embodiments, the configuration parameters such as window function, window length, and window shift when performing the inverse Fourier transform on the second complex spectrum can refer to the relevant configurations described above when performing the inverse Fourier transform on the first complex spectrum, and will not be repeated here.

[0144] In step 504, the loss value is calculated based on the difference between the first sample audio and the third sample audio.

[0145] Understandably, after obtaining the third sample audio, the electronic device can determine the loss value based on the difference between the third sample audio and the first sample audio.

[0146] In some embodiments, the electronic device may calculate a loss value based on a preset loss function, according to the difference between the first sample audio and the third sample audio.

[0147] In step 505, the audio processing model to be trained is iteratively trained based on the loss value to obtain the trained audio processing model.

[0148] Understandably, after obtaining the loss value, the electronic device can adjust the model parameters of the audio processing model to be trained based on the obtained loss value, and continue to iterate the training of the audio processing model until a preset number of iterations is reached or the audio processing model converges. At this point, the electronic device can obtain the audio processing model in the above embodiment.

[0149] In some embodiments, the electronic device may train the audio processing model to be trained using supervised learning and pseudo-regression.

[0150] In some embodiments, the electronic device may use the entire training set (first sample audio and second sample audio) to perform 1000 complete training iterations on the audio processing model to be trained.

[0151] In some embodiments, during the training of the audio processing model to be trained, the number of first sample audio inputs in each electronic device is 128.

[0152] In some embodiments, electronic devices may use an early stopping strategy to prevent overfitting of the audio processing model during the training process.

[0153] In some embodiments, the initial learning rate of the audio processing model can be 0.0001, and the learning rate can be dynamically adjusted during training.

[0154] In some embodiments, the optimizer used by an electronic device to train an audio processing model can be an Adaptive Moment Estimation (Adam) optimizer.

[0155] It should be noted that the above is merely an example of a training strategy for an audio processing model. The training strategy for an audio processing model can also be determined as needed, and this disclosure does not limit it.

[0156] In some embodiments, a loss value is calculated based on the difference between the third sample audio and the first sample audio, including at least one of the following: calculating a loss value based on the phase difference between the third sample audio and the first sample audio; determining a loss value based on the temporal loss between the third sample audio and the first sample audio; and determining a loss value based on the Fourier loss between the first sample audio and the third sample audio.

[0157] Understandably, after obtaining the third sample audio, the electronic device can determine the phase difference between the third sample audio and the first sample audio, and determine the loss value based on this phase difference. The electronic device can also calculate the temporal loss between the third sample audio and the first sample audio, and use this temporal loss as the final loss value. Furthermore, the electronic device can calculate the Fourier loss between the third sample audio and the first sample audio, and use this Fourier loss as the final loss value.

[0158] In some embodiments, the electronic device can acquire the amplitude of the third sample audio and the phase of the first sample audio. Then, the electronic device can calculate a first mean squared error loss of the phase of the third sample audio and the phase of the first sample audio using a phase loss function, and use the obtained first mean squared error loss as a loss value for adjusting the model parameters.

[0159] In some embodiments, the electronic device may also calculate a second mean square error loss between the third sample audio and the first sample audio using a time-domain loss function, and use the obtained second mean square error loss as a loss value for adjusting the model parameters.

[0160] In some embodiments, the three configurations for the electronic device to calculate the Fourier loss between the third sample audio and the first sample audio can be as follows: Configuration 1 (window length: 80 sampling points, number of sampling points for Fourier transform: 80 sampling points, window shift size: 40 sampling points), Configuration 2 (window length: 160 sampling points, number of sampling points for Fourier transform: 160 sampling points, window shift size: 40 sampling points), and Configuration 3 (window length: 320 sampling points, number of sampling points for Fourier transform: 320 sampling points, window shift size: 40 sampling points).

[0161] In some embodiments, the electronic device may also determine the loss value based on the phase loss between the third sample audio and the first sample audio, the temporal loss between the third sample audio and the first sample audio, and the Fourier loss between the third sample audio and the first sample audio.

[0162] For example, as shown in the following formula (1), the electronic device can determine the loss value used to adjust the model parameters by weighting the phase loss, temporal loss and Fourier loss between the third sample audio and the first sample audio:

[0163] L=λ1L stft +λ2L wav +λ3L phase (1);

[0164] Where L is the loss value used to adjust the model parameters, L stft L is the Fourier loss between the third audio sample and the first audio sample. wav L is the temporal loss between the third sample audio and the first sample audio. phase λ1, λ2, and λ3 represent the phase loss between the third sample audio and the first sample audio, where λ1, λ2, and λ3 are weighting coefficients.

[0165] In some embodiments, the sizes of λ1, λ2, and λ3 can be set as needed, and this disclosure does not limit this.

[0166] In this embodiment of the disclosure, the loss value used to adjust the model parameters is determined by the weighted sum of the phase loss, temporal loss and Fourier loss between the third sample audio and the first sample audio. This can make the adjustment of the model parameters more accurate, thereby enabling the obtained audio processing model to have better audio processing effect.

[0167] refer to Figure 6 , Figure 6 This is a block diagram illustrating an audio processing apparatus according to an exemplary embodiment. Figure 6 As shown, the audio processing device 600 includes: a first acquisition module 601 configured to acquire the amplitude spectrum of the audio to be processed; an extraction module 602 configured to encode the amplitude spectrum using an audio processing model to obtain a first feature vector; a first processing module 603 configured to quantize the first feature vector using an audio processing model to obtain a second feature vector; and a second processing module 604 configured to decode the second feature vector using an audio processing model to obtain a first complex spectrum; wherein the audio quality of the target audio generated based on the first complex spectrum is higher than the audio quality of the audio to be processed.

[0168] In some embodiments, the audio processing model includes a vector quantization module; the first processing module 603 is configured to perform residual vector quantization processing on the first feature vector through the vector quantization module to obtain a second feature vector.

[0169] In some embodiments, the vector quantization module includes N discrete vector sets, where N is an integer greater than 1; wherein, the first processing module 603 is configured to: determine the i-th target vector that is close to the i-th input vector from the i-th discrete vector set using an approximate nearest neighbor search algorithm; calculate the residual between the i-th input vector and the i-th target vector to obtain the i-th residual; wherein i is an integer from 1 to N, when i is 1, the i-th input vector is the first feature vector, and when i is greater than 1, the i-th input vector is the (i-1)-th residual; and determine the second feature vector based on the N residuals.

[0170] In some embodiments, the audio processing model further includes an encoding module and a Long Short-Term Memory (LSTM) module; wherein the extraction module 602 is configured to: extract features from the amplitude spectrum through the encoding module to obtain a third feature vector; and process the third feature vector in the time dimension through an LSTM module to obtain a first feature vector.

[0171] In some embodiments, the audio processing model includes M encoding modules and M long short-term memory (LSTM) modules, where M is an integer greater than 1; wherein the extraction module 602 is configured to: extract features from the amplitude spectrum or the (j-1)th fourth feature vector through the j-th encoding module to obtain the j-th fifth feature vector; wherein j is an integer from 1 to M; wherein when j is 1, the j-th encoding module is used to extract features from the amplitude spectrum; when j is greater than 1, the j-th encoding module is used to extract features from the (j-1)th fourth feature vector; and process the j-th fifth feature vector in the time dimension through the j-th LSTM module to obtain the j-th fourth feature vector; wherein the M-th fourth feature vector is the first feature vector.

[0172] In some embodiments, the bit rate of the audio to be processed is less than or equal to a first threshold.

[0173] refer to Figure 7 , Figure 7 This is a block diagram illustrating a model training apparatus according to an exemplary embodiment. Figure 7As shown, the model training device includes: a second acquisition module 701, configured to acquire a first sample audio and a second sample audio, wherein the second sample audio is the audio after compression of the first sample audio; an input module 702, configured to input the amplitude spectrum of the second sample audio into the audio processing model to be trained to obtain a second complex spectrum; a generation module 703, configured to generate a third sample audio based on the second complex spectrum; a calculation module 704, configured to calculate a loss value based on the difference between the first sample audio and the third sample audio; and a training module 705, configured to iteratively train the audio processing model to be trained based on the loss value to obtain a trained audio processing model.

[0174] In some embodiments, the calculation module 704 is configured to: calculate a loss value based on the phase difference between the third sample audio and the first sample audio; determine a loss value based on the temporal loss between the third sample audio and the first sample audio; and determine a loss value based on the Fourier loss between the third sample audio and the first sample audio.

[0175] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0176] Figure 8 This is a structural block diagram illustrating an electronic device 800 according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, tablet computer, smartwatch, in-vehicle device, or other communication device.

[0177] Reference Figure 8 The electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.

[0178] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with at least one of display, telephone call, data communication, camera operation, and recording operation. Processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0179] Memory 804 is configured to store various types of data to support operation on electronic device 800. Examples of such data include at least one of the following: instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, and videos. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0180] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0181] Multimedia component 808 includes a screen that provides an output interface between electronic device 800 and user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When electronic device 800 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0182] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0183] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, and buttons. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0184] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or one of its components, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetometer, a pressure sensor, and a temperature sensor.

[0185] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as Wi-Fi, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0186] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0187] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including executable instructions or a computer program, which can be executed by a processor 820 of an electronic device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0188] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform any of the audio processing methods described above in the embodiments of this disclosure. For example, the method includes:

[0189] The amplitude spectrum of the audio to be processed is obtained; the amplitude spectrum is encoded using an audio processing model to obtain a first feature vector; the first feature vector is quantized using an audio processing model to obtain a second feature vector; the second feature vector is decoded using an audio processing model to obtain a first complex spectrum; wherein the audio quality of the target audio generated based on the first complex spectrum is higher than that of the audio to be processed.

[0190] This disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the audio processing methods described above in this disclosure.

[0191] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0192] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An audio processing method, characterized in that, The method includes: Obtain the amplitude spectrum of the audio to be processed; The amplitude spectrum is encoded using an audio processing model to obtain a first feature vector. The first feature vector is quantized using the audio processing model to obtain the second feature vector. The second feature vector is decoded using the audio processing model to obtain the first complex spectrum; The target audio generated based on the first complex spectrum has a higher audio quality than the audio to be processed.

2. The method according to claim 1, characterized in that, The audio processing model includes a vector quantization module; The step of quantizing the first feature vector using the audio processing model to obtain the second feature vector includes: The vector quantization module performs residual vector quantization on the first feature vector to obtain the second feature vector.

3. The method according to claim 2, characterized in that, The vector quantization module includes N discrete vector sets, where N is an integer greater than 1; The step of performing residual vector quantization on the first feature vector through the vector quantization module to obtain the second feature vector includes: Using an approximate nearest neighbor search algorithm, the i-th target vector that is close to the i-th input vector is determined from the i-th discrete vector set; Calculate the residual between the i-th input vector and the i-th target vector to obtain the i-th residual; wherein i is an integer from 1 to N, when i is 1, the i-th input vector is the first feature vector, and when i is greater than 1, the i-th input vector is the (i-1)-th residual; The second eigenvector is determined based on N residuals.

4. The method according to claim 1, characterized in that, The audio processing model also includes an encoding module and a Long Short-Term Memory (LSTM) module. The step of encoding the amplitude spectrum using an audio processing model to obtain a first feature vector includes: The encoding module extracts features from the amplitude spectrum to obtain a third feature vector. The third feature vector is processed in the time dimension using an LSTM module to obtain the first feature vector.

5. The method according to claim 1, characterized in that, The audio processing model includes M encoding modules and M long short-term memory (LSTM) modules, where M is an integer greater than 1; The step of encoding the amplitude spectrum through the audio processing model to obtain the first feature vector includes: The j-th encoding module extracts features from the amplitude spectrum or the (j-1)-th fourth feature vector to obtain the j-th fifth feature vector; where j is an integer from 1 to M; wherein, when j is 1, the j-th encoding module is used to extract features from the amplitude spectrum; when j is greater than 1, the j-th encoding module is used to extract features from the (j-1)-th fourth feature vector. The j-th fifth feature vector is processed in the time dimension through the j-th LSTM module to obtain the j-th fourth feature vector; Among them, the Mth fourth feature vector is the first feature vector.

6. The method according to claim 1, characterized in that, The bit rate of the audio to be processed is less than or equal to a first threshold.

7. A model training method, characterized in that, The method includes: Obtain a first sample audio and a second sample audio, wherein the second sample audio is the compressed audio of the first sample audio; The amplitude spectrum of the second sample audio is input into the audio processing model to be trained to obtain the second complex spectrum; Based on the second complex spectrum, a third sample audio is generated; The loss value is calculated based on the difference between the first sample audio and the third sample audio; Based on the loss value, the audio processing model to be trained is iteratively trained to obtain a trained audio processing model.

8. The method according to claim 7, characterized in that, The loss value is calculated based on the difference between the first sample audio and the third sample audio, including at least one of the following: The loss value is calculated based on the phase difference between the third sample audio and the first sample audio. The loss value is determined based on the temporal loss between the third sample audio and the first sample audio; The loss value is determined based on the Fourier loss between the third sample audio and the first sample audio.

9. An audio processing device, characterized in that, The device includes: The first acquisition module is configured to acquire the amplitude spectrum of the audio to be processed; The extraction module is configured to encode the amplitude spectrum using an audio processing model to obtain a first feature vector; The first processing module is configured to quantize the first feature vector using the audio processing model to obtain a second feature vector. The second processing module is configured to decode the second feature vector using the audio processing model to obtain the first complex spectrum. The target audio generated based on the first complex spectrum has a higher audio quality than the audio to be processed.

10. A model training device, characterized in that, The device includes: The second acquisition module is configured to acquire a first sample audio and a second sample audio, wherein the second sample audio is the compressed audio of the first sample audio; The input module is configured to input the amplitude spectrum of the second sample audio into the audio processing model to be trained, so as to obtain the second complex spectrum; The generation module is configured to generate a third sample audio based on the second complex spectrum; The calculation module is configured to calculate a loss value based on the difference between the first sample audio and the third sample audio; The training module is configured to iteratively train the audio processing model to be trained based on the loss value to obtain a trained audio processing model.

11. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the method according to any one of claims 1 to 8.

12. A non-transitory computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions in the storage medium are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

13. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images