Method and device for training audio processing model, equipment and storage medium

Abstract

The embodiment of the invention provides a method and device for training an audio processing model, equipment and a storage medium. The method comprises the following steps: acquiring a training data set, wherein the training data set comprises first audio data with an initial sampling rate and second audio data with a target sampling rate; a threshold number of third audio data is determined by performing a threshold number of weighted superposition operation on the noise data and the second audio data based on a set of predetermined parameters, and the noise data and the first audio data have the same length; determining de-noised fourth audio data by inputting the noise data into an audio processing model; and adjusting parameters of the audio processing model based on the loss between the threshold number of third audio data and fourth audio data. Therefore, by gradually adding and denoising the noise in the training process, the high-frequency component of the audio can be efficiently recovered, and the stability and generalization ability of the model on different audio data are improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of information technology, and in particular, to methods, devices, equipment, and computer-readable storage media for training audio processing models. Background Art

[0002] In the field of audio processing, speech super-resolution technology aims to restore the missing high-frequency components by increasing the sampling rate of low-quality audio signals, thereby generating clearer and more realistic audio signals. This technology has broad application prospects in speech recognition, speech synthesis, and voice communication, especially in scenarios such as conference recording and distance education, which can significantly improve speech quality and understanding.

[0003] In the related technologies, audio processing methods based on speech super-resolution technology mainly rely on deep learning models. Although these methods can improve audio quality to a certain extent, they face problems such as high training difficulty and unstable generation effect. Especially when processing low-frequency band audio, it is often difficult to fully restore high-frequency components, resulting in limited improvement in the quality of audio signals.

[0004] Therefore, a more comprehensive and efficient solution for training audio processing models is needed, which can significantly improve the audio quality of the trained audio processing models. Summary of the invention

[0005] In a first aspect of the present disclosure, a method for training an audio processing model is provided. The method comprises: obtaining a training data set, the training data set comprising first audio data having an initial sampling rate and second audio data having a target sampling rate; determining a threshold amount of third audio data by performing a threshold number of weighted superposition operations on noise data and second audio data based on a set of predetermined parameters, wherein the noise data has the same length as the first audio data; determining denoised fourth audio data by inputting the noise data into the audio processing model; and adjusting the parameters of the audio processing model based on the loss between the threshold amount of third audio data and fourth audio data.

[0006] In a second aspect of the present disclosure, a device for training an audio processing model is provided. The device includes: an acquisition module configured to acquire a training data set, the training data set including first audio data with an initial sampling rate and second audio data with a target sampling rate; a noise superposition module configured to determine a threshold number of third audio data by performing a threshold number of weighted superposition operations on the noise data and the second audio data based on a set of predetermined parameters, wherein the noise data has the same length as the first audio data; a denoising module configured to determine denoised fourth audio data by inputting the noise data into the audio processing model; and a parameter adjustment module configured to adjust the parameters of the audio processing model based on the loss between the threshold number of third audio data and fourth audio data.

[0007] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect of the present disclosure.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to perform the method according to the first aspect of the present disclosure.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer executable instructions, which when executed by a device cause the device to perform the method according to the first aspect of the present disclosure.

[0010] It should be understood that the content described in this content section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0013] Figure 2 A flowchart showing a process of training an audio processing model according to some embodiments of the present disclosure;

[0014] Figure 3A A schematic diagram showing a process of training an audio processing model according to some embodiments of the present disclosure;

[0015] Figure 3B A schematic diagram showing an audio processing process according to some embodiments of the present disclosure is shown;

[0016] Figure 3C A schematic diagram showing the reasoning process of an audio processing model according to some embodiments of the present disclosure is shown;

[0017] Figure 3D A flowchart showing a process of audio processing according to some embodiments of the present disclosure;

[0018] Figure 4 A block diagram showing an apparatus for training an audio processing model according to some embodiments of the present disclosure; and

[0019] Figure 5 A block diagram of an electronic device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0020] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0021] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0022] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0023] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0024] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0026] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0027] As briefly mentioned above, in the field of speech processing, Speech Super Resolution (SSR) technology mainly targets situations where the original audio quality is poor. It learns high-frequency features and predicts the high-quality parts of the original audio, thereby effectively improving the audio quality. This technology aims to restore the missing high-frequency components by increasing the sampling rate of low-quality audio signals, thereby generating clearer and more realistic audio signals. Specifically, Speech Super Resolution uses deep convolutional neural networks (CNN) to increase the sampling rate of audio signals (such as from 8kHz to 16kHz), thereby restoring the high-frequency components of the audio and making the audio quality clearer and more natural.

[0028] In related technologies, with the development of deep learning technology, speech super-resolution technology usually mainly uses Generative Adversarial Networks (GAN) to generate high-frequency parts in audio, and uses CNN as a discriminator to evaluate the generated results. Although GAN performs well in generation tasks, it is prone to problems such as gradient explosion and excessive loss fluctuations during training, requiring repeated training and high training costs. In addition, since the training of the GAN network itself is relatively complex, a large amount of data is required for tuning.

[0029] In addition, the sound super-resolution technology in the related art mainly targets low-band speech (sampling rate of 0-8kHz) and expands it to a higher frequency band (8kHz-16kHz), but even so, the sound quality improvement of the high-frequency part is still limited. Especially in scenarios such as multi-person conferences, higher-quality speech is required to distinguish the speeches of different speakers. Therefore, the sound super-resolution technology in the related art lacks effective support for full-band audio generation, and the bandwidth limitation problem in the super-resolution process has not been effectively solved.

[0030] Therefore, in order to solve the deficiencies of audio processing models in related technologies in frequency band expansion and the complexity of model training, a more comprehensive and efficient solution for training audio processing models is needed that can effectively generate high-quality speech across the entire frequency band.

[0031] In an embodiment of the present disclosure, a scheme for training an audio processing model is provided. According to an embodiment of the present disclosure, a training data set is obtained, the training data set includes first audio data with an initial sampling rate and second audio data with a target sampling rate; a threshold number of third audio data is determined by weighted superposition operations of noise data and second audio data based on a set of predetermined parameters for a threshold number of times, and the noise data has the same length as the first audio data; denoised fourth audio data is determined by inputting the noise data into the audio processing model; and the parameters of the audio processing model are adjusted based on the loss between the threshold number of third audio data and fourth audio data.

[0032] Therefore, by introducing the forward and reverse diffusion process of the diffusion model, gradually adding noise to the training data and denoising it, the high-frequency components of the audio can be efficiently restored, reducing problems such as gradient explosion and convergence difficulties in training and ensuring the stability and generalization ability of the model on different audio data. In this way, not only can high-quality audio super-resolution from low-frequency bands to full-frequency bands be achieved, but also training time and resource consumption can be effectively reduced, ultimately improving the overall performance and practicality of audio processing.

[0033] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In the environment 100, the electronic device 110 obtains training audio data 120. For example, the training audio data 120 includes first audio data having an initial sampling rate and second audio data having a target sampling rate. In some embodiments, the electronic device 110 trains an audio processing model 130 based on the training audio data 120. Using the trained audio processing model 130, the electronic device 110 can obtain target audio data 140 having a target sampling rate.

[0034] In the environment 100, the electronic device 110 can be any type of device with computing capabilities, including a terminal device or a server device. The terminal device can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. The server device can include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0035] It should be understood that the structure and functionality of environment 100 are described for exemplary purposes only and does not imply any limitation on the scope of the present disclosure.

[0036] Figure 2 1 shows a flowchart of a process 200 for training an audio processing model according to some embodiments of the present disclosure. The process 200 may be implemented in the environment 100, for example, the process 200 may be implemented at the electronic device 110. In the following, for ease of discussion, reference will be made to Figure 1 The process 200 is described below.

[0037] In block 210 , the electronic device 110 acquires a training data set.

[0038] In some embodiments, the training data set includes first audio data having an initial sampling rate and second audio data having a target sampling rate.

[0039] In some embodiments, the electronic device 110 can construct a set of audio data with two different sampling rates for use in training the audio processing model 130. For example, one set of low-band audio data with an initial sampling rate (8 kHz sampling rate) and another set of full-band audio data with a target sampling rate (48 kHz sampling rate).

[0040] The full-band audio data with the target sampling rate (48kHz sampling rate) can be used as label data (GroundTruth) to provide a higher quality audio signal as a target output for the audio processing model 130, thereby helping the model learn how to recover higher quality full-band audio from low quality low-band audio. Full-band audio contains more details and high frequency components, which is crucial for audio super-resolution tasks.

[0041] Low-frequency audio data with an initial sampling rate (8kHz sampling rate) is used as the initial audio data to enable the model to learn how to gradually recover the missing high-frequency components from low-quality audio signals to complete the super-resolution task. Low-frequency audio usually contains the main information of the audio signal, while the high-frequency components are responsible for improving the details and clarity of the audio.

[0042] As an example, the electronic device 110 can obtain a training data set from a voice library data set, which can include a variety of voice samples, thereby providing rich training materials to help the model better understand the characteristics of different voices and improve generalization capabilities. For example, the length of the data set is 1064 hours and the bit depth is 16 bits.

[0043] In some embodiments, the electronic device 110 acquires the first audio data and the second audio data, and performs a pre-processing operation on the first audio data and the second audio data.

[0044] For example, the electronic device 110 captures each segment of acquired audio data at a length of 2 seconds (i.e., 16,000 sampling points) to ensure uniform length of the data. Alternatively, the electronic device 110 eliminates differences that may be caused by different audio sources or formats by unifying the format of the audio data (such as WAV format), thereby ensuring data consistency during the training process. The electronic device 110 can also convert the acquired audio data into a mono format to avoid possible redundant information in the stereo channel and simplify the model processing process.

[0045] In some embodiments, the electronic device 110 adds a blank data frame at the end of the first audio data and the second audio data. The downsampling and upsampling processes of audio data usually require a certain audio length to perform operations such as convolution and pooling. In order to avoid alignment problems or data loss problems in these operations, it is necessary to add a blank data frame to ensure that unnecessary truncation or distortion will not occur when the sampling operation is performed subsequently, thereby ensuring the calculation effect of each operation step.

[0046] As an example, for a downsampling scheme with parameters of (2, 2, 4, 4, 8, 8), this requires that the length of the audio data is a multiple of 4096. The electronic device 110 can add 384 "0" data frames at the end of the audio data with a length of 2 seconds (i.e., 16,000 sampling points) to make the size of the audio data meet the requirements of the model network structure.

[0047] In the disclosed embodiments, the training data is made consistent, operable and efficient through unified processing, standardized interception and padding of audio data. These processes not only improve the training efficiency of the model, but also ensure the consistency of data and high-quality input and output pairing, thus laying the foundation for candidate model training and speech super-resolution tasks.

[0048] Continue to refer Figure 2 In block 220 , the electronic device 110 determines a threshold amount of third audio data by performing a threshold number of weighted superposition operations on the noise data and the second audio data based on a set of predetermined parameters.

[0049] As an example, Figure 3A FIG. 3 is a schematic diagram of a process 300A for training an audio processing model 130 according to some embodiments of the present disclosure. The process 300A may be implemented at the electronic device 110 .

[0050] like Figure 3A As shown, in box 311, the electronic device 110 obtains second audio data with a target sampling rate (such as 48KHz) as training label data.

[0051] In block 312, the electronic device 110 samples a piece of noise data having the same length as the first audio data. The noise data may be obtained by random sampling based on Gaussian distribution.

[0052] Gaussian noise can also be called normal distribution noise, which is a noise signal with a statistical mean of 0 and a standard deviation of 1. The distribution of Gaussian noise conforms to the normal distribution, usually expressed as N(0,1), that is, a normal distribution with a mean of 0 and a variance of 1. The electronic device 110 can randomly generate a noise signal, and each data point conforms to the standard normal distribution. The Gaussian noise data can be used to simulate irregular, random and unpredictable changes in the real world during the model training process.

[0053] Continue to refer Figure 3A , in box 313, the electronic device 110 superimposes the noise data with the second audio data.

[0054] In some embodiments, the electronic device 110 determines the weight of the noise data and the weight of the second audio data based on a predetermined set of parameters. The weight of the noise data is greater than the corresponding weight of the noise data in the last weighted superposition operation, and the weight of the second audio data is less than the corresponding weight of the second audio data in the last weighted superposition operation. Then, the electronic device 110 performs a threshold number of weighted superposition operations based on the weight of the noise data and the weight of the second audio data to determine a threshold number of third audio data.

[0055] In box 314, the electronic device 110 repeats the superposition operation. In the training of the model, the forward diffusion process needs to gradually add noise to the audio data so that the audio signal is gradually converted into pure noise. By weighted addition of pure Gaussian noise to the original audio data and gradually increasing the noise, the model can learn how to recover a clear audio signal from the noise. This process provides the required noise and signal pairing for the denoising training of the model.

[0056] At block 315 , the electronic device 110 obtains a threshold amount of noise-stepped audio data.

[0057] As an example, the electronic device 110 performs 1000 weighted superposition operations on the noise data and the second audio data in the following manner to obtain 1000 audio data with gradually increasing noise:

[0058]

[0059] Among them, x t is the obtained audio data with noise, t indicates that the current is the tth superposition process, α is a set of pre-set parameters that determine the weighted ratio of noise to audio data in each superposition process, and t indicates the index of each superposition process. t-1 is the audio data obtained by the previous round of superposition, which is combined with the noise data ∈ t-1 After combining, generate new audio data x t .

[0060] Return to reference Figure 2 In block 230 , the electronic device 110 determines denoised fourth audio data by inputting the noise data into the audio processing model 130 .

[0061] For example, refer to Figure 3A In block 316, the electronic device 110 may sample a piece of Gaussian noise data each time the noise is repeatedly superimposed in block 314. Then, in block 317, the electronic device 110 inputs the noise data into the audio processing model 130 for denoising.

[0062] In box 318, the electronic device 110 obtains the denoised audio data. In this way, in the forward diffusion process of the model, a newly generated noise signal is input into the denoising network at each step of noise superposition, and is gradually trained in the denoising network by forward diffusion.

[0063] In some embodiments, the audio processing model 130 includes a diffusion model. The diffusion model is a generative model based on a random process and is widely used in image and audio generation tasks. In the disclosed embodiment, the diffusion model is used to restore the clarity of the audio signal by gradually adding noise to the audio data and gradually removing the noise through a denoising network. In this project, learning is used to restore low-quality audio to high-quality audio.

[0064] In some embodiments, the diffusion model includes a modified U-type convolutional neural network (U-net) as a denoising network, and the denoising network consists of an encoder module and a decoder module based on a depth-wise separable convolutional layer.

[0065] Depthwise separable convolution is an optimization of traditional convolution, which splits the convolution operation into two steps: depthwise convolution and pointwise convolution. Among them, depthwise convolution reduces the amount of calculation by performing convolution operations on each input channel separately without interfering with other channels. Pointwise convolution performs 1x1 convolution on each channel after depthwise convolution, which is mainly used to change the number of channels without changing the spatial resolution. The use of depthwise separable convolution reduces the amount of calculation and the number of parameters of the network, making the audio processing model 130 more efficient, but still able to extract important features in the audio signal.

[0066] In some embodiments, the electronic device 110 inputs the noise data into the encoder module. The electronic device 110 then uses the depthwise separable convolutional layer of the encoder module to perform downsampling and feature extraction operations on the noise data.

[0067] In the encoder module, depthwise separable convolutions are often combined with downsampling operations. By gradually downsampling, the encoder can effectively compress the audio data and extract more abstract features. This process also helps the model understand the overall structure of the audio data, not just the local details.

[0068] As an example, the denoising network may include six encoder blocks and six decoder blocks. Each block includes four depth-separable convolution blocks, each of which consists of two point convolution layers and one depth-separable convolution layer. The six encoder blocks use downsampling parameters of (2, 2, 4, 4, 8, 8) to perform downsampling operations.

[0069] In some embodiments, the electronic device 110 may store the output result of each encoder block in a residual array, so as to perform residual chaining in subsequent stages to help retain more feature information and avoid information loss.

[0070] In some embodiments, the encoder output is fed into two standard one-dimensional convolutional layers for convolution operations, and these convolution kernels are of size 1. Such convolution operations act as a bridge between the encoder and the decoder, ensuring that the lengths of the input and output signals are consistent. They ensure that the encoder output and the decoder input are smoothly connected, avoiding calculation errors caused by size mismatch.

[0071] In some embodiments, the electronic device 110 inputs the feature data corresponding to the noise data output by the encoder module into the decoder module. Then, the electronic device 110 determines the fourth audio data by performing upsampling and feature reconstruction operations on the noise data using the depthwise separable convolutional layer of the decoder module.

[0072] As an example, the decoder module is responsible for gradually restoring the details of the audio data. The decoder module can also use upsampling parameters of (8, 8, 4, 4, 2, 2) to perform upsampling. This means that the decoder module will gradually restore the size of the audio data to its original size. In addition, each decoder block can also include four depth-separable convolution blocks.

[0073] After obtaining the output of each decoder block, the electronic device 110 can also add the output to the residual connection of the corresponding encoder module part, so as to retain the information of the current decoder layer and combine the information of the encoder to process the audio data more accurately.

[0074] Return to reference Figure 2 At block 240 , the electronic device 110 adjusts parameters of the audio processing model 130 based on a loss between a threshold amount of the third audio data and the fourth audio data.

[0075] For example, refer to Figure 3A In box 319, the electronic device 110 calculates the loss between the label data obtained in each noise superposition process and the denoising result output by the model, and adjusts the parameters of the audio processing model 130 by comparing the loss between the denoising result and the label data.

[0076] In some embodiments, the loss is calculated based on a loss function. The loss function calculation includes at least one of the following: a set similarity metric loss function, a cross entropy loss function, and a mean square error loss function.

[0077] Dice Loss is a commonly used similarity metric used to measure the similarity between two sets, especially for image segmentation tasks. In audio processing models, Dice loss is used to measure the overlap between the denoised audio signal and the target audio signal (true label).

[0078] Cross entropy loss (BCE-Binary Cross Entropy Loss) is often used in classification tasks to measure the difference between the target probability distribution and the predicted probability distribution. In audio processing, BCE can be used to measure the difference between the denoised audio output by the model and the target audio.

[0079] MSE (Mean Squared Error Loss) is one of the most common regression loss functions, which is used to measure the squared difference between the predicted value and the true value.

[0080] The electronic device 110 may combine the three loss functions (Dice loss, BCE loss, and MSE loss) to form a composite loss function.

[0081] As an example, the electronic device 110 may determine the loss of audio data in the following manner:

[0082] L totle =L dice (x′ t , x t )+L bce (x′ t , x t )+L mse (x′ t , x t ) (2)

[0083] Among them, x t ′ is the denoised audio data output by the model, x t is the noisy frequency data obtained in each noise superposition process.

[0084] In the disclosed embodiment, a plurality of measurement methods are combined together through a composite loss function to ensure that during the training process, the model can balance the losses in various aspects and avoid training instability caused by a single indicator.

[0085] In some embodiments, the electronic device 110 uses a back propagation algorithm to adjust the parameters of the audio processing model 130 based on the calculated loss.

[0086] Backpropagation is an algorithm that updates model parameters by calculating gradients. It calculates the partial derivatives of each parameter in the model layer by layer through the loss function, and adjusts the parameters in the direction of gradient descent to minimize the loss.

[0087] Return to reference Figure 3AIn box 310, the electronic device 110 repeats the above-mentioned model training process until the loss converges or reaches a threshold training cycle (epoch).

[0088] In the disclosed embodiment, a loss function is used to measure the difference between the label data obtained in each noise superposition process and the denoising result output by the model, and the difference is minimized through the back propagation algorithm, so that the performance of the audio processing model can be optimized in a variety of problem scenarios. This method can not only effectively improve the prediction accuracy of the model, but also accelerate the convergence of the model, enhance the generalization ability of the model, and finally obtain a more accurate and robust audio processing model.

[0089] As an example, Figure 3B FIG. 3 is a schematic diagram of an audio processing process 300B according to some embodiments of the present disclosure. The process 300B may be implemented at the electronic device 110 .

[0090] In box 321, the electronic device 110 reads or loads the audio file to be processed for subsequent processing steps.

[0091] At block 322 , the electronic device 110 performs specific audio processing operations.

[0092] For example, the electronic device 110 first obtains the audio data to be processed and extracts relevant features or sampling rates therefrom. Before performing audio processing, the electronic device 110 may perform necessary preprocessing operations on the audio data, such as unified processing of the audio data, normalized interception and padding, etc. Then the electronic device 110 inputs the preprocessed audio data into the trained audio processing model for audio processing to obtain target audio data with a target sampling rate. Finally, the electronic device 110 saves the audio data processed by the model as the target audio output.

[0093] Figure 3C FIG. 3 is a schematic diagram of an inference process 300C of an audio processing model 130 according to some embodiments of the present disclosure. The process 300C may be implemented at the electronic device 110 .

[0094] In block 331 , the electronic device 110 acquires fifth audio data with an initial sampling rate to be processed.

[0095] In block 332 , the electronic device 110 acquires noise data having the same length as the fifth audio data.

[0096] For example, in the process of training the aforementioned audio processing model 130, i.e., the process of forward diffusion, noise data is gradually added to the audio data with the target sampling rate. The audio processing using the trained audio processing model 130 is a corresponding reverse diffusion process. The goal is to gradually remove the noise and restore the audio data with the target sampling rate (e.g., 48KHz) starting from the fifth audio data to be processed with an initial sampling rate (e.g., 8KHz) and the noise data. Therefore, the electronic device 110 first needs to obtain the fifth audio data to be processed with an initial sampling rate and the noise data having the same length as the fifth audio data.

[0097] In block 333, the electronic device 110 superimposes the noise data with the fifth audio data to be processed to obtain audio data for inputting the model. The audio data superimposed with noise will be used as input data of the audio processing model 130 to perform the inverse diffusion process.

[0098] In block 334, the electronic device 110 inputs the audio data with superimposed noise into the audio processing model 130 obtained by the aforementioned model training process. In block 335, the electronic device 110 uses the trained audio processing model 130 to perform a threshold number of denoising processes to obtain target audio data with a target sampling rate.

[0099] In block 336 , the electronic device 110 obtains target audio data having a target sampling rate.

[0100] As an example, the model denoising process is repeated 1000 times, simulating the process of gradually adding noise at each step in the model training (forward diffusion) process. Through each denoising, the audio processing model 130 continuously recovers the audio signal from the noise, gradually removes the noise and reconstructs the audio. After 1000 denoising processes, the audio processing model 130 finally outputs a target audio data that has been denoised and reconstructed to the target resolution (48KHz). These data are the audio data restored by the inverse diffusion process.

[0101] Return to reference Figure 3B In block 323 , the electronic device 110 may further post-process the audio processing result, for example, including additional optimization or detail correction, such as enhancing the sound quality, adjusting the volume, etc. Or publish the final processed audio file for the user to download.

[0102] Figure 3D FIG. 3 is a schematic diagram of a process 300D of audio processing according to some embodiments of the present disclosure. The process 300D may be implemented in the environment 100 , for example, 300D may be implemented at the electronic device 110 .

[0103] In block 341 , the electronic device 110 acquires fifth audio data to be processed having an initial sampling rate and noise data having the same length as the fifth audio data.

[0104] In block 342 , the electronic device 110 obtains fourth audio data by superimposing the fifth audio data and noise data having the same length as the fifth audio data and inputting the superimposed audio data to the parameter-adjusted audio processing model.

[0105] In box 343, the electronic device 110 obtains target audio data with a target sampling rate by further inputting the audio data output by the audio processing model into the audio processing model after parameter adjustment for a threshold number of denoising processes.

[0106] In the disclosed embodiment, by using a trained audio processing model, an inverse diffusion process is performed to recover clear audio data with a target resolution from noisy data in a step-by-step denoising manner. The trained audio processing model uses an encoder and decoder architecture to gradually remove noise and ultimately generate high-quality audio. In this way, the audio processing model can efficiently restore audio details in the inference stage, thereby achieving the task of audio super-resolution.

[0107] Figure 4 A block diagram of an apparatus 400 for training an audio processing model according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or include the electronic device 110.

[0108] The device 400 includes an acquisition module 410, which is configured to acquire a training data set, the training data set including first audio data having an initial sampling rate and second audio data having a target sampling rate; a noise superposition module 420, which is configured to determine a threshold amount of third audio data by performing a threshold number of weighted superposition operations on the noise data and the second audio data based on a set of predetermined parameters, wherein the noise data has the same length as the first audio data; a denoising module 430, which is configured to determine denoised fourth audio data by inputting the noise data into an audio processing model; and a parameter adjustment module 440, which is configured to adjust the parameters of the audio processing model based on the loss between the threshold amount of third audio data and fourth audio data.

[0109] In some embodiments, the acquisition module 410 is further configured to acquire the first audio data and the second audio data; and add a blank data frame at the end of the first audio data and the second audio data so that the length of the first audio data and the second audio data is an integer multiple of a predetermined downsampling parameter.

[0110] In some embodiments, the noise data is obtained by random sampling based on a Gaussian distribution.

[0111] In some embodiments, the noise superposition module 420 is further configured to determine the weight of the noise data and the weight of the second audio data based on parameters, the weight of the noise data is greater than the corresponding weight of the noise data in the last weighted superposition operation, and the weight of the second audio data is less than the corresponding weight of the second audio data in the last weighted superposition operation; and determine a threshold number of third audio data units by performing a threshold number of weighted superposition operations based on the weight of the noise data and the weight of the second audio data.

[0112] In some embodiments, the audio processing model includes a diffusion model. The diffusion model includes an encoder module and a decoder module based on a depth-separable convolutional layer. The denoising module 430 is further configured to input the noise data into the encoder module; perform downsampling operations and feature extraction operations on the noise data using the depth-separable convolutional layer of the encoder module; input feature data corresponding to the noise data output by the encoder module into the decoder module; and determine the fourth audio data by performing upsampling operations and feature reconstruction operations on the noise data using the depth-separable convolutional layer of the decoder module.

[0113] In some embodiments, the loss is calculated based on a loss function, and the loss function calculation includes at least one of the following: a set similarity metric loss function, a cross entropy loss function, and a mean square error loss function.

[0114] In some embodiments, the parameter adjustment module 440 is further configured to adjust the parameters of the audio processing model using a back propagation algorithm until the loss converges or reaches a threshold training cycle.

[0115] In some embodiments, the device 400 also includes an audio processing module 450, which is configured to obtain fifth audio data to be processed with an initial sampling rate and noise data having the same length as the fifth audio data; obtain sixth audio data by superimposing the fifth audio data and the noise data having the same length as the fifth audio data and inputting them into an audio processing model after parameter adjustment; and obtain target audio data with a target sampling rate by continuing to input the audio data output by the audio processing model into the audio processing model after parameter adjustment for denoising processing a threshold number of times.

[0116] The modules included in the device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units in the device 400 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0117] Figure 5 1 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 5 The electronic device 500 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein.

[0118] like Figure 5 As shown, the electronic device 500 is in the form of a general electronic device. The components of the electronic device 500 may include, but are not limited to, one or more processors or processing units 510, a memory 520, a storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processing unit 510 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 520. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 500.

[0119] The electronic device 500 typically includes a plurality of computer storage media. Such media may be any available media accessible to the electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 500.

[0120] The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 5As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 520 may include a computer program product 525 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0121] The communication unit 540 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 500 can be implemented in a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0122] The input device 550 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 560 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 500 may also communicate with one or more external devices (not shown) through the communication unit 540 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 500, or communicate with any device that allows the electronic device 500 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0123] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which one or more computer instructions are stored, wherein the one or more computer instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0124] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0125] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0126] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0127] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0128] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the marketplace, or to enable other persons of ordinary skill in the art to understand the implementations disclosed herein.

Claims

1. A method for training an audio processing model, comprising: Acquire a training data set, wherein the training data set includes first audio data having an initial sampling rate and second audio data having a target sampling rate; Determining a threshold amount of third audio data by performing a weighted superposition operation of the noise data and the second audio data a threshold number of times based on a set of predetermined parameters, wherein the noise data has the same length as the first audio data; Determining denoised fourth audio data by inputting the noise data into the audio processing model; as well as Based on a loss between the threshold amount of third audio data and the fourth audio data, a parameter of the audio processing model is adjusted.

2. The method according to claim 1, wherein obtaining a training data set comprises: Acquire the first audio data and the second audio data; A blank data frame is added at the end of the first audio data and the second audio data so that the lengths of the first audio data and the second audio data are integer multiples of a predetermined down-sampling parameter. The method according to claim 1 , wherein the noise data is obtained by random sampling based on Gaussian distribution.

4. The method of claim 1 , wherein determining the third audio data comprises: Determining, based on the parameters, a weight of the noise data and a weight of the second audio data, wherein the weight of the noise data is greater than a corresponding weight of the noise data in a previous weighted superposition operation, and the weight of the second audio data is less than a corresponding weight of the second audio data in the previous weighted superposition operation; as well as The threshold amount of third audio data is determined by performing the threshold number of weighted superposition operations based on the weight of the noise data and the weight of the second audio data.

5. The method of claim 1, wherein the audio processing model comprises a diffusion model, the diffusion model comprising an encoder module and a decoder module based on depthwise separable convolutional layers, wherein determining the fourth audio data comprises: inputting the noise data into the encoder module; Performing downsampling and feature extraction operations on the noisy data using a depthwise separable convolutional layer of the encoder module; Inputting the feature data corresponding to the noise data output by the encoder module into the decoder module; as well as The fourth audio data is determined by performing an upsampling operation and a feature reconstruction operation on the noise data using a depthwise separable convolutional layer of the decoder module.

6. The method according to claim 1, wherein the loss is calculated based on a loss function, and the loss function calculation includes at least one of the following: a set similarity metric loss function, a cross entropy loss function, and a mean square error loss function.

7. The method of claim 1, wherein adjusting the parameters of the audio processing model comprises: The parameters of the audio processing model are adjusted using a back propagation algorithm until the loss converges or reaches a threshold training cycle.

8. The method according to claim 1, further comprising: Acquire fifth audio data to be processed and having the initial sampling rate and noise data having the same length as the fifth audio data; Obtaining sixth audio data by superimposing the fifth audio data and noise data having the same length as the fifth audio data and inputting the superimposed audio data into the audio processing model after parameter adjustment; as well as The target audio data having the target sampling rate is obtained by continuously inputting the audio data output by the audio processing model into the audio processing model after parameter adjustment to perform denoising processing for the threshold number of times.

9. A device for training an audio processing model, comprising: An acquisition module is configured to acquire a training data set, wherein the training data set includes first audio data having an initial sampling rate and second audio data having a target sampling rate; a noise superposition module configured to determine a threshold amount of third audio data by performing a weighted superposition operation of the noise data and the second audio data a threshold number of times based on a set of predetermined parameters, the noise data having the same length as the first audio data; a denoising module, configured to determine denoised fourth audio data by inputting the noise data into the audio processing model; as well as The parameter adjustment module is configured to adjust the parameters of the audio processing model based on the loss between the threshold amount of the third audio data and the fourth audio data.

10. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 8.

Images