Audio data processing method, device, equipment, storage medium and program product

Abstract

The present application provides an audio data processing method, apparatus, device, storage medium, and program product. The method comprises: obtaining original audio data having a first resolution; encoding the original audio data to obtain a potential representation vector of the original audio data in a latent space; performing vector conversion on the potential representation vector via a preset vector conversion path to obtain a conversion vector; decoding the conversion vector to obtain target audio data having a second resolution; the second resolution being greater than the first resolution. Through the present application, accurate and efficient conversion from low-resolution original audio data to high-resolution target audio data can be achieved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to an audio data processing method, apparatus, device, storage medium, and program product. Background Art

[0002] With the rapid development of digital audio technology, high-resolution audio data is increasingly being used in various fields. High-resolution audio (HRA) is an example of high-resolution audio data. Its application in music, speech, and environmental sounds is growing. HRA typically refers to audio formats with sampling rates and bit depths exceeding CD quality (for example, 48kHz / 24-bit). HRA delivers richer detail and a wider dynamic range, enhancing the listening experience.

[0003] However, due to limitations in recording equipment, storage space, and transmission bandwidth, many audio contents are stored at lower sampling rates, resulting in degraded sound quality. Therefore, efficiently converting low-resolution audio to high-resolution audio has become an important research topic in the field of audio signal processing. Summary of the Invention

[0004] Embodiments of the present application provide a data processing method, apparatus, device, storage medium, and program product, which encode original audio data into a latent representation vector in a latent space, and then perform vector conversion on the latent representation vector through a preset vector conversion path, thereby obtaining a conversion vector capable of generating target audio data with high resolution, thereby achieving accurate and efficient conversion from low-resolution original audio data to high-resolution target audio data.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides an audio data processing method, which includes: obtaining original audio data having a first resolution; encoding the original audio data to obtain a latent representation vector of the original audio data in a latent space; performing vector conversion on the latent representation vector through a preset vector conversion path to obtain a conversion vector; decoding the conversion vector to obtain target audio data having a second resolution; the second resolution is greater than the first resolution.

[0007] An embodiment of the present application provides an audio data processing device, including: an acquisition module for acquiring original audio data having a first resolution; an encoding processing module for encoding the original audio data to obtain a latent representation vector of the original audio data in a latent space; a vector conversion module for performing vector conversion on the latent representation vector through a preset vector conversion path to obtain a conversion vector; and a decoding processing module for decoding the conversion vector to obtain target audio data having a second resolution; the second resolution is greater than the first resolution.

[0008] In the above scheme, the encoding processing module is also used to: extract the original feature vector from the waveform data of the original audio data; the dimension of the original feature vector is a preset dimension; and use a preset compression ratio to perform vector compression on the original feature vector to obtain the potential representation vector of the original audio data in the latent space.

[0009] In the above scheme, the encoding processing module is also used to: input the original audio data into a pre-trained audio data encoder; extract the original feature vector from the waveform data of the original audio data through the feature extraction layer of the audio data encoder; the potential number of channels of the audio data encoder is equal to the preset dimension.

[0010] In the above scheme, the device also includes: a scaling processing module, which is used to encode the original audio data to obtain the potential representation vector of the original audio data in the latent space, and then use a preset scaling factor to scale the potential representation vector to obtain a scaled vector; the vector conversion module is also used to: perform vector conversion on the scaled vector through the vector conversion path to obtain the conversion vector.

[0011] In the above solution, the vector conversion path includes a vector conversion function; the vector conversion module is further used to: perform random differentiation processing on the potential representation vector through the vector conversion function to obtain the conversion vector.

[0012] In the above scheme, the audio data processing method is implemented by a data processing model; the data processing model includes an audio data encoder, a vector converter and an audio data decoder; the device also includes: a model training module, which is used to train the data processing model through the following steps: obtaining sample data; the sample data includes sample audio data with a first resolution and reference audio data with a second resolution; inputting the sample audio data into the data processing model to be trained; encoding the sample audio data through the audio data encoder of the data processing model to be trained to obtain a sample latent representation vector of the sample audio data in the latent space; performing vector conversion on the sample latent representation vector through the vector converter of the data processing model to be trained to obtain a sample conversion vector; decoding the sample conversion vector through the audio data decoder of the data processing model to be trained to obtain predicted audio data with a third resolution; performing loss calculation based on the predicted audio data and the reference audio data to obtain a loss result; updating the model parameters in the data processing model to be trained based on the loss result to obtain a trained data processing model.

[0013] In the above scheme, the model training module is also used to: in response to the current model training process, at least one of the audio data encoder and the audio data decoder is trained, determine the reconstruction error value based on the predicted audio data and the reference audio data; determine the probability distribution error value based on the mean and variance of the sample potential representation vector; determine the multi-resolution error value based on the transformation data of the predicted audio data and the reference audio data at different resolutions; determine the multi-period loss value of the predicted audio data and the reference audio data in different periods; fuse the reconstruction error value, the probability distribution error value, the multi-resolution error value and the multi-period loss value to obtain the loss result; accordingly, update the model parameters in at least one of the audio data encoder and the audio data decoder based on the loss result.

[0014] In the above scheme, the model training module is also used to: in response to the current model training process, the vector converter is trained, and the loss result of the vector converter is determined based on the predicted audio data and the reference audio data; accordingly, the model parameters in the vector converter are updated based on the loss result.

[0015] An embodiment of the present application provides an electronic device, comprising: a memory for storing computer-executable instructions or computer programs; and a processor for implementing the audio data processing method provided in the embodiment of the present application when executing the computer-executable instructions or computer programs stored in the memory.

[0016] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the audio data processing method provided in the embodiment of the present application when executed by a processor.

[0017] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the audio data processing method provided in the embodiment of the present application is implemented.

[0018] The embodiments of the present application have the following beneficial effects:

[0019] In order to process original audio data with a low resolution (for example, the first resolution) into target audio data with a high resolution (for example, the second resolution), an embodiment of the present application can first encode the original audio data with the first resolution to obtain a latent representation vector of the original audio data in the latent space; then, the latent representation vector is vector-converted through a preset vector conversion path, thereby realizing data conversion of the original audio data in the latent space and obtaining a conversion vector that also belongs to the latent space; finally, the conversion vector is decoded to obtain target audio data with a second resolution, which is greater than the first resolution. In this way, by encoding the original audio data into a latent representation vector in the latent space and then vector-converting the latent representation vector through a preset vector conversion path, a conversion vector capable of generating target audio data with a high resolution is obtained, thereby achieving accurate and effective conversion from low-resolution original audio data to high-resolution target audio data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is an optional flowchart of the audio data processing method provided in an embodiment of the present application;

[0021] Figure 2 This is a schematic diagram of an implementation flow of encoding raw audio data provided by an embodiment of the present application;

[0022] Figure 3 This is another optional flowchart of the audio data processing method provided in an embodiment of the present application;

[0023] Figure 4 Schematic diagram of a data processing model training method provided in an embodiment of the present application;

[0024] Figure 5 This is a schematic diagram of an implementation flow of loss calculation based on predicted audio data and reference audio data provided by an embodiment of the present application;

[0025] Figure 6 This is another implementation flow diagram of loss calculation based on predicted audio data and reference audio data provided by an embodiment of the present application;

[0026] Figure 7 This is a diagram of the overall architecture of the audio super-resolution system provided in an embodiment of the present application;

[0027] Figure 8 Schematic diagram of the implementation process of the audio super-resolution method provided in the embodiment of the present application;

[0028] Figure 9 This is a comparison diagram of the technical implementation process of LBM provided in the embodiment of the present application and the technical implementation process of AudioSR in the related art;

[0029] Figure 10 This is a structural block diagram of an audio data processing device provided in an embodiment of the present application;

[0030] Figure 11 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0032] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0034] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or portion of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal. It can be implemented in whole or in part using software, hardware (such as processing circuits or memory), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the functionality of the module or unit.

[0035] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0036] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0037] Before further explaining the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0038] 1) In response to: This field indicates the conditions or states upon which the executed operation depends. When the conditions or states are met, the executed operation or operations may be performed in real time or with a set delay. Unless otherwise specified, the executed operations may be executed in any order.

[0039] 2) Human-computer interaction interface, which is used to provide human-computer interaction functions / interface for displaying image information.

[0040] For example, graphical user interface (GUI) displays, such as augmented reality (AR) interface, virtual reality (VR) interface, voice user interface (VUI), interactive projection interface (using projection technology to display information on a plane), eye movement detection interface (interface controlled by detecting the user's line of sight), holographic interface (three-dimensional holograms formed by projecting images through holographic projection technology, which can be seen without wearing special glasses), multimodal interface (interface that combines multiple interaction methods, such as touch, vision, and hearing), brain-machine interface (BMI) interface, etc.

[0041] 3) High-Resolution Audio: This refers to an audio format with a sampling rate and bit depth higher than CD quality (for example, 48kHz / 24-bit). High-Resolution Audio delivers richer details and a wider dynamic range, enhancing the listening experience. High-Resolution Audio files are typically significantly larger than CD-quality audio files. For example, a 96kHz / 24-bit audio file can be more than twice the size of a 44.1kHz / 16-bit file. To fully experience the benefits of high-resolution audio, high-quality audio equipment and a player are required. Ordinary headphones or speakers may not fully demonstrate the advantages of high-resolution audio. High-Resolution Audio captures greater detail, including high-frequency overtones, subtle background sounds, and complex timbre variations. These details are more clearly presented on high-quality audio equipment. Due to its higher bit depth, high-resolution audio can better handle sounds from extremely low to extremely high volumes, avoiding distortion or compression in music with a wide dynamic range. High-Resolution Audio also more closely resembles the sound quality of the original recording, providing a more natural and authentic sound. For example, in live music recordings, high-resolution audio can better recreate the atmosphere and sense of space of the venue.

[0042] 4) Sampling rate: This refers to the number of times an audio signal is sampled per unit time, typically measured in Hertz (Hz). For example, a 48kHz sampling rate means the audio signal is sampled 48,000 times per second. High-resolution audio often uses sampling rates higher than the CD-quality 44.1kHz. Common high-resolution sampling rates include 96kHz, 192kHz, and even higher. Higher sampling rates capture a wider frequency range, particularly high frequencies that are difficult for the human ear to perceive. For example, a 96kHz sampling rate can capture frequencies up to 48kHz, while a CD-quality 44.1kHz sampling rate can only capture frequencies up to 22.05kHz. While the human ear's hearing range is generally between 20Hz and 20kHz, higher sampling rate audio preserves greater detail and high-frequency information. This detail is particularly evident in the overtones of certain instruments, the high-frequency harmonics of percussion instruments, and natural ambient sounds such as birdsong and wind.

[0043] 5) Bit depth: refers to the number of binary bits used for each sampling point to represent the amplitude of the audio signal. The higher the bit depth, the higher the amplitude accuracy that can be represented. For example, 24-bit depth can represent 2 24 (approximately 16.77 million) different amplitude levels, while 16-bit depth can only represent 2 16 (65536) levels. High-resolution audio typically uses a higher bit depth than CD-quality 16-bit. A common high-resolution bit depth is 24-bit, which provides a wider dynamic range and lower quantization error. Higher bit depths provide a wider dynamic range—the difference between the quietest and loudest sounds. They also reduce quantization noise, making the audio sound purer.

[0044] In the field of audio super-resolution (ASR), related technologies mainly include methods based on generative adversarial networks (GAN), diffusion models, and variational autoencoders (VAE).

[0045] Among them, GAN-based methods train the adversarial relationship between the generator and the discriminator, enabling the generator to generate realistic high-resolution audio from low-resolution audio. However, the GAN training process is unstable and prone to artifacts, which affects the quality of the generated audio.

[0046] Diffusion model-based methods achieve high-quality data generation by simulating the process of data moving from a noise distribution to a target distribution. Although diffusion models perform well in terms of generation quality, their inference process usually requires multiple sampling steps, resulting in slow inference speed.

[0047] Among VAE-based methods, AudioSR is a representative work. It uses a diffusion-based generative model to upsample audio in the mel-spectrogram space, generating high-resolution target audio from a low-resolution signal. This method has achieved significant progress in processing a variety of audio types, such as music, speech, and sound effects. Among mel-VAE-based methods, AudioSR employs a dual-cascade paradigm combining mel-spectrograms and VAEs. First, low-resolution audio is converted to a mel-spectrogram. A high-resolution mel-spectrogram is then generated using a VAE and diffusion model. Finally, a neural vocoder is used to convert the mel-spectrogram back into a waveform. However, this cascade approach has at least the following issues: The reconstruction effect is affected by the cascaded compression network paradigm, and errors in each stage may accumulate, affecting the final reconstruction quality. The multi-stage processing requires a complex network architecture, increasing the training and inference costs of the model. Spectral-based super-resolution struggles to target low-frequency information. Processing in the spectral domain may result in loss of low-frequency information, requiring additional post-processing steps to compensate, which can introduce an imbalance between low and high frequencies.

[0048] In summary, various implementation methods in the ASR field in the related art all have the problem of inaccurate results and great difficulty in generating high-resolution audio from low-resolution audio.

[0049] To address at least one of the aforementioned issues in the related art, an embodiment of the present application proposes an audio data super-resolution method that constructs a variational autoencoder (VAE) in waveform space and applies a Schrödinger Bridge Model (SBM) in the VAE's latent space. This audio data super-resolution method is also known as an audio data processing method. Specifically, on the one hand, the embodiment of the present application constructs the VAE directly in the waveform space of the audio data, avoiding information loss during spectral conversion, simplifying the model structure, and improving reconstruction quality. On the other hand, the embodiment of the present application adopts an improved KL loss, and during the VAE training process, the loss term of the KL divergence is improved to obtain optimal reconstruction and generation capabilities. On the other hand, the embodiment of the present application utilizes the inherent similarity between low-resolution and high-resolution audio data in the waveform latent space, combined with the advantages of the Schrödinger Bridge Model, to establish an efficient data-to-data sampling path between the low-resolution and high-resolution audio data, thereby achieving high-quality audio data super-resolution.

[0050] The audio data processing method provided in the embodiment of the present application provides a solution for accurately and effectively generating target audio data with a second resolution, that is, an implementation process for accurately and effectively converting audio data with a low resolution (e.g., a first resolution) into target audio data with a high resolution (e.g., a second resolution). Specifically, in the audio data processing method provided in the embodiment of the present application, first, original audio data with a first resolution is obtained; then, the original audio data is encoded to obtain a latent representation vector of the original audio data in a latent space; then, the latent representation vector is vector-converted through a preset vector conversion path to obtain a conversion vector; finally, the conversion vector is decoded to obtain target audio data with a second resolution. In this way, by encoding the original audio data into a latent representation vector of a latent space, and then vector-converting the latent representation vector through a preset vector conversion path, a conversion vector capable of generating target audio data with a high resolution is obtained, thereby achieving accurate and effective conversion from low-resolution original audio data to high-resolution target audio data.

[0051] The audio data processing method provided in the embodiment of the present application can be implemented by an electronic device, which can be an audio data processing device. The audio data processing device (i.e., electronic device) provided in the embodiment of the present application can be implemented as a terminal or as a server. In one implementation, the audio data processing device provided in the embodiment of the present application can be implemented as any terminal with data processing capabilities, such as a laptop computer, a tablet computer, a desktop computer, an intelligent robot, etc.; in another implementation, the audio data processing device provided in the embodiment of the present application can also be implemented as a server, wherein the server can be an independent physical server or a server cluster or distributed system composed of multiple physical servers. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiment of the present application.

[0052] The audio data processing method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0053] Figure 1 This is an optional flow chart of the audio data processing method provided by the embodiment of the present application. The method can be applied to an electronic device, which can be a server. That is, the audio data processing method of each embodiment of the present application can be executed by the server, or can also be executed through interaction between the server and the terminal. The following will be explained exemplarily by taking the electronic device as the server. Figure 1 As shown, the method includes the following steps S101 to S104:

[0054] Step S101: Acquire original audio data with a first resolution.

[0055] Here, raw audio data refers to audio data that has not been processed or compressed and is directly collected from the source device.

[0056] Audio data is a digital signal that records sound and can be used in fields such as music, voice communication, and audio production. Raw audio data has the following attributes: sampling rate, number of channels, and format. The sampling rate refers to the number of samples taken per second. Examples include 44.1kHz (CD-quality audio), 48kHz (professional audio), and 96kHz (high-fidelity audio). Channels include mono, stereo, and multichannel (such as 5.1 and 7.1). Raw audio formats include lossless formats such as WAV, AIFF, and FLAC.

[0057] In an embodiment of the present application, when obtaining raw audio data having a first resolution, in order to obtain audio data having the first resolution (e.g., 48kHz / 24-bit resolution), a specific audio recording device may be used to record and obtain the raw audio data of the audio type. Alternatively, the raw audio data may be received from another electronic device; or the raw audio data may be downloaded from a specific website.

[0058] For raw audio files containing recorded original audio data, the audio data can be stored in lossless formats such as WAV, AIFF, or FLAC. These formats preserve the original audio quality and support audio data at 48kHz / 24-bit or higher resolutions. For example, in music production, a raw audio file may contain multiple tracks, each recorded at high resolution for precise adjustment during post-mixing and mastering.

[0059] The resolution of audio data is a key indicator used to describe the detail and quality of audio data. The original audio data obtained in the embodiment of the present application has a first resolution, which is related to factors such as sampling rate, quantization accuracy, and spatial resolution.

[0060] The primary determinant of the primary resolution of audio data is the sampling rate. The sampling rate refers to the number of times the audio signal is sampled per second, and is measured in Hertz (Hz). Sampling rates can include 44.1kHz, 48kHz, and 96kHz. The higher the sampling rate, the more accurately the frequency range of the audio signal can be captured. For example, a 44.1kHz audio signal is of CD quality and can meet most everyday listening needs; a 48kHz audio signal is a standard sampling rate for professional audio, providing a higher frequency range and better compatibility; and a 96kHz audio signal is high-fidelity audio, which can be used in high-end audio equipment and can capture richer high-frequency details.

[0061] The first resolution of the original audio data in the embodiment of the present application can be a relatively low resolution or an arbitrary resolution. In this way, through the audio data processing method provided in the embodiment of the present application, the original audio data with a low first resolution can be converted into target audio data with a higher second resolution.

[0062] Step S102: Encode the original audio data to obtain a latent representation vector of the original audio data in a latent space.

[0063] Encoding raw audio data here refers to the process of converting it into a more compact, efficient, or vector-friendly form. Encoding involves steps such as data preprocessing, feature extraction, and dimensionality reduction and compression. Through encoding, complex raw audio data can be mapped into a low-dimensional latent space, providing a more efficient representation for subsequent tasks.

[0064] During the encoding process, data preprocessing involves formatting, normalizing, cropping, or performing other necessary transformations on the raw audio data to make it suitable for subsequent encoding operations. The purpose of the data preprocessing step is to convert the raw audio data into a format suitable for model processing. For audio data, this can include converting the audio file to a uniform sampling rate, cropping the audio data into fixed-length segments, and normalizing the audio data to a specific numerical range.

[0065] The feature extraction step involves extracting key features from the raw audio data that effectively characterize the core information of the raw audio data. The purpose of the feature extraction step is to reduce the dimensionality of the raw audio data while retaining its key features. In this embodiment of the present application, features such as mel spectrum, cepstrum, and short-time energy can be extracted from the audio data.

[0066] Dimensionality reduction and compression involves mapping high-dimensional raw audio data to a lower-dimensional space through mathematical transformations or model learning, while preserving the original audio data's key information. This can be achieved through both linear and nonlinear methods. Linear methods include principal component analysis and linear discriminant analysis, while nonlinear methods include autoencoders, variational autoencoders, or generative adversarial networks. Encoders (such as autoencoders, variational autoencoders, or generative adversarial networks) encode raw audio data into latent representation vectors in a latent space. These latent representation vectors can be used for subsequent tasks, such as vector transformations. Autoencoder methods compress data into a low-dimensional representation through an encoder and then reconstruct the data through a decoder. Variational autoencoders encode data into the distribution parameters (mean and variance) of a latent space through an encoder and generate a latent representation vector through reparameterization techniques. Generative adversarial networks learn the data distribution and generate new data samples through adversarial training of a generator and a discriminator.

[0067] The following is a detailed description of the encoding process provided in the embodiment of the present application. Figure 2 , Figure 2 It is shown that the encoding process of the original audio data in step S102 can be implemented by the following steps S1021 and S1022:

[0068] Step S1021 , extracting an original feature vector from the waveform data of the original audio data; the dimension of the original feature vector is a preset dimension.

[0069] In an embodiment of the present application, extracting the original feature vector from the waveform data of the original audio data can be achieved in the following manner: inputting the original audio data into a pre-trained audio data encoder; then, extracting the original feature vector from the waveform data of the original audio data through the feature extraction layer of the audio data encoder; the potential number of channels of the audio data encoder is equal to the preset dimension.

[0070] Here, the waveform data of the original audio data refers to the representation of the original audio data in the time domain. For audio data, the waveform data is the amplitude of the sound signal changing over time. The audio waveform is a one-dimensional time series that represents the change in the amplitude of the sound signal over time.

[0071] Raw feature vectors are feature representations extracted from the waveform data of raw audio data, capturing the core information of the raw audio data. These raw feature vectors can be used for subsequent processing and analysis. The dimensionality of the raw feature vectors is pre-defined, determined during the design of the audio data encoder, and this dimensionality determines the size and complexity of the feature vectors.

[0072] An audio data encoder is a pretrained model that maps high-dimensional waveform data into a low-dimensional feature space. It can be implemented as a deep learning model, such as a convolutional neural network (CNN) or a recurrent neural network (RNN). The feature extraction layer in the audio data encoder is responsible for extracting useful features from the waveform data. These layers can include convolutional layers, pooling layers, and fully connected layers. The number of potential channels in the audio data encoder is equal to the preset dimensionality, meaning that the dimension of the output feature vector of the audio data encoder is fixed.

[0073] The process of extracting original feature vectors from waveform data of original audio data may include: inputting the original audio data into an audio data encoder; then, the audio data encoder processes the input original audio data through a feature extraction layer to extract original feature vectors. Since the feature vectors output by the audio data encoder are low-dimensional, they can capture the core features of the input audio data.

[0074] In an embodiment of the present application, when extracting the original feature vector from the waveform data of the original audio data, the audio data can be converted into a uniform sampling rate and quantization bit depth, and cropped into segments of fixed length. Then, feature extraction methods such as Mel spectrum and cepstrum are used to convert the waveform data of the audio into a feature vector.

[0075] The preset dimensions in the embodiments of the present application can be selected based on the specific task and data set. A higher dimension can capture more details but will increase the computational complexity.

[0076] Step S1022 : performing vector compression on the original feature vector using a preset compression ratio to obtain a latent representation vector of the original audio data in the latent space.

[0077] Vector compression is the process of mapping a high-dimensional feature vector to a low-dimensional space. This process aims to reduce the dimensionality of the feature vector while retaining as much of the original information as possible. The goal of vector compression is to generate a low-dimensional latent representation vector that effectively represents the core features of the original audio data in the latent space.

[0078] When compressing the original feature vector using a preset compression ratio, you can choose an appropriate compression method based on the task requirements and data characteristics. For example, for sequence data, you can use LSTM or GRU. You can also preset the compression ratio based on the task requirements, that is, determine the dimensionality of the low-dimensional latent representation vector. The compression ratio is typically a hyperparameter that can be adjusted experimentally. If using model-based methods (such as autoencoders, VAEs, and GANs), the model can be trained using a large amount of labeled or unlabeled data. During training, the low-dimensional representation is learned by optimizing an objective function (such as reconstruction error or KL divergence).

[0079] A latent space is a low-dimensional, continuous vector space into which the raw audio data is mapped by an audio encoder (e.g., a VAE encoder). In this vector space, the high-dimensional features of the raw audio data are compressed into a set of low-dimensional latent variables (or potential representation vectors) that capture the core features and structure of the raw audio data. The dimensionality of the latent space is typically much lower than that of the raw audio data.

[0080] A latent representation vector (LRV) represents a specific data point in a latent space. The LRV is a low-dimensional vector obtained by mapping the raw audio data into the latent space through an audio encoder. The LRV captures the core features and structure of the original audio data while removing redundant information. As a low-dimensional vector, the LRV represents the original audio data in a more compact form. Each dimension of the LRV corresponds to a semantic feature of the original audio data. For example, in the latent representation of raw speech data, certain dimensions may correspond to features such as intonation and speaking rate. Operations such as interpolation, weighting, and editing can be performed on the LRV in the latent space. These operations can be used to generate new data samples or modify certain features of existing data. The LRV can be decoded back to the original data space through a decoder to reconstruct the original data or generate new data samples. In an embodiment of the present application, raw feature vectors are extracted from the waveform data of the original audio data and vectorized using a preset compression ratio to obtain a latent representation vector. This process improves data processing efficiency and reduces computational complexity by obtaining the latent representation vector of the original audio data in a latent space. Because raw audio data is typically high-dimensional, for example, audio data may contain hundreds of thousands or even millions of samples. Directly processing this high-dimensional data results in extremely high computational complexity, making it difficult to process in real time or run on resource-constrained devices. By extracting low-dimensional raw feature vectors and further compressing them into a latent space, the data dimensionality can be significantly reduced. Low-dimensional representation vectors have lower computational complexity, speeding up subsequent processing and improving overall system efficiency. Furthermore, they can reduce storage requirements, as high-dimensional data requires a large amount of storage space. For example, a high-definition audio file may occupy several or even tens of GB of storage space. Through vector compression, the data can be compressed into a more compact representation, thereby reducing storage requirements. Furthermore, low-dimensional representation vectors require less data during transmission, improving data transmission efficiency and reducing bandwidth requirements. Furthermore, data interpretability and semantics can be enhanced. Using a pre-trained audio data encoder, semantically meaningful feature vectors can be extracted. These feature vectors capture core data characteristics, such as intonation and rhythm. The latent representation vectors obtained through vector compression exhibit enhanced semantics in the latent space. The latent space is an optimized low-dimensional space in which similar data points are close together and data points of different categories are effectively distinguished. Furthermore, low-dimensional latent representation vectors are easier to analyze and interpret, as each dimension may correspond to a specific semantic feature.

[0081] Step S103 , performing vector conversion on the potential representation vector through a preset vector conversion path to obtain a conversion vector.

[0082] Here, a vector transformation path refers to the implementation path that transforms the latent representation vector into another form (the transformation vector). In probabilistic generative models, a vector transformation path specifically refers to the random probability trajectory between two distributions.

[0083] In some embodiments, after encoding the original audio data to obtain a latent representation vector of the original audio data in the latent space, the latent representation vector may be scaled using a preset scaling factor to obtain a scaled vector. Accordingly, in step S103, the vector conversion of the latent representation vector using a preset vector conversion path may be performed by vector conversion of the scaled vector using the vector conversion path to obtain the converted vector.

[0084] Here, scaling refers to performing a multiplication operation on each dimension of the latent representation vector to adjust the scale of the latent representation vector. The scaling factor is a preset parameter that controls the degree of scaling. Scaling can be uniform (using the same scaling factor for all dimensions) or non-uniform (using different scaling factors for different dimensions). Scaling can adjust the scale of the latent representation vector to make it more suitable for subsequent processing steps. For example, in some machine learning models, the scale of the input data needs to be within a specific range (such as [0, 1] or [-1, 1]), and scaling can achieve this.

[0085] In an embodiment of the present application, by scaling the potential representation vector, the expressive power of some features in the potential representation vector can be enhanced. For example, if certain features are not significant enough at the original scale, appropriate scaling can make these features more influential in subsequent processing. Scaling can also adjust the potential representation vector to a scale that matches the model input. For example, some generative models may require the scale of the input vector to be within a specific range, and scaling can ensure this. Scaling can also avoid overflow or underflow problems in numerical calculations. For example, in the gradient descent process, excessively large eigenvalues ​​may cause the gradient to explode, while excessively small eigenvalues ​​may cause the gradient to disappear. Appropriate scaling can improve the stability of numerical calculations.

[0086] In an embodiment of the present application, the vector conversion path may include a vector conversion function, which is a specific mathematical function or model for realizing vector conversion. The vector conversion function receives a potential representation vector as input and outputs a conversion vector. In step S103, the vector conversion of the potential representation vector is performed through a preset vector conversion path, which can be achieved in the following way: the potential representation vector is subjected to random differential processing through the vector conversion function to obtain the conversion vector. Random differential processing refers to the introduction of randomness in the conversion process, which is usually achieved through a stochastic differential equation (SDE, Stochastic Differential Equation). This method can increase the diversity and flexibility of the conversion vector. In one implementation method, the potential representation vector is subjected to random differential processing through a vector conversion function, which can be achieved using a stochastic differential equation (such as Langevin dynamics).

[0087] In an embodiment of the present application, the vector conversion function may be a function in a Schrödinger bridge model. The Schrödinger Bridge model is a model based on probability and dynamics, which is used to establish an optimal transfer path between two probability distributions. The Schrödinger Bridge model originates from the Schrödinger equation in quantum mechanics, but is used to generate and convert data in machine learning. The Schrödinger Bridge model finds the optimal path by minimizing the relative entropy between the two distributions. In the vector conversion process, the Schrödinger Bridge model can be used to design a vector conversion function, which can be achieved by the following steps: first, define the initial and target distributions, and determine the initial distribution and target distribution of the potential representation vector (i.e., the distribution of the audio data at the second resolution); then, establish a transfer path, and use the Schrödinger Bridge model to establish the optimal transfer path from the initial distribution to the target distribution; finally, generate a conversion vector, and generate a conversion vector by sampling on the transfer path.

[0088] In embodiments of the present application, latent representation vectors are transformed using a predefined vector transformation path (such as the Schrödinger bridge model) to generate transformation vectors capable of reconstructing higher-resolution target audio data. This process involves stochastic differentiation and probabilistic optimization, which can improve data diversity and quality. This approach has broad application prospects in fields such as audio enhancement. Specifically, the vector transformation of latent representation vectors in embodiments of the present application can generate higher-resolution data because the resulting transformation vectors can reconstruct higher-resolution target audio data. That is, after decoder processing, the transformation vectors can reconstruct higher-resolution audio data. For example, high-resolution audio can be generated from the latent representation vectors of low-resolution audio. Data diversity can also be increased because the introduction of randomness allows for the generation of diverse transformation vectors, thereby increasing data diversity. By minimizing relative entropy, the Schrödinger bridge model can find the optimal transition path and generate high-quality transformation vectors. Furthermore, data quality can be improved. The Schrödinger bridge model optimizes the transition path to ensure that the generated transformation vectors are capable of better reconstructing the target data. Furthermore, the optimized path and stochastic differentiation process can reduce distortion during the reconstruction process, improving data quality.

[0089] Step S104 : decoding the conversion vector to obtain target audio data with a second resolution.

[0090] Here, the second resolution is greater than the first resolution. The transformed vector is decoded using a decoder to map the low-dimensional transformed vector back to the high-dimensional original data space, yielding the target audio data at the second resolution. In implementation, the transformed vector can be decoded using any of the following decoders: a fully connected decoder, a recurrent decoder, and a Transformer decoder. The fully connected decoder progressively maps the low-dimensional vector to a high-dimensional space through multiple fully connected layers. The recurrent decoder is suitable for sequential data, such as audio sequences, by progressively generating each element in the sequence using a recurrent neural network (such as a long short-term memory (LSTM) or a gated recurrent unit (GRU). The recurrent decoder captures temporal dependencies and generates temporally consistent high-resolution data. The Transformer decoder, based on the self-attention mechanism, is suitable for long-sequence data. It progressively generates data through multiple self-attention layers and feedforward network layers. The Transformer decoder captures global dependencies and generates high-quality, high-resolution data.

[0091] In an embodiment of the present application, a low-dimensional vector (conversion vector) after vector conversion is input into a decoder. This vector contains all the information required to generate high-resolution data. The decoder gradually maps the low-dimensional vector to a high-dimensional space through a series of layers. The decoder may involve but is not limited to the following operations: upsampling, feature extraction, activation processing of the activation function, and normalization processing. The high-resolution data output by the decoder can be high-resolution audio data (i.e., target audio data with a second resolution). For different data types, the output format is also different. For example, for audio data, the output is a high-sampling-rate audio waveform whose shape is (time step, number of channels).

[0092] The audio data processing method provided in an embodiment of the present application can, in order to process original audio data with a low resolution (for example, a first resolution) into target audio data with a high resolution (for example, a second resolution), first encode the original audio data with the first resolution to obtain a latent representation vector of the original audio data in a latent space; then, perform vector conversion on the latent representation vector through a preset vector conversion path, thereby achieving data conversion of the original audio data in the latent space and obtaining a conversion vector that also belongs to the latent space; finally, decode the conversion vector to obtain target audio data with a second resolution, which is greater than the first resolution. In this way, by encoding the original audio data into a latent representation vector in the latent space and then performing vector conversion on the latent representation vector through a preset vector conversion path, a conversion vector capable of generating target audio data with a high resolution is obtained, thereby achieving accurate and effective conversion from low-resolution original audio data to high-resolution target audio data.

[0093] The audio data processing method provided in the embodiments of this application has broad application value in the field of audio super-resolution. By improving the resolution and quality of audio data, the user experience and application effects can be significantly enhanced. The following examples illustrate the application scenarios of the audio data processing method provided in the embodiments of this application.

[0094] In audio super-resolution scenarios, the audio data processing method provided in the embodiments of this application can be used to input low-sampling-rate audio (first resolution) and output high-sampling-rate audio (second resolution). For example, in the field of audio processing, low-sampling-rate audio can be converted to high-sampling-rate audio to improve audio fidelity and clarity.

[0095] Below, the audio data processing method provided in the embodiment of the present application will be described in combination with any one of the above application scenarios. Figure 3This is another optional flow chart of the audio data processing method provided in the embodiment of the present application, such as Figure 3 As shown, the method includes the following steps S201 to S211:

[0096] Step S201: The terminal receives original audio data with a first resolution input by a user.

[0097] In step S202 , the terminal encapsulates the original audio data into a data processing request.

[0098] Step S203: The terminal sends a data processing request to the server.

[0099] Here, the original audio data can be encapsulated into a data processing request, and the terminal can send the data processing request to the server. In some embodiments, the terminal can send the data processing request using a protocol such as Hypertext Transfer Protocol (HTTP) or Web Socket. After receiving the data processing request, the server parses the data processing request and obtains the original audio data input by the user.

[0100] In step S204 , the server inputs the original audio data into a pre-trained audio data encoder in response to the data processing request.

[0101] In step S205, the server extracts original feature vectors from the waveform data of the original audio data through a feature extraction layer of the audio data encoder. The number of potential channels of the audio data encoder is equal to the preset dimension.

[0102] In step S206 , the server performs vector compression on the original feature vector using a preset compression ratio to obtain a latent representation vector of the original audio data in the latent space.

[0103] Step S207: The server scales the latent representation vector using a preset scaling factor to obtain a scaled vector.

[0104] Step S208: The server performs random differentiation processing on the scaling vector using a vector conversion function to obtain a conversion vector.

[0105] Step S209: The server decodes the conversion vector to obtain target audio data with a second resolution; the second resolution is greater than the first resolution.

[0106] Step S210: The server sends target audio data with a second resolution to the terminal.

[0107] Step S211: The terminal plays the target audio data with the second resolution.

[0108] In some embodiments, the above-mentioned audio data processing method can also be implemented by a pre-trained data processing model, that is, the original audio data with a first resolution can be input into the pre-trained data processing model, and the target audio data with a second resolution can be obtained by the pre-trained data processing model. Based on this, the embodiment of the present application further provides a data processing model training method, Figure 4 This is a flow chart of the training method of the data processing model provided in the embodiment of the present application. The training method of the data processing model can be implemented by a model training module. The model training module can be a module in an electronic device. That is, the execution subject of the training method of the data processing model can be a server or a terminal. The embodiment of the present application takes the execution subject of the training method of the data processing model as an example. Figure 4 As shown, the method includes the following steps S301 to S307:

[0109] Step S301: Obtain sample data.

[0110] The sample data includes sample audio data having a first resolution and reference audio data having a second resolution.

[0111] Step S302: input the sample audio data into the data processing model to be trained.

[0112] Step S303: Encode the sample audio data through the audio data encoder of the data processing model to be trained to obtain a sample latent representation vector of the sample audio data in the latent space.

[0113] It should be noted that in step S303, the sample audio data is encoded by the audio data encoder of the data processing model to be trained. The sample original feature vector can be first extracted from the waveform data of the sample audio data by the audio data encoder of the data processing model to be trained; the dimension of the sample original feature vector can be a preset dimension; then, the sample original feature vector is vector compressed using a preset compression ratio to obtain a sample latent representation vector of the sample audio data in the latent space.

[0114] Step S304: Perform vector conversion on the sample potential representation vector through the vector converter of the data processing model to be trained to obtain a sample conversion vector.

[0115] It should be noted that the vector conversion of the sample latent representation vector in step S304, performed by the vector converter (which can be implemented as a Schrödinger bridge model) of the data processing model to be trained, is implemented in the same manner as the vector conversion of the latent representation vector via the preset vector conversion path in step S103 above. The vector conversion process in step S304 can be understood by referring to the relevant explanation of step S103 above. The only difference is that the object of vector conversion in step S304 is the sample latent representation vector, while the object of vector conversion in step S103 is the latent representation vector.

[0116] Step S305 : decoding the sample conversion vector by the audio data decoder of the data processing model to be trained to obtain predicted audio data with a third resolution.

[0117] It should be noted that the decoding of the sample conversion vectors by the audio data decoder of the data processing model to be trained in step S305 is implemented in the same manner as the decoding of the conversion vectors in step S104 above. The decoding process in step S305 can be understood by referring to the relevant explanation of step S104 above. The only difference is that the decoding process in step S305 is performed on the sample conversion vectors, while the decoding process in step S104 is performed on the conversion vectors.

[0118] Step S306 , performing loss calculation based on the predicted audio data and the reference audio data to obtain a loss result.

[0119] In some embodiments, see Figure 5 , Figure 5 It is shown that the loss calculation based on the predicted audio data and the reference audio data in step S306 can be implemented by the following steps S3061 and S3065:

[0120] Step S3061 , in response to the current model training process being to train at least one of the audio data encoder and the audio data decoder, a reconstruction error value is determined based on the predicted audio data and the reference audio data.

[0121] Step S3062: Determine the probability distribution error value based on the mean and variance of the sample potential representation vector.

[0122] Step S3063: Determine a multi-resolution error value based on the transformed data of the predicted audio data and the reference audio data at different resolutions.

[0123] Step S3064: Determine multi-period loss values ​​of the predicted audio data and the reference audio data in different periods.

[0124] Step S3065: Fusing the reconstruction error value, the probability distribution error value, the multi-resolution error value, and the multi-cycle loss value to obtain a loss result.

[0125] In an embodiment of the present application, if the current model training process is to train at least one of the audio data encoder and the audio data decoder, the reconstruction error value, the probability distribution error value, the multi-resolution error value and the multi-cycle loss value can be calculated respectively, and then, based on the calculated reconstruction error value, the probability distribution error value, the multi-resolution error value and the multi-cycle loss value, the total error value is determined, and the total error value is determined as the loss result of the model training, so that the model parameters in at least one of the audio data encoder and the audio data decoder are updated based on the loss result.

[0126] In other embodiments, see Figure 6 , Figure 6 The loss calculation based on the predicted audio data and the reference audio data in step S306 is shown, which can also be achieved by the following steps S3066:

[0127] Step S3066, in response to the current model training process being to train the vector converter, a loss result of the vector converter is determined based on the predicted audio data and the reference audio data.

[0128] In an embodiment of the present application, if the current model training process is to train a vector converter, after obtaining the loss result, the model parameters in the vector converter can be updated based on the loss result.

[0129] It should be noted that, in the training process of the data processing model in the embodiment of the present application, the audio data encoder and the audio data decoder may be trained first, and then the vector converter may be trained. When training the audio data encoder and the audio data decoder, the audio data encoder and the audio data decoder may be trained simultaneously, or the audio data encoder may be trained first, and then the audio data decoder may be trained.

[0130] In an embodiment of the present application, the method of simultaneously training the audio data encoder and the audio data decoder can achieve end-to-end learning, so that the entire data processing model can automatically adjust the parameters of the audio data encoder and the audio data decoder to minimize the reconstruction error. This training method can ensure the coordinated optimization between the audio data encoder and the audio data decoder and improve the overall performance. In addition, end-to-end training can optimize the global performance of the entire data processing model, rather than the local optimum. The parameter adjustment of the audio data encoder and the audio data decoder is based on the loss function of the entire data processing model, which can better capture the global characteristics of the input data. By training simultaneously, a large amount of training data can also be better utilized to reduce the risk of overfitting. The parameter adjustment of the audio data encoder and the audio data decoder is based on the entire data set, rather than a local data set. In addition, end-to-end training can reduce training time because there is no need to train the audio data encoder and the audio data decoder separately. For example, in the image super-resolution task, simultaneously training the audio data encoder and audio data decoder can ensure that the generated images have higher quality and fewer artifacts; in the audio enhancement task, simultaneously training the audio data encoder and audio data decoder can ensure that the generated audio has higher fidelity and less noise; in the video super-resolution task, simultaneously training the audio data encoder and audio data decoder can ensure that the generated video has higher visual quality and fewer artifacts.

[0131] In an embodiment of the present application, a method of first training the audio data encoder and then training the audio data decoder, by first training the audio data encoder, can ensure that the audio data encoder can effectively extract the core features of the data. Then, based on these features, training the audio data decoder can ensure that the audio data decoder can better reconstruct the data. Training the audio data encoder first can reduce the complexity of the training process. The training of the audio data encoder can be carried out independently without considering the parameter adjustment of the audio data decoder, which can simplify the training process and reduce training time. Training the audio data encoder first can also improve the stability of the training process, because the parameter adjustment of the audio data encoder is based on the entire data set, rather than a local data set, so it can reduce fluctuations in the training process and improve the stability of the training. In addition, by training the audio data encoder first, the expressive power of the features can also be enhanced. The audio data encoder can learn more effective feature representations and provide better input for the audio data decoder.

[0132] In an embodiment of the present application, a method of first training the audio data encoder and the audio data decoder and then training the vector converter can ensure the coordinated optimization between the audio data encoder and the audio data decoder by first training the audio data encoder and the audio data decoder. Then, the vector converter is trained based on these optimized audio data encoders and audio data decoders to ensure that the vector converter can better process the potential representation vector. At the same time, through staged training, the complexity of the training process can be reduced, and the training of the audio data encoder and the audio data decoder can be carried out independently without considering the parameter adjustment of the vector converter, which can simplify the training process and reduce training time. In addition, through staged training, the stability of the training process can also be improved, because the parameter adjustment of the audio data encoder and the audio data decoder is based on the entire data set, rather than a local data set, which can reduce fluctuations in the training process and improve the stability of the training.

[0133] Step S307: Update the model parameters in the data processing model to be trained based on the loss result to obtain a trained data processing model.

[0134] In an embodiment of the present application, if the current model training process is to train at least one of the audio data encoder and the audio data decoder, the loss result obtained is the loss result obtained by fusing the reconstruction error value, the probability distribution error value, the multi-resolution error value and the multi-cycle loss value, and the model parameters in at least one of the audio data encoder and the audio data decoder can be updated based on the loss result.

[0135] If the current model training process is to train the vector converter, the obtained loss result is the loss result used to train the vector converter, so the model parameters in the vector converter can be updated based on the loss result.

[0136] The data processing model training method provided in the embodiments of the present application calculates different loss function terms when training different training objects. For example, when training an audio data encoder and an audio data decoder, the loss result is a fusion of the reconstruction error value, the probability distribution error value, the multi-resolution error value, and the multi-period loss value. By fusing multiple error values, the data processing model not only focuses on the global reconstruction error but also considers the distribution of the latent representation, errors at different resolutions, and time periods. This makes the reconstructed data closer to the original data in multiple dimensions. Furthermore, the introduction of multi-resolution error values ​​and multi-period loss values ​​makes the data processing model more robust when processing data of different scales and time periods, reducing errors caused by scale changes or time fluctuations. The introduction of the probability distribution error value brings the latent representation vector closer to the preset distribution, improving the smoothness and continuity of the latent space and facilitating subsequent vector conversion and generation tasks. By comprehensively considering multiple errors, the data processing model can better generalize to unseen data during training, reducing the risk of overfitting. For another example, when training a vector transformer, the loss result is determined based on the predicted audio data and the reference audio data. In this way, by directly comparing the predicted data (i.e., predicted audio data) and the reference data (i.e., reference audio data), the vector converter can learn a more accurate vector conversion path, and the generated high-resolution data is closer to the real data. And the design of the loss function can ensure that the generated data is semantically consistent with the reference data, avoiding the generation of unreasonable artifacts or distortion. At the same time, the loss function based on the predicted data and the reference data can provide more direct feedback, making the training of the vector converter more efficient and stable. Of course, in some embodiments, different parameter data can also be introduced. In this way, by introducing a variety of reference data during the training process, the vector converter can better adapt to different input conditions and improve the generalization ability of the vector converter.

[0137] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0138] The present embodiment provides an audio data processing method. This audio data processing method is implemented using an audio data processing model, which can also be referred to as a latent bridge model (LBM). The latent bridge model proposed in this embodiment is based on the Schrödinger bridge model and is modeled in the latent space of a VAE, distinguishing itself from AudioSR, which processes audio in the mel-spectrogram space. The latent bridge model constructs the VAE directly in waveform space and improves the KL loss of the VAE to achieve optimal reconstruction and generation capabilities. Furthermore, by leveraging the inherent similarity between low-resolution and high-resolution audio data in the latent space of waveforms, the latent bridge model combines the advantages of the Schrödinger bridge model to achieve high-quality audio data super-resolution. Compared to AudioSR, the latent bridge model has a simpler network structure, avoiding the problems of reduced reconstruction quality and network complexity that may be associated with the cascade paradigm. Furthermore, the latent bridge model excels in preserving low-frequency information and generating high-frequency details, without the need for additional post-processing steps. Experimental results show that the latent bridging model significantly outperforms AudioSR in terms of LSD, LSD-LF, LSD-HF and Visqol in zero-shot tests on the ESC-50 and VCTK datasets, demonstrating the excellent performance of the latent bridging model in audio data super-resolution tasks.

[0139] Below, the potential bridging model (i.e., the above-mentioned audio data processing model) and the audio data processing method of the embodiment of the present application are described.

[0140] This embodiment provides a general audio super-resolution system based on a potential bridging model. The above-mentioned audio data processing method is implemented through the audio super-resolution system. The audio super-resolution system is suitable for super-resolution tasks of various audio signals such as speech and music. The overall architecture of the audio super-resolution system is as follows: Figure 7 As shown, it includes an audio input module 701, a preprocessing module 702, a VAE encoder 703 (corresponding to the above-mentioned audio data encoder), a Schrödinger bridge model 704 (corresponding to the above-mentioned vector converter), a VAE decoder 705 (corresponding to the above-mentioned audio data decoder) and an audio output module 706.

[0141] During implementation, the audio super-resolution system first receives a low-resolution audio signal (this low-resolution audio signal can have a low sampling rate, and the low resolution can be, for example, the first resolution described above) from a mobile device (e.g., a terminal) via an audio input module 701. The preprocessing module 702 normalizes and denoises the input audio signal to produce a preprocessed audio signal, thereby improving the stability and accuracy of subsequent processing. The preprocessed audio signal is then input into a VAE encoder 703, which converts the time-domain waveform of the preprocessed audio signal into a latent space, generating a corresponding latent representation vector.

[0142] In the latent space, the audio super-resolution system utilizes a Schrödinger bridge model 704 to establish a direct mapping between low-resolution and high-resolution latent representations. By optimizing a stochastic differential equation (SDE), the Schrödinger bridge model 704 learns the optimal sampling path from the low-resolution latent representation vector to the high-resolution latent representation vector. During the training process of this Schrödinger bridge model 704, a modified KL divergence loss function can be employed to ensure that the generated high-resolution latent representation vector preserves the low-frequency information of the original audio signal while enhancing the reconstruction quality of high-frequency details.

[0143] After training, during the inference phase, the audio super-resolution system maps the low-resolution audio signal into a latent space via the VAE encoder 703 and generates the corresponding high-resolution latent representation vector (the aforementioned transformation vector) using the Schrödinger bridge model 704. Subsequently, the VAE decoder 705 converts the high-resolution latent representation vector into a time-domain waveform, resulting in a high-sampling-rate audio signal. Finally, the audio output module 706 performs post-processing on the generated high-resolution audio signal, including denoising and smoothing, to improve audio quality.

[0144] The audio super-resolution system of the embodiment of the present application can be deployed on the server side, receiving low-resolution audio signals uploaded by terminal devices through a network interface, and returning high-resolution audio signals to the terminal devices after processing. Terminal devices may include smartphones, personal computers, tablets, etc. Through this audio super-resolution system, users can efficiently convert low-resolution audio files into high-quality, high-resolution audio, which is suitable for application scenarios such as music streaming, voice communication, or intelligent voice assistants.

[0145] The embodiment of the present application addresses the following technical problem 1: The VAE model in the related art has deficiencies in reconstruction quality and potential representation structure, and cannot effectively support high-quality super-resolution generation. The VAE modeling method of optimizing structure and scaling loss term is adopted to solve this technical problem 1. Specifically, the embodiment of the present application compresses the original audio signal into a compact potential representation vector by constructing a VAE encoder in the waveform space. In the latent space, the structure of the original signal is preserved, providing an interpretable basis for subsequent Schrödinger bridge modeling. In addition, in order to optimize the performance of VAE in balancing reconstruction quality and generation bridging capabilities, the embodiment of the present application also systematically adjusted and compared the KL divergence term. Experiments have found that reducing the KL term to 1e -7 The magnitude of can significantly improve the reconstruction quality and sampling controllability of the latent space without sacrificing distribution constraints.

[0146] The embodiment of the present application addresses the following technical problem 2: Modeling the bridge model in the related art directly in the VAE space may cause instability and inefficiency. In order to further enhance the stability of the modeling, the embodiment of the present application proposes to scale the latent space by 0.25 times to compress the spatial variance and reduce the drift during training and sampling. In terms of modeling configuration, a VAE encoder structure with a compression ratio of 512 is selected, audio with a sampling rate of 48kHz is used as input, and latent variables with 64-dimensional channels are used as the modeling basis. This configuration compresses the computational overhead while retaining the structural features, providing a high-quality and easy-to-sample latent space distribution for the Schrödinger bridge model.

[0147] The universal audio super-resolution method based on the latent Schrödinger bridge model proposed in the embodiment of the present application introduces Schrödinger bridge modeling in the latent space learned by the variational autoencoder to construct the optimal probabilistic path between low-resolution and high-resolution latent representations, thereby achieving more efficient and higher-quality audio upsampling. Figure 8 This is a schematic diagram of the implementation flow of the audio super-resolution method provided in the embodiment of the present application, see Figure 8 , the method comprises the following steps:

[0148] Step S401: construct an encoder network from waveform space to latent space (i.e., the above-mentioned VAE encoder).

[0149] Here, we can first obtain a pair of low-resolution audio samples in the training dataset And the high-resolution audio corresponding to the low-resolution audio sample Then, the low-resolution audio samples are transformed into and high-resolution audio The waveforms are mapped to the latent space to obtain the latent representation vector of the low-resolution audio sample and the latent representation vector for high-resolution audio , as shown in the following formulas (1) and (2):

[0150] (1);

[0151] (2).

[0152] To improve modeling stability, a scaling factor can be applied here Normalize the latent space features uniformly to obtain the normalized potential representation vector , see the following formula (3) for details:

[0153] (3).

[0154] Here, the latent representation vector The latent representation vector including the above low-resolution audio samples and the latent representation vector for high-resolution audio .

[0155] The embodiment of the present application uses a VAE encoder with a compression ratio of 512 and sets the number of potential channels to 64, which retains key audio information while ensuring a high compression rate, providing a good foundation for subsequent bridging modeling.

[0156] Step S402 : constructing a Schrödinger bridge generation path (corresponding to the above-mentioned vector conversion path) in the latent space.

[0157] Here, first, in the latent space, we construct a (prior value) to (target value), the forward process is defined as a stochastic differential process (SDE), see the following formula (4):

[0158] (4);

[0159] in, represents the hidden state at time t, is the drift function, is the diffusion coefficient, represents standard Wiener noise.

[0160] During the training process, the optimal path starting from the low-resolution latent representation vector and targeting the high-resolution latent representation vector is fitted, so that the sampling process follows the data-to-data transition path rather than the traditional noise-to-data path, thereby improving sampling quality and efficiency.

[0161] Step S403: model training and loss function design.

[0162] In the embodiment of the present application, during the training of the VAE module (the VAE module here includes the above-mentioned VAE encoder and VAE decoder), the following objective function (5) can be used for optimization:

[0163] (5);

[0164] in, represents the total loss function of the VAE module (i.e., the loss result during the training process of at least one of the audio data encoder and the audio data decoder), It consists of several different loss terms, each of which has a weight coefficient. The following is an explanation of each parameter in formula (5): Represents the weight coefficient of the reconstruction loss in the waveform domain; Represents the reconstruction loss (i.e., the reconstruction error value mentioned above), which is used to measure the difference between the output reconstructed by the VAE module and the original input. Represents the weight coefficient of KL divergence loss; Represents the KL divergence loss (i.e., the above-mentioned probability distribution error value), which is used to measure the difference between the potential distribution output by the VAE encoder and the standard normal distribution. It should be noted that the above-mentioned KL divergence loss in the embodiment of the present application is used to constrain the distribution of latent variables to be close to the standard normal distribution. In order to balance the modeling ability and the bridging performance, the embodiment of the present application can be set =1e -7 , and supports arbitrary forms of distance metrics for reconstruction loss, such as L2 loss and perceptual loss. Represents the weight coefficient of the multi-resolution short-time Fourier transform (STFT) loss; Represents the multi-resolution STFT loss (i.e., the multi-resolution error value mentioned above), which is used to measure the difference between audio signals at different frequency resolutions. Represents the weight coefficient of multi-period loss; represents the multi-period loss (i.e., the multi-period loss value mentioned above), which measures the differences in the audio signal at different periods. These loss terms work together to enable the VAE encoder to learn the latent distribution of the input audio signal, thereby obtaining a latent representation vector of the input audio signal in latent space and ultimately generating output audio that is similar to the original input audio. The weight coefficients balance the contributions of the different loss terms and can be adjusted based on the specific task.

[0165] In the embodiment of the present application, the above reconstruction loss It can be calculated by the following formula (6):

[0166] (6)

[0167] Reconstruction loss It measures the difference between the reconstructed output of the VAE module and the original input. represents the original input data (ie, the input audio signal, the original audio data), and the subscript 48 indicates that the resolution of the input audio signal may be the resolution corresponding to the 48 kHz sampling rate. Represents the output data reconstructed by the VAE module (i.e., the audio signal output by the VAE decoder), which has the same sampling rate as the original input data. = indicates that the square of the L2 norm (Euclidean distance) is used to calculate the difference. Therefore, formula (6) represents the original input data With the reconstructed output data The L2 norm square difference between the original input data With the reconstructed output data The square of the Euclidean distance between them. Reconstruction loss The purpose is to enable the VAE module to reconstruct an output that is as similar as possible to the original input.

[0168] In the embodiment of the present application, the potential representation vector z can be expressed by the following formula (7): It can be expressed by the following formula (8):

[0169] (7);

[0170] (8)

[0171] Formula (7) and Formula (8) describe the encoding and decoding process of the VAE module, where Formula (7) represents the role of the VAE encoder. The VAE encoder is a function that converts the original input data into Mapped to a latent representation vector z in a latent space. Represents the parameters of the VAE encoder, which are optimized during the training process so that the input data can be effectively encoded into the latent space. Formula (8) represents the role of the VAE decoder (Decoder), which is also a function that maps the potential representation vector z in the latent space back to the space of the original input data to generate the reconstructed output data . Represents the parameters of the VAE decoder, which are optimized during training to be able to generate output data from the latent space that is similar to the original input data.

[0172] During the training process of the VAE module, the parameters of the VAE encoder and VAE decoder are jointly optimized to minimize the loss function. Through this process, the VAE module is able to learn the latent distribution of the input data and generate output data similar to the original input.

[0173] In the embodiment of the present application, the above KL divergence loss It can be calculated by the following formula (9):

[0174] (9)

[0175] KL divergence loss It measures the difference between the potential distribution of the VAE encoder output (that is, the potential representation vector output by the VAE encoder) and the standard normal distribution. The mean of the latent representation vector z. Denotes the variance of the latent representation vector z. D denotes the dimension of the latent representation vector z. The summation symbol ∑ in formula (9) denotes the summation of the latent representation vectors of all dimensions. KL divergence loss The purpose is to make the distribution of the potential representation vector z as close as possible to the standard normal distribution, that is, a normal distribution with a mean of 0 and a variance of 1. By minimizing the KL divergence loss, the VAE encoder can learn the potential distribution of the original input data and generate outputs similar to the original input data.

[0176] In the embodiment of the present application, the multi-resolution STFT loss It can be calculated by the following formula (10):

[0177] (10)

[0178] Multi-resolution STFT loss It measures the difference between the original input data and the reconstructed output data at different frequency resolutions. Where N represents the number of different resolutions of the STFT. Represents the original input data STFT at the i-th resolution. Represents the reconstructed output data STFT at the i-th resolution. Therefore, Equation (10) represents the original input data and the reconstructed output data The sum of the L2-norm squared differences between STFTs at different resolutions. Multi-resolution STFT loss The purpose is to make the reconstructed output data as close as possible to the original input data at different frequency resolutions, thereby improving the quality of the reconstructed signal.

[0179] In the embodiment of the present application, the multi-cycle loss It can be calculated by the following formula (11):

[0180] (11)

[0181] Multi-period loss It measures the difference between the original input data and the reconstructed output data at different periods. Represents the original input data After the output of the Multi-Period Discriminator (MPD). Represents the reconstructed output data The output of the multi-cycle discriminator. Therefore, formula (11) represents the original input data and the reconstructed output data The L2-norm squared difference between the outputs of the multi-cycle discriminator. Multi-cycle loss The purpose is to make the reconstructed output data as close as possible to the original input data at different periods, thereby improving the quality and authenticity of the reconstructed signal.

[0182] Here, a multi-period discriminator is a discriminator used in audio signal processing. It captures the periodic characteristics of a signal by analyzing it at different periods. In the context of the VAE module, the multi-period discriminator can be used to improve the quality of the generated audio signal, making it closer to the real audio signal.

[0183] In the embodiment of the present application, during the training of the Schrödinger bridge model, a unified generation objective function can be used. , measures the difference between the predicted trajectory and the target potential distribution, where the loss function of the Schrödinger bridge model is It is expressed as the following formulas (12) to (14):

[0184] (12);

[0185] (13);

[0186] in, (14);

[0187] Among them, formula (12) represents the loss function of the Schrödinger bridge model. The loss function of the Schrödinger bridge model is It measures the difference between the predicted trajectory and the target potential distribution. Specifically, Indicates the distribution of data Sampling , and, through the filter (Filter) of the sample The data obtained after filtering Perform expectation calculations. Indicates the expected calculation for time step t. Represents the prediction function of the Schrödinger bridge model. The prediction function of the Schrödinger bridge model uses parameters θ, current state , time step t and initial state to predict the target state. represents the target state. Therefore, the loss function of the Schrödinger bridge model is Indicates the predicted status and target state The purpose of this loss function is to enable the Schrödinger bridge model to accurately predict the target state, thereby improving the prediction performance of the Schrödinger bridge model.

[0188] Formula (13) means that the data Encode through the VAE encoder to obtain the potential representation vector in the latent space Specifically, is a vector in the latent space that represents the data encoded by the VAE encoder. k is an index value that belongs to the set (t, 0, 1) and represents different time steps or data points. Indicates that the data is encoded using VAE In the VAE module, the VAE encoder is a function that converts the original input data into A latent representation vector mapped into a latent space This latent representation vector It can be a mean and a variance that define the normal distribution of the latent variable. Through this encoding process, the VAE encoder is able to learn the latent distribution of the original input data, so that it can generate output data similar to the original input.

[0189] Formula (14) describes the potential representation vector in the Schrödinger bridge model Specifically, and represents the parameters in the Schrödinger bridge model, and Used to define the latent representation vector distribution. Indicates the mean , the covariance matrix is The normal distribution of , where, in formula (14), the mean , the covariance matrix . and are all parameters in the Schrödinger bridge model. and Used to calculate the latent representation vector The mean of . and It is also a parameter in the Schrödinger bridge model. and Used to calculate the latent representation vector The variance of . and represents the initial and target states in the Schrödinger bridge model. It is also a parameter in the Schrödinger bridge model. Used to calculate the latent representation vector Therefore, Formula (14) expresses the potential representation vector The probability distribution of is a normal distribution, whose mean and variance are determined by the parameters of the Schrödinger bridge model and the initial and target states. This normal distribution is used for prediction and inference in the Schrödinger bridge model.

[0190] In some embodiments, the loss function of the Schrödinger bridge model is For LBM, we directly optimize the L2 loss between the noise prediction at trajectory time step t and the actual noise at step t. The goal is to make The sampling process starting from (prior value) naturally converges to (target value, i.e., the latent representation vector of the reference audio data).

[0191] Step S404: sampling and evaluation analysis.

[0192] In the inference phase, the audio super-resolution system first inputs low-resolution audio (corresponding to the original audio data with the first resolution mentioned above) to the VAE encoder, and the low-resolution potential representation vector is obtained by the VAE encoder. , a high-resolution potential representation vector is obtained by sampling the Schrödinger bridge model , expressed as the following formula (15):

[0193] (15);

[0194] in, It represents the sampling process of the Schrödinger bridge model, wherein, in the embodiment of the present application, 50-step sampling can be adopted.

[0195] Then pass the VAE decoder The high-resolution latent representation vector Restore to high-resolution audio waveform , as shown in the following formula (16):

[0196] (16).

[0197] Figure 9 This is a comparison diagram of the technical implementation process of LBM provided in the embodiment of the present application and the technical implementation process of AudioSR in the related art. Figure 9 The upper part is the technical implementation process of AudioSR. It can be seen that AudioSR uses a double-cascade compression annotation paradigm; Figure 9 The lower part is the technical implementation process of LBM. It can be seen that LBM directly establishes a Schrödinger bridge mapping between waveform latent spaces.

[0198] Furthermore, based on the training process of the audio data processing model provided in the above embodiment, the embodiment of the present application provides an any-to-any training method.

[0199] Because the training dataset (i.e., the sample data described above) contains relatively few high-sampling-rate audio, directly training a 48kHz audio data processing model using only 48kHz audio results in suboptimal model training. Training the audio data processing model with audio at various sampling rates, while increasing the amount of data, significantly reduces training quality (because most datasets contain low-sampling-rate data). Therefore, the present invention proposes an any-to-any training paradigm.

[0200] Specifically, during training, the sampling rate of the low-resolution audio and the actual frequency of the high-resolution audio (sampling rate divided by 2) are both used as additional inputs to the audio data processing model to be trained. In this way, the understanding of the frequencies of the two different boundary distributions enables LBM to clearly learn the any-to-any upsampling process of different frequency bands, thereby improving the upsampling performance to the target resolution sampling stage. Therefore, the loss function when training the audio data processing model is It can be expressed as the following formula (17):

[0201] (17).

[0202] in, represents the parameters of the Schrödinger bridge model; Represents the potential representation vector in the latent space of relatively high-resolution audio obtained from any to any training method, Represents the potential representation vector obtained after the relatively low-resolution audio in any-to-any training method is encoded by the VAE encoder; represents the diffusion time step of the Schrödinger bridge model; T represents the maximum value of the diffusion time step of the Schrödinger bridge model; The potential representation vector representing the original input audio at time step t of the Schrödinger bridge trajectory; Indicates the frequency of the input audio, Indicates the frequency of the target audio. and They represent the coefficients related to the offset and diffusion of the random process in the forward process of the Schrödinger bridge model in the audio data processing model.

[0203] The audio data processing method provided in this application introduces a Schrödinger bridge generation path into the latent space of a VAE module, avoiding the lengthy sampling path from Gaussian noise. This effectively shortens the generation trajectory and improves the sampling efficiency and detail of the data processing model in high-resolution audio reconstruction. Experimental evaluations were conducted on common audio tasks such as Environmental Sound Classification (ESC-50) and the Voice Cloning Toolkit (VCTK), including zero-shot testing scenarios. Evaluation metrics include full-band log spectral distance (LSD), low-frequency (LSD-LF), high-frequency (LSD-HF), scale-invariant signal-to-noise ratio (SI-SNR), and the perceptual speech quality score (ViSQOL). Experimental results demonstrate that the method of the present embodiment significantly outperforms existing AudioSR cascaded approaches in terms of reconstruction accuracy, detail restoration, and frequency consistency. Table 1 compares the high-resolution audio reconstruction results from 8kHz to 48kHz using the method of the present embodiment with those from existing AudioSR methods, using various evaluation metrics. Table 2 compares the high-resolution audio reconstruction results from 12kHz to 48kHz using the method of the present embodiment with those from existing AudioSR methods, using various evaluation metrics. As can be seen, the method of the present embodiment outperforms existing AudioSR methods in terms of LSD (lower is better), LSD-LF (lower is better), LSD-HF (lower is better), SI-SNR (higher is better), and WLS (lower is better), for both the 8kHz to 48kHz and 12kHz to 48kHz sampling rates.

[0204] Table 1

[0205]

[0206] Table 2

[0207]

[0208] The embodiment of the present application systematically optimizes the structure of the VAE module potential space and proposes a KL scaling strategy (scaling to 1e -7 Refined designs such as the level (0.25), latent space scaling (0.1), and compression ratio (512) effectively improve the modelability and bridge stability of the coding space. This structural optimization directly promotes the bridge modeling effect, making the Schrödinger bridge model more stable and more convergent in the latent space, resulting in more natural and smooth audio generation quality.

[0209] In summary, this embodiment of the present application proposes a method for integrating the latent space learned by the VAE module with the Schrödinger bridge model in audio super-resolution tasks. This method implements a data-to-data bridging sampling path between low-resolution and high-resolution, effectively improving audio generation efficiency and restoration quality. This method, in turn, implements the combined application of the VAE module and the Schrödinger bridge model for general audio super-resolution tasks. To address the issue of uncontrollable latent space, this embodiment of the present application also designs a scaling KL loss and latent representation scaling scheme, which enhances the sampling stability and structural balance of the latent vector, forming the foundation for a high-quality latent space suitable for bridging modeling. This embodiment of the present application embeds the Schrödinger bridge sampling process within the latent space of the VAE module and, leveraging the tight coupling of the prior distribution and the target distribution, constructs a low-noise, low-trajectory energy sampling path, improving generation performance and reducing generation error. Furthermore, this embodiment of the present application selects a 512x compression ratio and a 64-channel structure as the encoder configuration, balancing lightweight representation capabilities while providing robust cross-domain generalization for general audio (including speech, music, and ambient sound). At the same time, the embodiments of the present application demonstrate strong generalization capabilities on multiple speech and non-speech datasets, and are suitable for tasks such as high-sampling rate music restoration, historical audio restoration, speech enhancement and synthesis, and cross-scenario speech understanding.

[0210] In some embodiments, the aforementioned audio data processing method can also be implemented as an alternative to traditional diffusion models. While traditional diffusion models (such as NU-Wave 2) have achieved some success in speech super-resolution tasks, these models may have limitations when processing complex audio signals such as music and sound effects, particularly in terms of generating and inferring high-frequency details. In contrast, the Schrödinger bridge model, through its data-by-data sampling path, can more effectively capture and reconstruct the complex harmonics and transient details in music signals, improving both generation efficiency and quality.

[0211] In some embodiments, the above-mentioned audio data processing method can also be implemented as an alternative based on a generative adversarial network. Since generative adversarial networks are also widely used in the field of music signal processing, however, the training process of generative adversarial networks may be unstable and prone to artifacts. For super-resolution tasks of music and sound effects, generative adversarial networks may require more complex network architectures and longer training times. In contrast, the Schrödinger bridge model provides a stable and efficient training and generation mechanism suitable for super-resolution tasks of high-quality music signals.

[0212] In some embodiments, the aforementioned audio data processing methods can also be implemented as an alternative to frequency-domain methods. Frequency-domain methods process high-frequency information by performing frequency-domain transforms (such as short-time Fourier transforms) on the audio signal. However, these methods can introduce artifacts during the inverse transform process, affecting the generated quality. The Schrödinger bridge model can sample directly in the time domain, avoiding the errors and computational overhead associated with frequency-domain transforms, making it more suitable for super-resolution tasks for music and sound effects.

[0213] The following is an analysis and enumeration of the application scenarios of the audio data processing method provided in the embodiments of the present application.

[0214] Music production and restoration: During the music production process, the limitations of recording equipment or quality issues of historical recordings may result in a low sampling rate for the audio signal, affecting the sound quality. By using the super-resolution technology provided in the embodiments of this application, low-sampling-rate music signals can be upgraded to high sampling rates, restoring and enhancing the high-frequency details of the music and improving the overall sound quality. Sound effect design and enhancement: In audio production for movies, games, etc., the quality of the sound effects is crucial to the user experience. Through the technology provided in the embodiments of this application, low-quality or low-sampling-rate sound effects can be super-resolution processed to generate high-quality sound effects and enhance the audio-visual experience. Digitization and restoration of historical audio materials: Due to the limitations of recording equipment, many historical audio materials have problems of low sampling rate and poor sound quality. Using the technology provided in the embodiments of this application, these historical audios can be digitized and super-resolution processed to restore their original sound quality, which has important cultural and research value. Online music streaming services: In a network environment with limited bandwidth, online music streaming services may use low-sampling-rate audio streams to reduce the amount of data transmitted. By applying the super-resolution technology provided in the embodiments of this application on the client, the music playback quality on the user end can be improved without increasing the bandwidth burden, thereby improving the user experience.

[0215] It should be noted that the audio data processing method provided in the embodiment of the present application can also realize the application of image and video super-resolution or realize the super-resolution of other audio signals. Among them, when realizing the application of image and video super-resolution, although the embodiment of the present application takes audio super-resolution as an example for explanation, the idea of ​​the Schrödinger bridge model can also be extended to the super-resolution tasks of images and videos. For example, high-resolution images or video frames can be generated through a data-to-data sampling path to improve image quality, and play a role in image restoration, video streaming optimization and other fields. When realizing the super-resolution of other audio signals, the technology of the embodiment of the present application is not limited to music and sound effects, but can also be extended to super-resolution tasks of other types of audio signals, such as environmental sounds and broadcast signals. For example, in environmental sound monitoring, this technology can be used to improve the sampling rate of audio signals and more accurately analyze and identify sound events.

[0216] Based on the audio data processing method described in the above embodiment, Figure 10 A structural block diagram of an audio data processing device provided in an embodiment of the present application is shown. The audio data processing device 100 may be a device in an electronic device (for example, a server). The audio data processing device may be implemented in software, which may be software in the form of a program and a plug-in, and includes the following software modules: an acquisition module 101, for acquiring original audio data having a first resolution; an encoding processing module 102, for encoding the original audio data to obtain a latent representation vector of the original audio data in a latent space; a vector conversion module 103, for performing vector conversion on the latent representation vector through a preset vector conversion path to obtain a conversion vector; a decoding processing module 104, for decoding the conversion vector to obtain target audio data having a second resolution; the second resolution is greater than the first resolution.

[0217] In some embodiments, the encoding processing module 102 is further used to: extract an original feature vector from the waveform data of the original audio data; the dimension of the original feature vector is a preset dimension; and perform vector compression on the original feature vector using a preset compression ratio to obtain a potential representation vector of the original audio data in the latent space.

[0218] In some embodiments, the encoding processing module 102 is also used to: input the original audio data into a pre-trained audio data encoder; extract the original feature vector from the waveform data of the original audio data through the feature extraction layer of the audio data encoder; the potential number of channels of the audio data encoder is equal to the preset dimension.

[0219] In some embodiments, the device further includes: a scaling processing module for encoding the original audio data to obtain a latent representation vector of the original audio data in the latent space, and then scaling the latent representation vector using a preset scaling factor to obtain a scaled vector; the vector conversion module 103 is further used to: perform vector conversion on the scaled vector through the vector conversion path to obtain the conversion vector.

[0220] In some embodiments, the vector conversion path includes a vector conversion function; the vector conversion module 103 is further configured to: perform random differentiation processing on the potential representation vector through the vector conversion function to obtain the conversion vector.

[0221] In some embodiments, the audio data processing method is implemented by a data processing model; the data processing model includes an audio data encoder, a vector converter and an audio data decoder; the device also includes: a model training module, which is used to train the data processing model through the following steps: obtaining sample data; the sample data includes sample audio data with a first resolution and reference audio data with a second resolution; inputting the sample audio data into the data processing model to be trained; encoding the sample audio data through the audio data encoder of the data processing model to be trained to obtain a sample latent representation vector of the sample audio data in a latent space; performing vector conversion on the sample latent representation vector through the vector converter of the data processing model to be trained to obtain a sample conversion vector; decoding the sample conversion vector through the audio data decoder of the data processing model to be trained to obtain predicted audio data with a third resolution; performing loss calculation based on the predicted audio data and the reference audio data to obtain a loss result; updating the model parameters in the data processing model to be trained based on the loss result to obtain a trained data processing model.

[0222] In some embodiments, the model training module is also used to: in response to the current model training process being to train at least one of the audio data encoder and the audio data decoder, determine a reconstruction error value based on the predicted audio data and the reference audio data; determine a probability distribution error value based on the mean and variance of the sample potential representation vector; determine a multi-resolution error value based on the transformed data of the predicted audio data and the reference audio data at different resolutions; determine multi-period loss values ​​of the predicted audio data and the reference audio data in different periods; fuse the reconstruction error value, the probability distribution error value, the multi-resolution error value and the multi-period loss value to obtain the loss result; accordingly, update the model parameters in at least one of the audio data encoder and the audio data decoder based on the loss result.

[0223] In some embodiments, the model training module is also used to: in response to the current model training process training the vector converter, determine the loss result of the vector converter based on the predicted audio data and the reference audio data; and accordingly, update the model parameters in the vector converter based on the loss result.

[0224] It should be noted that the description of the device embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. For technical details not disclosed in the device embodiment, please refer to the description of the method embodiment of the present application for understanding.

[0225] An embodiment of the present application provides an electronic device, Figure 11 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 11 As shown, the electronic device 130 includes: at least one processor 131 ( Figure 11 Only one is shown in the figure), a memory 132, and computer executable instructions 133 stored in the memory 132 and executable on at least one processor 131. When the processor 131 executes the executable instructions 133, the steps of any of the above-mentioned audio data processing method embodiments are implemented.

[0226] The electronic device may include but is not limited to a processor 131 and a memory 132. It will be understood by those skilled in the art that Figure 11 This is merely an example of the electronic device 130 and does not constitute a limitation on the electronic device 130 . The electronic device 130 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0227] The processor 131 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPG), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0228] In some embodiments, the memory 132 may be an internal storage unit of the electronic device 130, such as a hard drive or memory of the electronic device 130. In other embodiments, the memory 132 may also be an external storage device of the electronic device 130, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card, etc. equipped on the electronic device 130. Furthermore, the memory 132 may include both an internal storage unit of the electronic device 130 and an external storage device. The memory 132 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of a computer program. The memory 132 may also be used to temporarily store data that has been output or is about to be output.

[0229] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the audio data processing method described in the present invention.

[0230] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the audio data processing method provided in the embodiment of the present application, for example, Figure 1 The audio data processing method shown.

[0231] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0232] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0233] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in an HTML document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0234] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0235] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A method for processing audio data, characterized in that: The method comprises: Obtaining raw audio data having a first resolution; Encoding the original audio data to obtain a latent representation vector of the original audio data in a latent space; Scaling the latent representation vector using a preset scaling factor to obtain a scaled vector; The scaling vector is subjected to random differentiation processing using a preset vector conversion function to obtain a conversion vector; wherein the vector conversion function is a function in a Schrödinger bridge model; during the vector conversion process, the vector conversion function is designed using the Schrödinger bridge model through the following steps: determining an initial distribution and a target distribution of a potential representation vector; establishing a transfer path from the initial distribution to the target distribution using the Schrödinger bridge model; and generating the conversion vector by sampling on the transfer path; The conversion vector is decoded to obtain target audio data with a second resolution; the second resolution is greater than the first resolution.

2. The method according to claim 1, characterized in that The encoding process of the original audio data to obtain a potential representation vector of the original audio data in a latent space includes: Extracting an original feature vector from the waveform data of the original audio data; the dimension of the original feature vector is a preset dimension; The original feature vector is compressed using a preset compression ratio to obtain a potential representation vector of the original audio data in a latent space.

3. The method according to claim 2, characterized in that The extracting of the original feature vector from the waveform data of the original audio data comprises: Inputting the raw audio data into a pre-trained audio data encoder; The original feature vector is extracted from the waveform data of the original audio data through the feature extraction layer of the audio data encoder; the potential channel number of the audio data encoder is equal to the preset dimension.

4. The method according to any one of claims 1 to 3, characterized in that The audio data processing method is implemented by a data processing model; the data processing model includes an audio data encoder, a vector converter, and an audio data decoder; the method further includes: training the data processing model by the following steps: Acquire sample data; the sample data includes sample audio data having a first resolution and reference audio data having a second resolution; Inputting the sample audio data into the data processing model to be trained; Encoding the sample audio data by the audio data encoder of the data processing model to be trained to obtain a sample potential representation vector of the sample audio data in a latent space; Performing vector conversion on the sample potential representation vector by a vector converter of the data processing model to be trained to obtain a sample conversion vector; Decoding the sample conversion vector using an audio data decoder of the data processing model to be trained to obtain predicted audio data with a third resolution; Performing loss calculation based on the predicted audio data and the reference audio data to obtain a loss result; Based on the loss result, the model parameters in the data processing model to be trained are updated to obtain a trained data processing model.

5. The method according to claim 4, characterized in that The performing loss calculation based on the predicted audio data and the reference audio data to obtain a loss result includes: In response to the current model training process training at least one of the audio data encoder and the audio data decoder, determining a reconstruction error value based on the predicted audio data and the reference audio data; Determining a probability distribution error value based on the mean and variance of the sample potential representation vector; determining a multi-resolution error value based on transformation data of the predicted audio data and the reference audio data at different resolutions; determining multi-period loss values ​​of the predicted audio data and the reference audio data in different periods; fusing the reconstruction error value, the probability distribution error value, the multi-resolution error value, and the multi-period loss value to obtain the loss result; Accordingly, updating the model parameters in the data processing model to be trained based on the loss result includes: Model parameters in at least one of the audio data encoder and the audio data decoder are updated based on the loss result.

6. The method according to claim 4, characterized in that The performing loss calculation based on the predicted audio data and the reference audio data to obtain a loss result includes: In response to the current model training process training the vector converter, determining a loss result of the vector converter based on the predicted audio data and the reference audio data; Correspondingly, updating the model parameters in the data processing model to be trained based on the loss result includes: updating the model parameters in the vector converter based on the loss result.

7. An audio data processing device, characterized in that: The device comprises: An acquisition module, configured to acquire original audio data having a first resolution; An encoding processing module, configured to perform encoding processing on the original audio data to obtain a potential representation vector of the original audio data in a latent space; a scaling processing module, configured to scale the latent representation vector using a preset scaling factor to obtain a scaled vector; a vector conversion module, configured to perform random differentiation processing on the scaled vector using a preset vector conversion function to obtain a conversion vector; wherein the vector conversion function is a function in a Schrödinger bridge model; during the vector conversion process, the vector conversion function is designed using the Schrödinger bridge model through the following steps: determining an initial distribution and a target distribution of a potential representation vector; establishing a transfer path from the initial distribution to the target distribution using the Schrödinger bridge model; and generating the conversion vector by sampling on the transfer path; A decoding processing module is used to decode the conversion vector to obtain target audio data with a second resolution; the second resolution is greater than the first resolution.

8. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer-executable instructions or computer programs; a processor for implementing the audio data processing method according to any one of claims 1 to 6 when executing the computer-executable instructions or computer programs stored in the memory.

9. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the audio data processing method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the audio data processing method according to any one of claims 1 to 6 is implemented.

Images