Audio data processing method, device, equipment, storage medium and program product

Abstract

The present application provides an audio data processing method, apparatus, device, storage medium, and program product; comprising: performing data conversion on original audio data having a first resolution to obtain first audio data having a second resolution; performing interpolation processing on the first audio data to obtain second audio data having a third resolution; determining a first latent representation vector of the second audio data in a first latent space, and a first smoothed representation vector of the second audio data in the first latent space; performing vector conversion on the first latent representation vector and the first smoothed representation vector through a preset first vector conversion path to obtain a first conversion vector; and decoding the first conversion vector to obtain target audio data having a third resolution. Through the present application, the reconstruction quality of audio data can be improved on the basis of improving the resolution of audio data.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an audio data processing method, apparatus, device, storage medium, and program product. Background Art

[0002] With the surge in demand for higher data fidelity in audio applications such as film and television production and virtual reality (VR / AR), audio super-resolution technology has become a core research area in audio signal processing. This technology's core goal is to reconstruct low-quality audio data into a high-quality signal, thereby recovering more high-frequency detail. In scenarios where detail restoration is crucial, achieving breakthroughs in ultra-high-resolution reconstruction (for example, 96kHz or even 192kHz audio) has significant application value.

[0003] However, related technologies are limited to modeling paradigms below low resolution (e.g., 48kHz audio). For ultra-high resolution audio data (e.g., 96kHz and 192kHz, etc.), directly modeling ultra-high resolution audio data usually faces severe challenges and cannot effectively reconstruct ultra-high resolution audio data. Summary of the Invention

[0004] The embodiments of the present application provide an audio data processing method, apparatus, device, storage medium, and program product. Through the present application, the reconstruction quality of audio data can be improved while improving the resolution of audio data.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides an audio data processing method, the method comprising: performing data conversion on original audio data having a first resolution to obtain first audio data having a second resolution; performing interpolation processing on the first audio data to obtain second audio data having a third resolution; the third resolution is greater than the second resolution, and the second resolution is greater than the first resolution; determining a first latent representation vector of the second audio data in a first latent space, and a first smoothed representation vector of the second audio data in the first latent space; using the first smoothed representation vector as a conversion starting point and the first latent representation vector as a conversion condition, performing vector conversion on the first latent representation vector and the first smoothed representation vector through a preset first vector conversion path to obtain a first conversion vector; and decoding the first conversion vector to obtain target audio data having the third resolution.

[0007] An embodiment of the present application provides an audio data processing device, including: a conversion module, used to perform data conversion on original audio data with a first resolution to obtain first audio data with a second resolution; an interpolation processing module, used to perform interpolation processing on the first audio data to obtain second audio data with a third resolution; the third resolution is greater than the second resolution, and the second resolution is greater than the first resolution; a determination module, used to determine a first latent representation vector of the second audio data in a first latent space, and a first smoothed representation vector of the second audio data in the first latent space; a vector conversion module, used to perform vector conversion on the first latent representation vector and the first smoothed representation vector through a preset first vector conversion path with the first smoothed representation vector as a conversion starting point and the first latent representation vector as a conversion condition to obtain a first conversion vector; a decoding processing module, used to decode the first conversion vector to obtain target audio data with the third resolution.

[0008] In the above scheme, the determination module is also used to: encode the second audio data to obtain a first latent representation vector of the second audio data in a first latent space; and smooth the first latent representation vector to obtain a first smoothed representation vector of the second audio data in the first latent space.

[0009] In the above solution, the device further includes a filtering processing module, and the filtering processing module is used to perform filtering processing on the second audio data to obtain filtered data.

[0010] In the above solution, the determination module is further used to: perform encoding processing on the filtered data to obtain a first latent representation vector of the second audio data in a first latent space.

[0011] In the above scheme, the determination module is also used to: extract a first eigenvector from the filtered data; the dimension of the first eigenvector is a preset dimension; and perform vector compression on the first eigenvector using a preset compression ratio to obtain a first latent representation vector of the second audio data in the first latent space.

[0012] In the above solution, the interpolation processing module is further used to: perform linear interpolation processing on the waveform data of the first audio data to obtain second audio data with the third resolution.

[0013] In the above scheme, the first vector conversion path includes a vector conversion function; the vector conversion function is a function in a pre-trained vector converter; the vector conversion module is also used to: input the first smooth representation vector and the first latent representation vector into the vector converter; perform vector splicing on the first smooth representation vector and the first latent representation vector through the preprocessing layer of the vector converter to obtain a spliced ​​vector; perform random differentiation processing on the spliced ​​vector through the vector conversion function of the vector converter to obtain the first conversion vector.

[0014] In the above scheme, the device also includes a re-execution module, which is used to: perform interpolation processing on the target audio data to obtain third audio data with a fourth resolution; the fourth resolution is greater than the third resolution; determine a second latent representation vector of the third audio data in the second latent space, and a second smooth representation vector of the third audio data in the second latent space; use the second smooth representation vector as a conversion starting point and the second latent representation vector as a conversion condition, perform vector conversion on the second latent representation vector and the second smooth representation vector through a preset second vector conversion path to obtain a second conversion vector; and perform decoding processing on the second conversion vector to obtain the target audio data with the fourth resolution.

[0015] In the above scheme, the audio data processing method is implemented by a data processing model; the data processing model includes a plurality of data conversion modules cascaded in sequence; the data conversion module includes: an audio data encoder, a vector converter and an audio data decoder; the device also includes a training module, the training module is used to: obtain sample data; the sample data includes first sample audio data with a fifth resolution and reference audio data with a sixth resolution; interpolate the first sample audio data to obtain second sample audio data with the sixth resolution; input the second sample audio data into the data conversion module to be trained; encode the second sample audio data through the audio data encoder of the data conversion module to be trained, and obtain the sample latent space of the second sample audio data in the third latent space. in the representation vector; smoothing the sample potential representation vector to obtain a sample smoothed representation vector of the second sample audio data in the third latent space; taking the sample smoothed representation vector as the conversion starting point and the sample potential representation vector as the conversion condition, performing vector conversion on the sample potential representation vector and the sample smoothed representation vector through the vector converter of the data conversion module to be trained to obtain a sample conversion vector; decoding the sample conversion vector through the audio data decoder of the data conversion module to be trained to obtain predicted audio data with the sixth resolution; performing loss calculation based on the predicted audio data and the reference audio data to obtain a loss result; updating the model parameters in the data conversion module to be trained based on the loss result to obtain a trained data conversion module.

[0016] In the above scheme, the training module is also used to: in response to the current training process being to train at least one of the audio data encoder and the audio data decoder, determine a reconstruction error value based on the predicted audio data and the reference audio data; determine a probability distribution error value based on the mean and variance of the sample potential representation vector; determine a multi-resolution error value based on the transformed data of the predicted audio data and the reference audio data at different resolutions; determine the multi-period loss values ​​of the predicted audio data and the reference audio data in different periods; and fuse the reconstruction error value, the probability distribution error value, the multi-resolution error value and the multi-period loss value to obtain the loss result.

[0017] In the above solution, the training module is further used to: update the model parameters in at least one of the audio data encoder and the audio data decoder based on the loss result.

[0018] In the above scheme, the training module is also used to: in response to the current training process being after the training of the audio data encoder and the audio data decoder is completed, train the vector converter, and determine the loss result of the vector converter based on the predicted audio data and the reference audio data.

[0019] In the above solution, the training module is further used to: update the model parameters in the vector converter based on the loss result.

[0020] An embodiment of the present application provides an electronic device, which includes: a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to implement the audio data processing method provided in the embodiment of the present application.

[0021] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the audio data processing method provided in the embodiment of the present application when executed by a processor.

[0022] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the audio data processing method provided in the embodiment of the present application is implemented.

[0023] The embodiments of the present application have the following beneficial effects:

[0024] The method converts original audio data of a first resolution into first audio data of a second resolution, interpolates the first audio data to generate second audio data of a third resolution, extracts a first latent representation vector and a first smoothed representation vector of the second audio data in a first latent space, performs vector transformation on the first latent representation vector and the first smoothed representation vector using a first vector transformation path to obtain a first transformed vector, and finally decodes the first transformed vector to obtain target audio data of a third resolution. In this way, by combining staged resolution enhancement with latent space feature optimization, an intermediate resolution foundation is constructed in the data conversion stage, resolution expansion is achieved through the interpolation stage, artifacts and unstable high-frequency details are removed in the latent representation extraction stage, and only low-frequency and partial mid-frequency energy are retained. The stability of the first smoothed representation vector is integrated with the detail fidelity of the first latent representation vector in the vector transformation stage, and the optimized high-dimensional features are mapped to the target resolution space in the decoding stage. This forms a collaborative optimization mechanism across resolution levels and latent space dimensions, effectively suppressing artifacts and distortion introduced by the interpolation process, enhancing the accuracy of high-frequency detail reconstruction and the continuity of low-frequency structure evolution, thereby improving the reconstruction quality of audio data while improving the audio data resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is an optional flowchart of the audio data processing method provided in an embodiment of the present application;

[0026] Figure 2 is a schematic diagram of a process for determining a first potential representation vector and a first smoothed representation vector provided by an embodiment of the present application;

[0027] Figure 3 is a schematic diagram of a process for determining a first potential representation vector provided by an embodiment of the present application;

[0028] Figure 4 is a schematic diagram of a process for determining a first conversion vector provided by an embodiment of the present application;

[0029] Figure 5 This is another optional flowchart of the audio data processing method provided in an embodiment of the present application;

[0030] Figure 6 is a schematic diagram of a process for determining target audio data having a fourth resolution provided by an embodiment of the present application;

[0031] Figure 7 1 is a flow chart of a training method for a data conversion module provided in an embodiment of the present application;

[0032] Figure 8 Schematic diagram of the process of determining loss results provided by an embodiment of the present application;

[0033] Figure 9 Schematic diagram of the overall architecture of the audio super-resolution system provided in an embodiment of the present application;

[0034] Figure 10 Schematic diagram of the implementation process of the audio super-resolution method provided in the embodiment of the present application;

[0035] Figure 11 This is a comparison diagram of the technical implementation process of LBM provided in the embodiment of the present application and the technical implementation process of AudioSR in the related art;

[0036] Figure 12 This is a structural block diagram of an audio data processing device provided in an embodiment of the present application;

[0037] Figure 13 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0039] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0040] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0041] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or portion of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal. It can be implemented in whole or in part using software, hardware (such as processing circuits or memory), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the functionality of the module or unit.

[0042] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0043] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0044] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0045] 1) In response to: This field indicates the conditions or states upon which the executed operation depends. When the conditions or states are met, the executed operation or operations may be performed in real time or with a set delay. Unless otherwise specified, the executed operations may be executed in any order.

[0046] 2) High-Resolution Audio: This refers to an audio format with a sampling rate and bit depth higher than CD quality (for example, 48kHz / 24-bit). High-Resolution Audio delivers richer details and a wider dynamic range, enhancing the listening experience. High-Resolution Audio files are typically significantly larger than CD-quality audio files. For example, a 96kHz / 24-bit audio file can be more than twice the size of a 44.1kHz / 16-bit file. To fully experience the benefits of high-resolution audio, high-quality audio equipment and a player are required. Ordinary headphones or speakers may not fully demonstrate the advantages of high-resolution audio. High-Resolution Audio captures greater detail, including high-frequency overtones, subtle background sounds, and complex timbre variations. These details are more clearly presented on high-quality audio equipment. Due to its higher bit depth, high-resolution audio can better handle sounds from extremely low to extremely high volumes, avoiding distortion or compression in music with a wide dynamic range. High-Resolution Audio also more closely resembles the sound quality of the original recording, providing a more natural and authentic sound. For example, in live music recordings, high-resolution audio can better recreate the atmosphere and sense of space of the venue.

[0047] 3) Sampling rate: This refers to the number of times an audio signal is sampled per unit time, typically measured in Hertz (Hz). For example, a 48kHz sampling rate means the audio signal is sampled 48,000 times per second. High-resolution audio often uses sampling rates higher than the CD-quality 44.1kHz. Common high-resolution sampling rates include 96kHz, 192kHz, and even higher. Higher sampling rates capture a wider frequency range, particularly high frequencies that are difficult for the human ear to perceive. For example, a 96kHz sampling rate can capture frequencies up to 48kHz, while a CD-quality 44.1kHz sampling rate can only capture frequencies up to 22.05kHz. While the human ear's hearing range is generally between 20Hz and 20kHz, higher sampling rate audio preserves greater detail and high-frequency information. This detail is particularly evident in the overtones of certain instruments, the high-frequency harmonics of percussion instruments, and natural ambient sounds such as birdsong and wind.

[0048] 4) Bit depth: This refers to the number of binary bits used for each sample point to represent the amplitude of the audio signal. The higher the bit depth, the greater the amplitude precision that can be represented. For example, a 24-bit depth can represent 224 (approximately 16.77 million) different amplitude levels, while a 16-bit depth can only represent 216 (65,536) levels. The bit depth of high-resolution audio is usually higher than the 16-bit CD quality. A common high-resolution bit depth is 24 bits, which can provide a higher dynamic range and lower quantization error. A higher bit depth can provide a wider dynamic range, that is, the difference between the quietest and the loudest sounds. In addition, a higher bit depth can reduce quantization noise, making the audio sound purer.

[0049] 5) Latent Space: It is an abstract multidimensional space formed by the model by compressing data features.

[0050] 6) Variational Autoencoder (VAE): A generative AI method that combines deep learning and probabilistic graphical models. It is primarily used for data generation, feature learning, and latent variable inference. The core idea of ​​VAE is to map data into a latent space through variational inference and generate new data from the latent space. For example, it can generate images (such as faces and handwritten digits), text, and music.

[0051] 7) Variational Autoencoder Encoder (VAE-Encoder): Responsible for mapping the input data to the probability distribution parameters (e.g., mean and variance) of the latent space.

[0052] In order to better understand the audio data processing method provided in the embodiment of the present application, the audio data processing method in the related art is first described below.

[0053] In the field of audio super-resolution (ASR), to achieve universal audio super-resolution for any input sampling rate up to 48kHz, related technologies have proposed ASR methods. ASR employs a two-stage architecture for super-resolution processing. In the first stage, a variational autoencoder (VAE) is used to map the low-resolution waveform to the mel-spectrographic space and construct a latent space representation. A generator based on a diffusion model (DM) is then trained in the latent space to recover the high-resolution mel-spectrographic latent space representation. Finally, the estimated mel-spectrogram is converted into a high-sampling-rate audio waveform via a VAE decoder and a high-fidelity generative adversarial network (HiFi-GAN) vocoder, completing the reconstruction process. Furthermore, to maintain low-frequency stability, ASR introduces a post-processing module that replaces the low-frequency components of the generated waveform with the corresponding components of the original low-sampling-rate audio to improve naturalness and fidelity. However, ASR is a two-stage cascade architecture that requires reconstruction in Mel space before waveform synthesis via a vocoder. This can easily lead to information distortion during the spectrogram-to-waveform conversion process, resulting in cumulative cascade errors. Furthermore, scaling to 96kHz or 192kHz requires training extremely large-scale diffusion models and vocoders, which is costly, slow to converge, and poorly adaptable to high sampling rates. Furthermore, while post-processing strategies for replacing low frequencies mitigate low-frequency drift, they are inconsistent across different modalities (for example, music and speech), leading to a loss of high-frequency detail.

[0054] To build a more compact and data-aware super-resolution modeling path, avoiding the noise propagation and redundant representation issues associated with modeling in the original waveform or spectrogram space, a related technique has proposed a latent bridge model (LBM). LBM uses a variational autoencoder structure to compress and model the audio waveform, and introduces a Schrödinger bridge modeling mechanism in its latent space to achieve high-quality super-resolution from low-resolution representations at arbitrary sampling rates to high-resolution representations at 48kHz. The specific process involves: in the encoding phase, an audio waveform at any sampling rate (e.g., 8kHz, 16kHz, or 24kHz) is encoded into a low-dimensional latent vector using a waveform VAE. In the bridge modeling phase, a Schrödinger bridge model is trained in the latent space to learn the data-to-data sampling path from the low-resolution latent distribution to the high-resolution latent distribution. In the decoding phase, the VAE decoder at the target sampling rate is used to convert the reconstructed latent vector into a high-resolution waveform. Furthermore, mechanisms such as optimized KL scaling factors and latent variable scaling are proposed to improve the stability and fidelity of the bridge path. While maintaining a lightweight model structure, this method can achieve high-quality super-resolution of general audio (including speech, music, and sound effects) from any sampling rate up to 48kHz. It significantly outperforms the traditional Mel spectrum plus diffusion model paradigm in metrics such as log spectral distance, objective audio-visual evaluation of virtual speech quality, and log spectral distance in high-frequency bands. However, when directly modeling the latent space representation at 96kHz or 192kHz, the model's super-resolution effect is poor due to the dramatic increase in information density, making it difficult to fully and meticulously model high frequencies. Furthermore, when sampling directly from a low sampling rate to 192kHz, the sampling trajectory spans much larger than the low-frequency range, making it difficult for the modeling path to converge and prone to loss of detail in the ultra-high frequency band. Furthermore, the method lacks a segmented optimal modeling mechanism, and a single model covers the entire frequency band, making it difficult to achieve optimal modeling quality in every frequency band.

[0055] In summary, the related technologies lack a cascade modeling mechanism and cannot gradually and high-quality realize ultra-high-resolution (96kHz or 192kHz) audio super-resolution tasks. In addition, the audio spectrum generated in the previous stage in the related technologies has artifacts or high-frequency errors. Directly using the latent space representation of the audio spectrum generated in the previous stage as the sampling starting point will interfere with the bridge modeling quality.

[0056] In response to the problems existing in the related art, an embodiment of the present application provides an audio data processing method based on the Cascaded Latent Bridge Modeling (C-LBM) method. By constructing an independent VAE module and a corresponding latent bridge model for each target sampling rate, the method performs staged modeling between multiple resolution levels and gradually restores the high-frequency content. This structure can independently optimize the modeling quality in each frequency band and ultimately synthesize high-resolution audio with greater generalization and controllability. In addition, based on the structured prior enhancement mechanism of waveform filtering, the method performs low-pass filtering on the waveform generated in the previous stage to remove artifacts and unstable high-frequency details, retaining only low-frequency and partial mid-frequency energy; then, the degraded waveform is encoded and Gaussian smoothed using the VAE at the current sampling rate to obtain a more robust latent space representation as a new sampling starting point; at the same time, the original high-resolution latent representation is retained as a condition to guide the latent bridge to accurately generate a high-fidelity target representation.

[0057] The audio data processing method provided in the embodiments of the present application can be applied to electronic devices such as laptop computers, tablet computers, and desktop computers. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0058] The audio data processing method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0059] Figure 1 This is an optional flow chart of the audio data processing method provided in the embodiment of the present application. The method can be applied to an electronic device, which can be a server or a terminal. That is, the audio data processing method of each embodiment of the present application can be executed by a server, or by a terminal, or by interaction between a server and a terminal. The following will be explained by taking the electronic device as an example. Figure 1 As shown, the method includes the following steps S101 to S105:

[0060] Step S101 : performing data conversion on original audio data with a first resolution to obtain first audio data with a second resolution.

[0061] In the embodiments of the present application, raw audio data refers to audio data that is not processed or compressed and is collected directly from the source device. Audio data is a digital signal that records sound and can be used in fields such as music, voice communication, or audio production. Raw audio data has the following attribute parameters: sampling rate, number of channels, and format. Among them, the sampling rate refers to the number of samples per second. For example, the sampling rate can be 44.1kHz (CD quality), 48kHz (professional audio), and 96kHz (high-fidelity audio). The number of channels includes mono, stereo, and multi-channel (such as 5.1 channels and 7.1 channels). Raw audio formats include lossless formats such as WAV, AIFF, and FLAC.

[0062] In some embodiments, data conversion is performed on original audio data having a first resolution to obtain first audio data having a second resolution, which can be achieved in the following manner: first, the original audio data having the first resolution is acquired; then, the original audio data is encoded to obtain a latent representation vector of the original audio data in a latent space; then, the latent representation vector is vector-converted through a preset vector conversion path to obtain a conversion vector; finally, the conversion vector is decoded to obtain the first audio data having the second resolution.

[0063] When obtaining raw audio data having a first resolution, in order to obtain audio data having the first resolution (e.g., 48kHz / 24-bit resolution), a specific audio recording device may be used to record and obtain the raw audio data of the audio type. Alternatively, the raw audio data may be received from another electronic device or downloaded from a specific website.

[0064] For raw audio files containing recorded original audio data, the audio data can be stored in lossless formats such as WAV, AIFF, or FLAC. These formats preserve the original audio quality and support audio data at 48kHz / 24-bit or higher resolutions. For example, in music production, a raw audio file may contain multiple tracks, each recorded at high resolution for precise adjustment during post-mixing and mastering.

[0065] The resolution of audio data is a key indicator used to describe the detail and quality of audio data. The original audio data obtained in the embodiment of the present application has a first resolution, which is related to factors such as sampling rate, quantization accuracy, and spatial resolution.

[0066] The primary determinant of the primary resolution of audio data is the sampling rate. The sampling rate refers to the number of times the audio signal is sampled per second, and is measured in Hertz (Hz). Sampling rates can include 44.1kHz, 48kHz, and 96kHz. The higher the sampling rate, the more accurately the frequency range of the audio signal can be captured. For example, a 44.1kHz audio signal is of CD quality and can meet most everyday listening needs; a 48kHz audio signal is a standard sampling rate for professional audio, providing a higher frequency range and better compatibility; and a 96kHz audio signal is high-fidelity audio, which can be used in high-end audio equipment and can capture richer high-frequency details.

[0067] The first resolution of the original audio data in the embodiment of the present application can be a relatively low resolution or an arbitrary resolution. In this way, through the audio data processing method provided in the embodiment of the present application, the original audio data with a low first resolution can be converted into first audio data with a higher second resolution.

[0068] Encoding raw audio data here refers to the process of converting it into a more compact, efficient, or vector-friendly form. Encoding involves steps such as data preprocessing, feature extraction, and dimensionality reduction and compression. Through encoding, complex raw audio data can be mapped into a low-dimensional latent space, providing a more efficient representation for subsequent tasks.

[0069] During the encoding process, data preprocessing involves formatting, normalizing, cropping, or performing other necessary transformations on the raw audio data to make it suitable for subsequent encoding operations. The purpose of the data preprocessing step is to convert the raw audio data into a format suitable for model processing. For audio data, this can include converting the audio file to a uniform sampling rate, cropping the audio data into fixed-length segments, and normalizing the audio data to a specific numerical range.

[0070] The feature extraction step involves extracting key features from the raw audio data that effectively characterize the core information of the raw audio data. The purpose of the feature extraction step is to reduce the dimensionality of the raw audio data while retaining its key features. In this embodiment of the present application, features such as mel spectrum, cepstrum, and short-time energy can be extracted from the audio data.

[0071] Dimensionality reduction and compression involves mapping high-dimensional raw audio data to a lower-dimensional space through mathematical transformations or model learning, while preserving the original audio data's key information. This can be achieved through both linear and nonlinear methods. Linear methods include principal component analysis and linear discriminant analysis, while nonlinear methods include autoencoders, variational autoencoders, or generative adversarial networks. Encoders (such as autoencoders, variational autoencoders, or generative adversarial networks) encode raw audio data into latent representation vectors in a latent space. These latent representation vectors can be used for subsequent tasks, such as vector transformations. Autoencoder methods compress data into a low-dimensional representation through an encoder and then reconstruct the data through a decoder. Variational autoencoders encode data into the distribution parameters (mean and variance) of a latent space through an encoder and generate a latent representation vector through reparameterization techniques. Generative adversarial networks learn the data distribution and generate new data samples through adversarial training of a generator and a discriminator.

[0072] The following is a detailed description of the encoding process provided in the embodiments of the present application. Encoding the raw audio data can be achieved by first extracting a raw feature vector from the waveform data of the raw audio data; the dimension of the raw feature vector is a preset dimension; then, vector compression is performed on the raw feature vector using a preset compression ratio to obtain a latent representation vector of the raw audio data in a latent space.

[0073] In an embodiment of the present application, extracting the original feature vector from the waveform data of the original audio data can be achieved in the following manner: inputting the original audio data into a pre-trained audio data encoder; then, extracting the original feature vector from the waveform data of the original audio data through the feature extraction layer of the audio data encoder; the potential number of channels of the audio data encoder is equal to the preset dimension.

[0074] Here, the waveform data of the original audio data refers to the representation of the original audio data in the time domain. For audio data, the waveform data is the amplitude of the sound signal changing over time. The audio waveform is a one-dimensional time series that represents the change in the amplitude of the sound signal over time.

[0075] Raw feature vectors are feature representations extracted from the waveform data of raw audio data, capturing the core information of the raw audio data. These raw feature vectors can be used for subsequent processing and analysis. The dimensionality of the raw feature vectors is pre-defined, determined during the design of the audio data encoder, and this dimensionality determines the size and complexity of the feature vectors.

[0076] An audio data encoder is a pretrained model that maps high-dimensional waveform data into a low-dimensional feature space. It can be implemented as a deep learning model, such as a convolutional neural network (CNN) or a recurrent neural network (RNN). The feature extraction layer in the audio data encoder is responsible for extracting useful features from the waveform data. These layers can include convolutional layers, pooling layers, and fully connected layers. The number of potential channels in the audio data encoder is equal to the preset dimensionality, meaning that the dimension of the output feature vector of the audio data encoder is fixed.

[0077] The process of extracting original feature vectors from waveform data of original audio data may include: inputting the original audio data into an audio data encoder; then, the audio data encoder processes the input original audio data through a feature extraction layer to extract original feature vectors. Since the feature vectors output by the audio data encoder are low-dimensional, they can capture the core features of the input audio data.

[0078] In an embodiment of the present application, when extracting the original feature vector from the waveform data of the original audio data, the audio data can be converted into a uniform sampling rate and quantization bit depth, and cropped into segments of fixed length. Then, feature extraction methods such as Mel spectrum and cepstrum are used to convert the waveform data of the audio into a feature vector.

[0079] The preset dimensions in the embodiments of the present application can be selected based on the specific task and data set. A higher dimension can capture more details but will increase the computational complexity.

[0080] Vector compression is the process of mapping a high-dimensional feature vector to a low-dimensional space. This process aims to reduce the dimensionality of the feature vector while retaining as much of the original information as possible. The goal of vector compression is to generate a low-dimensional latent representation vector that effectively represents the core features of the original audio data in the latent space.

[0081] When compressing the original feature vector using a preset compression ratio, you can choose an appropriate compression method based on the task requirements and data characteristics. For example, for sequence data, you can use LSTM or GRU. You can also preset the compression ratio based on the task requirements, that is, determine the dimensionality of the low-dimensional latent representation vector. The compression ratio is typically a hyperparameter that can be adjusted experimentally. If using model-based methods (such as autoencoders, VAEs, and GANs), the model can be trained using a large amount of labeled or unlabeled data. During training, the low-dimensional representation is learned by optimizing an objective function (such as reconstruction error or KL divergence).

[0082] A latent space is a low-dimensional, continuous vector space into which the raw audio data is mapped by an audio encoder (e.g., a VAE encoder). In this vector space, the high-dimensional features of the raw audio data are compressed into a set of low-dimensional latent variables (or potential representation vectors) that capture the core features and structure of the raw audio data. The dimensionality of the latent space is typically much lower than that of the raw audio data.

[0083] A latent representation vector (LRV) represents a specific data point in a latent space. The LRV is a low-dimensional vector obtained by mapping the raw audio data into the latent space through an audio encoder. The LRV captures the core features and structure of the original audio data while removing redundant information. As a low-dimensional vector, the LRV represents the original audio data in a more compact form. Each dimension of the LRV corresponds to a semantic feature of the original audio data. For example, in the latent representation of raw speech data, certain dimensions may correspond to features such as intonation and speaking rate. Operations such as interpolation, weighting, and editing can be performed on the LRV in the latent space. These operations can be used to generate new data samples or modify certain features of existing data. The LRV can be decoded back to the original data space through a decoder to reconstruct the original data or generate new data samples. In an embodiment of the present application, raw feature vectors are extracted from the waveform data of the original audio data and vectorized using a preset compression ratio to obtain a latent representation vector. This process improves data processing efficiency and reduces computational complexity by obtaining the latent representation vector of the original audio data in a latent space. Because raw audio data is typically high-dimensional, for example, audio data may contain hundreds of thousands or even millions of samples. Directly processing this high-dimensional data results in extremely high computational complexity, making it difficult to process in real time or run on resource-constrained devices. By extracting low-dimensional raw feature vectors and further compressing them into a latent space, the data dimensionality can be significantly reduced. Low-dimensional representation vectors have lower computational complexity, speeding up subsequent processing and improving overall system efficiency. Furthermore, they can reduce storage requirements, as high-dimensional data requires a large amount of storage space. For example, a high-definition audio file may occupy several or even tens of GB of storage space. Through vector compression, the data can be compressed into a more compact representation, thereby reducing storage requirements. Furthermore, low-dimensional representation vectors require less data during transmission, improving data transmission efficiency and reducing bandwidth requirements. Furthermore, data interpretability and semantics can be enhanced. Using a pre-trained audio data encoder, semantically meaningful feature vectors can be extracted. These feature vectors capture core data characteristics, such as intonation and rhythm. The latent representation vectors obtained through vector compression exhibit enhanced semantics in the latent space. The latent space is an optimized low-dimensional space in which similar data points are close together and data points of different categories are effectively distinguished. Furthermore, low-dimensional latent representation vectors are easier to analyze and interpret, as each dimension may correspond to a specific semantic feature.

[0084] Here, a vector transformation path refers to the implementation path that transforms the latent representation vector into another form (the transformation vector). In probabilistic generative models, a vector transformation path specifically refers to the random probability trajectory between two distributions.

[0085] In some embodiments, after encoding the original audio data to obtain a latent representation vector of the original audio data in a latent space, the latent representation vector may be scaled using a preset scaling factor to obtain a scaled vector. Accordingly, performing vector conversion on the latent representation vector using a preset vector conversion path may involve performing vector conversion on the scaled vector using the vector conversion path to obtain the converted vector.

[0086] Here, scaling refers to performing a multiplication operation on each dimension of the latent representation vector to adjust the scale of the latent representation vector. The scaling factor is a preset parameter that controls the degree of scaling. Scaling can be uniform (using the same scaling factor for all dimensions) or non-uniform (using different scaling factors for different dimensions). Scaling can adjust the scale of the latent representation vector to make it more suitable for subsequent processing steps. For example, in some machine learning models, the scale of the input data needs to be within a specific range (such as [0, 1] or [-1, 1]), and scaling can achieve this.

[0087] In an embodiment of the present application, by scaling the potential representation vector, the expressive power of some features in the potential representation vector can be enhanced. For example, if certain features are not significant enough at the original scale, appropriate scaling can make these features more influential in subsequent processing. Scaling can also adjust the potential representation vector to a scale that matches the model input. For example, some generative models may require the scale of the input vector to be within a specific range, and scaling can ensure this. Scaling can also avoid overflow or underflow problems in numerical calculations. For example, in the gradient descent process, excessively large eigenvalues ​​may cause the gradient to explode, while excessively small eigenvalues ​​may cause the gradient to disappear. Appropriate scaling can improve the stability of numerical calculations.

[0088] In an embodiment of the present application, the vector conversion path may include a vector conversion function, which is a specific mathematical function or model for realizing vector conversion. The vector conversion function receives a potential representation vector as input and outputs a conversion vector. In step S103, the vector conversion of the potential representation vector is performed through a preset vector conversion path, which can be achieved in the following way: the potential representation vector is subjected to random differential processing through the vector conversion function to obtain the conversion vector. Random differential processing refers to the introduction of randomness in the conversion process, which is usually achieved through a stochastic differential equation (SDE, Stochastic Differential Equation). This method can increase the diversity and flexibility of the conversion vector. In one implementation method, the potential representation vector is subjected to random differential processing through a vector conversion function, which can be achieved using a stochastic differential equation (such as Langevin dynamics).

[0089] In an embodiment of the present application, the vector conversion function may be a function in a Schrödinger bridge model. The Schrödinger Bridge model is a model based on probability and dynamics, which is used to establish an optimal transfer path between two probability distributions. The Schrödinger Bridge model originates from the Schrödinger equation in quantum mechanics, but is used to generate and convert data in machine learning. The Schrödinger Bridge model finds the optimal path by minimizing the relative entropy between the two distributions. In the vector conversion process, the Schrödinger Bridge model can be used to design a vector conversion function, which can be achieved by the following steps: first, define the initial and target distributions, and determine the initial distribution and target distribution of the potential representation vector (i.e., the distribution of the audio data at the second resolution); then, establish a transfer path, and use the Schrödinger Bridge model to establish the optimal transfer path from the initial distribution to the target distribution; finally, generate a conversion vector, and generate a conversion vector by sampling on the transfer path.

[0090] In embodiments of the present application, a latent representation vector is transformed using a preset vector transformation path (such as a Schrödinger bridge model) to generate transformation vectors capable of reconstructing higher-resolution first audio data. This process involves stochastic differentiation and probabilistic optimization, which can improve data diversity and quality. This approach has broad application prospects in fields such as audio enhancement. Specifically, the vector transformation of the latent representation vector in embodiments of the present application can generate higher-resolution data because the resulting transformation vectors can reconstruct higher-resolution first audio data. That is, after decoder processing, the transformation vectors can reconstruct higher-resolution audio data. For example, high-resolution audio can be generated from the latent representation vectors of low-resolution audio. Data diversity can also be increased because the introduction of randomness allows for the generation of diverse transformation vectors, thereby increasing data diversity. By minimizing relative entropy, the Schrödinger bridge model can find the optimal transition path and generate high-quality transformation vectors. Furthermore, data quality can be improved. The Schrödinger bridge model optimizes the transition path to ensure that the generated transformation vectors are capable of better reconstructing the target data. Furthermore, the optimized path and stochastic differentiation process can reduce distortion during the reconstruction process, improving data quality.

[0091] Here, the second resolution is greater than the first resolution. The transformed vector is decoded using a decoder to map the low-dimensional transformed vector back to the high-dimensional original data space, yielding the first audio data at the second resolution. In implementation, the transformed vector can be decoded using any of the following decoders: a fully connected decoder, a recurrent decoder, and a Transformer decoder. The fully connected decoder progressively maps the low-dimensional vector to a high-dimensional space through multiple fully connected layers. The recurrent decoder is suitable for sequential data, such as audio sequences, by progressively generating each element in the sequence using a recurrent neural network (such as a long short-term memory (LSTM) or a gated recurrent unit (GRU). The recurrent decoder captures temporal dependencies and generates temporally consistent high-resolution data. The Transformer decoder, based on the self-attention mechanism, is suitable for long-sequence data. It progressively generates data through multiple self-attention layers and feedforward network layers. The Transformer decoder captures global dependencies and generates high-quality, high-resolution data.

[0092] In an embodiment of the present application, a low-dimensional vector (conversion vector) after vector conversion is input into a decoder. This vector contains all the information required to generate high-resolution data. The decoder gradually maps the low-dimensional vector to a high-dimensional space through a series of layers. The decoder may involve but is not limited to the following operations: upsampling, feature extraction, activation processing of the activation function, and normalization. The high-resolution data output by the decoder can be high-resolution audio data (i.e., first audio data with a second resolution). For example, for audio data, the output is a high-sampling-rate audio waveform whose shape is (time step, number of channels).

[0093] Step S102: performing interpolation processing on the first audio data to obtain second audio data with a third resolution.

[0094] In an embodiment of the present application, the third resolution is greater than the second resolution, and the second resolution is greater than the first resolution. Interpolation is a method of inferring new data points from known data points (first audio data with a second resolution). The purpose is to generate second audio data with a higher resolution through a mathematical algorithm without increasing the actual amount of information. The third resolution is the target resolution after interpolation. The value of the third resolution (for example, the number of pixels or sampling rate, etc.) is greater than the original second resolution, which manifests as finer details or higher time / space density. The second audio data refers to the result data generated after interpolation. The resolution of the second audio data (the third resolution) is higher than the original first audio data (the second resolution), but in essence it is obtained through algorithmic derivation rather than physical acquisition. For example, through interpolation, audio sampled at 48 kHz can be upconverted to 96 kHz.

[0095] In some embodiments, interpolating the first audio data to obtain second audio data having a third resolution can be achieved by performing linear interpolation on waveform data of the first audio data to obtain second audio data having a third resolution.

[0096] In this embodiment of the present application, the waveform data of the first audio data refers to the representation of the first audio data in the time domain. For audio data, waveform data is the amplitude of the sound signal that changes over time. An audio waveform is a one-dimensional time series that represents the change in the amplitude of the sound signal over time.

[0097] Linear interpolation is a method of calculating the midpoint between two adjacent known data points based on their distance. Specifically, each original sampling point in the waveform data of the first audio data is marked along the time axis. A preset linear interpolation formula is then applied to insert a new interpolation point between each two original sampling points. Finally, all original sampling points and interpolation points are arranged in chronological order to generate the second audio data at the third resolution.

[0098] Here, through linear interpolation processing, the low-resolution (second resolution) first audio data is upconverted to the high-resolution (third resolution) second audio data, achieving resolution improvement at the lowest cost, thereby reducing the computational burden while maintaining data quality.

[0099] Step S103 : determining a first latent representation vector of the second audio data in the first latent space, and a first smoothed representation vector of the second audio data in the first latent space.

[0100] In this embodiment of the present application, the first latent space refers to a low-dimensional, continuous vector space to which the second audio data is mapped by the audio data encoder corresponding to the third resolution (for example, a VAE encoder). In this vector space, the high-dimensional features of the second audio data are compressed into a set of low-dimensional latent variables (i.e., the first latent representation vector), which can capture the core features and structure of the second audio data. The dimensionality of the first latent space is typically much lower than that of the second audio data.

[0101] The first latent representation vector refers to the representation of a specific data point in the first latent space. The first latent representation vector is a low-dimensional vector obtained by mapping the second audio data to the first latent space using an audio data encoder corresponding to the third resolution. The first latent representation vector can capture the core features and structure of the second audio data while removing redundant information. The first latent representation vector is a low-dimensional vector that can represent the second audio data in a more compact form. Each dimension of the first latent representation vector can correspond to a semantic feature of the second audio data. For example, in the latent representation of the second speech data, certain dimensions can correspond to features such as intonation and speaking speed. The first latent representation vector can perform operations in the first latent space, such as interpolation, weighting, editing, and the like. These operations can be used to generate new data samples or modify certain features of existing data.

[0102] In some embodiments, see Figure 2 , Figure 2In step S103, the server determines the first latent representation vector of the second audio data in the first latent space, and the first smoothed representation vector of the second audio data in the first latent space. The following steps S1031 to S1032 can be performed:

[0103] Step S1031: Encode the second audio data to obtain a first latent representation vector of the second audio data in a first latent space.

[0104] In some embodiments, before encoding the second audio data to obtain the first latent representation vector of the second audio data in the first latent space, the server may further perform the following steps: filtering the second audio data to obtain filtered data.

[0105] Here, filtering is a signal processing technique that extracts desired information or eliminates interference from the original second audio data by selectively retaining or suppressing specific frequencies or features. Filtering can be performed in the form of low-pass filtering, which allows low-frequency signals to pass while attenuating high-frequency components in the second audio data. The filtered data is the result of filtering the original second audio data. The filtered data retains the target features (low-frequency structure) while removing interfering components (high-frequency noise).

[0106] After obtaining the second audio data with the third resolution, it is first necessary to perform filtering processing on the second audio data to obtain filtered data. The filtering processing effectively removes potential high-frequency artifacts and incomplete structures in the second audio data.

[0107] Accordingly, the server encodes the second audio data to obtain the first latent representation vector of the second audio data in the first latent space, which can be achieved by encoding the filtered data to obtain the first latent representation vector of the second audio data in the first latent space.

[0108] It should be noted that encoding the filtered data in this step to obtain the first latent representation vector of the second audio data in the first latent space is implemented in the same manner as encoding the original audio data in step S101 to obtain the latent representation vector of the original audio data in the latent space. The encoding process in this step can be understood by referring to the relevant explanation of step S101. The only difference is that in step S101, the encoding process is performed using an audio data encoder corresponding to the first resolution, and the encoding process is performed on the original audio data; while in this step, the encoding process is performed using an audio data encoder corresponding to the third resolution, and the encoding process is performed on the filtered data.

[0109] Here, filtering is introduced before encoding the second audio data. The first latent representation vector is obtained by generating filtered data and encoding based on it. This process effectively enhances the structural guidance and robustness of the generative model through the synergistic mechanism of signal degradation and feature purification. Filtering actively suppresses high-frequency noise and artifacts, allowing the encoding process to focus on stable structural features dominated by low frequencies, avoiding the distortion of the latent space caused by noise interference, thereby generating a degraded but more robust latent representation substrate. While inheriting the macroscopic structure of the previous stage, this latent representation eliminates detailed interference through physical prior constraints, providing a feature starting point closer to the real data distribution for the subsequent generation path, significantly improving the consistency of cross-stage generation. This deep coupling of signal processing and deep learning models forms structurally enhanced feature guidance in the latent space dimension, allowing the generation process to actively repair spectral defects while taking into account local continuity constraints and global structural controllability, ultimately achieving quality optimization and error correction closed loop for the multi-stage generation system.

[0110] In some embodiments, see Figure 3 , Figure 3 It is shown that the server encodes the filtered data to obtain a first latent representation vector of the second audio data in the first latent space, which can be achieved by executing the following steps 301 to 302:

[0111] Step 301: extract a first eigenvector from filtered data.

[0112] It should be noted that the extraction of the first eigenvector from the filtered data in step S301 is implemented in the same manner as the extraction of the original eigenvector from the waveform data of the original audio data in step S101 above. The extraction process in step S301 can be understood by referring to the relevant explanation of step S101 above. The only difference is that the object extracted in step S101 is the waveform data of the original audio data, while the object extracted in step S301 is the filtered data.

[0113] Step 302: Perform vector compression on the first feature vector using a preset compression ratio to obtain a first latent representation vector of the second audio data in a first latent space.

[0114] It should be noted that the vector compression of the first feature vector using a preset compression ratio in step S302 to obtain the first latent representation vector of the second audio data in the first latent space is implemented in the same manner as the vector compression of the original feature vector using a preset compression ratio in step S101 to obtain the latent representation vector of the original audio data in the latent space. The vector compression process in step S302 can be understood by referring to the relevant explanation of step S101. The only difference is that the object of vector compression in step S101 is the original feature vector, while the object of vector compression in step S302 is the first feature vector.

[0115] Through steps 301 and 302, first eigenvectors are extracted from the filtered data and vector compressed using a preset compression ratio to obtain a first latent representation vector. This process yields the first latent representation vector of the second audio data in the first latent space. This improves data processing efficiency and reduces computational complexity. Second audio data is typically high-dimensional, for example, it may contain hundreds of thousands or even millions of samples. Directly processing this high-dimensional data results in extremely high computational complexity, making it difficult to process in real time or run on resource-constrained devices. By extracting low-dimensional first eigenvectors and further compressing them into the first latent space, the data dimensionality can be significantly reduced. Low-dimensional representation vectors have lower computational complexity, speeding up subsequent processing and improving overall system efficiency. Furthermore, they can reduce storage requirements, as high-dimensional data requires significant storage space. For example, a high-definition audio file may occupy several gigabytes or even tens of gigabytes of storage space. Vector compression can compress the data into a more compact representation, reducing storage requirements. Furthermore, low-dimensional representation vectors require less data for transmission, improving data transmission efficiency and reducing bandwidth requirements. Furthermore, data interpretability and semantics can be enhanced. Using a pre-trained audio data encoder, semantically meaningful feature vectors can be extracted. These feature vectors capture core data characteristics, such as intonation and rhythm. The first latent representation vector obtained through vector compression exhibits enhanced semantics in the first latent space. The first latent space is an optimized low-dimensional space in which similar data points are close together and data points of different categories are effectively distinguished. Furthermore, the low-dimensional first latent representation vector is easier to analyze and interpret, as each dimension may correspond to a specific semantic feature.

[0116] Step S1032: Smoothing the first latent representation vector to obtain a first smoothed representation vector of the second audio data in the first latent space.

[0117] In this embodiment of the present application, smoothing refers to performing a local averaging or weighting operation on the first latent representation vector to eliminate random fluctuations or local mutations in the first latent space and make the overall trend more consistent. The smoothing process can be Gaussian smoothing, for example. The first smoothed representation vector is the feature representation of the first latent representation vector after the smoothing process.

[0118] After obtaining the first latent representation vector, it is smoothed to obtain a first smoothed representation vector. By imposing local continuity constraints on the first latent space dimension, this effectively suppresses abnormal fluctuations and inter-dimensional mutations in the first latent representation vector, enhancing the topological coherence of the feature distribution. This smoothing operation transforms the discretized latent feature map into a smoothly transitioning manifold structure, thereby ensuring that the starting point of the subsequent generation path has greater physical rationality. This guides the model to prioritize the underlying data regularities during the sampling process, thereby reducing structural conflicts and random artifacts in the generated results.

[0119] Step S104 , using the first smoothed representation vector as a conversion starting point and the first latent representation vector as a conversion condition, performs vector conversion on the first latent representation vector and the first smoothed representation vector through a preset first vector conversion path to obtain a first conversion vector.

[0120] In the embodiment of the present application, the starting point of the conversion is the first smoothed representation vector after smoothing the first latent space. The first smoothed representation vector is used as the initial state of the generation path. The first smoothed representation vector provides a structurally stable and noise-suppressed baseline representation, ensuring that the generation process starts from a coherent topological structure and avoiding pattern confusion caused by sudden changes in the latent space dimension. The conversion condition is the original unsmoothed latent code, that is, the first latent representation vector. The first latent representation vector retains the complete characteristics of the second audio data (filtered data) after filtering. The first latent representation vector is used as a constraint condition for the generation path. By injecting detailed information about the original data into the model, the first latent representation vector guides the generation direction to be close to the real data distribution.

[0121] In some embodiments, the first vector conversion path includes a vector conversion function; the vector conversion function is a function in a pre-trained vector converter, and is a specific mathematical function or model that implements vector conversion. The vector conversion function receives the first latent representation vector and the first smoothed representation vector as input, and outputs a first conversion vector. Figure 4 , Figure 4 In step S104, the server uses the first smoothed representation vector as a conversion starting point and the first latent representation vector as a conversion condition, performs vector conversion on the first latent representation vector and the first smoothed representation vector through a preset first vector conversion path, and obtains a first conversion vector. This can be achieved by executing the following steps S1041 to S1043:

[0122] Step S1041: input the first smoothed representation vector and the first latent representation vector into a vector converter.

[0123] Step S1042 : performing vector concatenation on the first smoothed representation vector and the first latent representation vector through a preprocessing layer of a vector converter to obtain a concatenated vector.

[0124] Step S1043 : Perform random differentiation processing on the spliced ​​vector using a vector conversion function of a vector converter to obtain a first conversion vector.

[0125] Here, the vector converter includes a preprocessing layer for performing vector concatenation on the first smoothed representation vector and the first latent representation vector. Vector concatenation refers to the operation of concatenating the first smoothed representation vector and the first latent representation vector end to end along a specific dimension. For example, if the first smoothed representation vector (dimension 128) and the first latent representation vector (dimension 128) are concatenated along the channel dimension, the result will generate a new 256-dimensional vector, namely the concatenated vector. Stochastic differential processing refers to the introduction of randomness in the vector conversion process, which is usually implemented through stochastic differential equations (SDEs). This method can increase the diversity and flexibility of the converted vectors. In one implementation, the concatenated vector is subjected to stochastic differential processing through a vector conversion function, which can be implemented using stochastic differential equations (such as Langevin dynamics).

[0126] In an embodiment of the present application, the vector conversion function may be a function in a Schrödinger bridge model. The Schrödinger Bridge model is a model based on probability and dynamics, used to establish an optimal transfer path between two probability distributions. The Schrödinger Bridge model originates from the Schrödinger equation in quantum mechanics, but is used to generate and convert data in machine learning. The Schrödinger Bridge model finds the optimal path by minimizing the relative entropy between the two distributions. In the vector conversion process, the Schrödinger Bridge model can be used to design a vector conversion function, which can be achieved by the following steps: first, defining the initial and target distributions, determining the initial distribution and target distribution of the spliced ​​vector (i.e., the distribution of the audio data at the third resolution); then, establishing a transfer path, and using the Schrödinger Bridge model to establish the optimal transfer path from the initial distribution to the target distribution; finally, generating a first conversion vector, and generating the first conversion vector by sampling on the transfer path.

[0127] Here, by using the first smoothed representation vector as a structurally stable transformation starting point and the first latent representation vector as a detail-preserving transformation condition, and constructing a transformation path based on vector splicing and stochastic differentiation, this technique achieves dynamic fusion and refined control of the latent space feature evolution process. The smoothed starting point provides a noise-suppressing and topologically continuous initial state for the generation path, while the latent condition constrains the authenticity of the generation direction by preserving the original details. The vector splicing of the two forms a holographic coupled representation of global-to-local information in the feature dimension, while stochastic differentiation accurately models the cross-scale correlations and gradient patterns of features by analyzing the nonlinear dynamics of the spliced ​​vectors. This synergistic mechanism, while suppressing sudden changes in the latent space dimensions, ensures that the generation process inherits the macroscopic structural stability of the smoothed base while adaptively integrating the microscopic detail features of the original latent vector. Ultimately, this achieves a multidimensional balance between structural coherence, detail restoration, and noise robustness in the generation path, significantly improving the quality controllability and semantic consistency of cross-media content generation.

[0128] Step S105 : Decode the first conversion vector to obtain target audio data with a third resolution.

[0129] It should be noted that decoding the first transformation vector in step S105 to obtain target audio data having the third resolution is implemented in the same manner as decoding the transformation vector in step S101 to obtain target audio data having the second resolution. The decoding process in step S105 can be understood by referring to the explanation of step S101. The only difference is that the decoding process in step S101 is performed on the transformation vector, while the decoding process in step S105 is performed on the first transformation vector.

[0130] In an embodiment of the present application, raw audio data having a first resolution is converted to obtain first audio data having a second resolution, the first audio data is interpolated to generate second audio data having a third resolution, a first latent representation vector and a first smoothed representation vector of the second audio data are extracted in a first latent space, the first latent representation vector and the first smoothed representation vector are vector-converted using a first vector conversion path to obtain a first transformed vector, and finally the first transformed vector is decoded to obtain target audio data having a third resolution. In this manner, by combining staged resolution enhancement with latent space feature optimization, an intermediate resolution foundation is constructed in the data conversion stage, resolution expansion is achieved through the interpolation stage, artifacts and unstable high-frequency details are removed in the latent representation extraction stage, and only low-frequency and partial mid-frequency energy are retained. The stability of the first smoothed representation vector is integrated with the detail fidelity of the first latent representation vector in the vector conversion stage, and the optimized high-dimensional features are mapped to the target resolution space in the decoding stage. This forms a collaborative optimization mechanism across resolution levels and latent space dimensions, effectively suppressing artifacts and distortion introduced by the interpolation process, enhancing the accuracy of high-frequency detail reconstruction and the continuity of low-frequency structure evolution, thereby improving the reconstruction quality of audio data while increasing the resolution of the audio data.

[0131] The audio data processing method provided in the embodiments of this application has broad application value in multiple fields, such as audio super-resolution. By improving the resolution and quality of data, the user experience and application effects can be significantly enhanced. The following examples illustrate the application scenarios of the audio data processing method provided in the embodiments of this application.

[0132] In audio super-resolution scenarios, the audio data processing method provided in the embodiments of this application can be used to input low-sampling-rate audio (first resolution) and output high-sampling-rate audio (third resolution). For example, in the field of audio processing, low-sampling-rate audio can be converted to high-sampling-rate audio to improve audio fidelity and clarity.

[0133] Below, the audio data processing method provided in the embodiment of the present application will be described in combination with any one of the above application scenarios. Figure 5 This is another optional flow chart of the audio data processing method provided in the embodiment of the present application, such as Figure 5 As shown, the method includes the following steps S201 to S213:

[0134] Step S201: The terminal receives original audio data with a first resolution input by a user.

[0135] In step S202 , the terminal encapsulates the original audio data into a data processing request.

[0136] Step S203: The terminal sends a data processing request to the server.

[0137] Here, the original audio data at the first resolution can be encapsulated into a data processing request, and the terminal can send the data processing request to the server. In some embodiments, the terminal can send the data processing request using a protocol such as Hypertext Transfer Protocol (HTTP) or Web Socket. After receiving the data processing request, the server parses the data processing request and obtains the original audio data at the first resolution input by the user.

[0138] Step S204 : In response to the data processing request, the server performs data conversion on the original audio data with the first resolution to obtain first audio data with a second resolution.

[0139] Step S205: The server performs linear interpolation processing on the waveform data of the first audio data to obtain second audio data with a third resolution.

[0140] Step S206: The server encodes the second audio data to obtain a first latent representation vector of the second audio data in a first latent space.

[0141] Step S207: The server performs smoothing processing on the first latent representation vector to obtain a first smoothed representation vector of the second audio data in the first latent space.

[0142] In step S208 , the server inputs the first smoothed representation vector and the first latent representation vector into a vector converter.

[0143] In step S209 , the server performs vector concatenation on the first smoothed representation vector and the first latent representation vector through a preprocessing layer of a vector converter to obtain a concatenated vector.

[0144] In step S210 , the server performs random differentiation processing on the concatenated vector using a vector conversion function of a vector converter to obtain a first conversion vector.

[0145] Step S211: The server decodes the first conversion vector to obtain target audio data with a third resolution.

[0146] In some embodiments, see Figure 6 , Figure 6 It shows that after the server decodes the first conversion vector to obtain target audio data with the third resolution, it can also perform the following steps S601 to S604 to implement:

[0147] Step S601: interpolate the target audio data to obtain third audio data with a fourth resolution.

[0148] In the embodiment of the present application, the fourth resolution is greater than the third resolution.

[0149] It should be noted that the interpolation processing of the target audio data in step S601 to obtain the third audio data having the fourth resolution is implemented in the same manner as the interpolation processing of the first audio data in step S102 to obtain the second audio data having the third resolution. The interpolation process in step S601 can be understood by referring to the relevant explanation of step S102. The only difference is that the interpolation processing in step S102 is performed on the first audio data, while the interpolation processing in step S601 is performed on the target audio data having the third resolution.

[0150] Step S602 : Determine a second latent representation vector of the third audio data in the second latent space, and a second smoothed representation vector of the third audio data in the second latent space.

[0151] It should be noted that determining the second latent representation vector of the third audio data in the second latent space and the second smoothed representation vector of the third audio data in the second latent space in step S602 is implemented in the same manner as determining the first latent representation vector of the second audio data in the first latent space and the first smoothed representation vector of the second audio data in the first latent space in step S103 above. The determination process in step S602 can be understood by referring to the relevant explanation of step S103 above. The only difference is that the targets determined in step S103 are the first latent representation vector and the first smoothed representation vector, while the targets determined in step S602 are the second latent representation vector and the second smoothed representation vector.

[0152] Step S603 , using the second smoothed representation vector as a conversion starting point and the second latent representation vector as a conversion condition, performs vector conversion on the second latent representation vector and the second smoothed representation vector through a preset second vector conversion path to obtain a second conversion vector.

[0153] It should be noted that in step S603, the vector conversion of the second latent representation vector and the second smoothed representation vector is performed using the second smoothed representation vector as the conversion starting point and the second latent representation vector as the conversion condition, to obtain the second converted vector. This is similar to the implementation of the vector conversion of the first latent representation vector and the first smoothed representation vector using the first smoothed representation vector as the conversion starting point and the first latent representation vector as the conversion condition, to obtain the first converted vector using the preset first vector conversion path in step S104. The vector conversion process in step S604 can be understood by referring to the relevant explanation in step S104. The only difference is that the objects of the vector conversion in step S104 are the first latent representation vector and the first smoothed representation vector, while the objects of the vector conversion in step S604 are the second latent representation vector and the second smoothed representation vector.

[0154] Step S604: Decode the second conversion vector to obtain target audio data with a fourth resolution.

[0155] It should be noted that decoding the second transformation vector in step S604 to obtain target audio data having the fourth resolution is implemented in the same manner as decoding the transformation vector in step S101 to obtain target audio data having the second resolution. The decoding process in step S604 can be understood by referring to the explanation of step S101. The only difference is that the decoding process in step S101 is performed on the transformation vector, while the decoding process in step S604 is performed on the second transformation vector.

[0156] Step S212: The server sends the target audio data to the terminal.

[0157] Step S213: The terminal displays the target audio data.

[0158] In some embodiments, the above-mentioned audio data processing method can also be implemented through a pre-trained data processing model, which includes a plurality of data conversion modules cascaded in sequence; the data conversion module includes: an audio data encoder, a vector converter, and an audio data decoder. In other words, the first audio data with a second resolution can be input into the pre-trained data conversion module, and the target audio data with a third resolution can be obtained through the pre-trained data conversion module. Based on this, the embodiment of the present application further provides a training method for a data conversion module, Figure 7This is a flow chart of the training method of the data conversion module provided in the embodiment of the present application. The training method of the data conversion module can be implemented by a model training module. The model training module can be a module in an electronic device. That is, the execution subject of the training method of the data conversion module can be a server or a terminal. The embodiment of the present application takes the execution subject of the training method of the data conversion module as an example. See Figure 7 , Figure 7 It shows that the server can train each data conversion module through the following steps S701 to S709:

[0159] Step S701: Obtain sample data.

[0160] In the embodiment of the present application, the sample data includes first sample audio data with a fifth resolution and reference audio data with a sixth resolution.

[0161] Step S702: interpolate the first sample audio data to obtain second sample audio data with a sixth resolution.

[0162] It should be noted that the interpolation processing performed on the first sample audio data in step S702 is implemented in the same manner as the interpolation processing performed on the first audio data in step S102 above. The interpolation process in step S702 can be understood by referring to the relevant explanation of step S102 above. The only difference is that the interpolation processing in step S702 is performed on the first sample audio data, while the interpolation processing in step S102 is performed on the first audio data.

[0163] Step S703: input the second sample audio data into the data conversion module to be trained.

[0164] Step S704: Encode the second sample audio data using the audio data encoder of the data conversion module to be trained to obtain a sample latent representation vector of the second sample audio data in the third latent space.

[0165] It should be noted that in step S704, the second sample audio data is encoded by the audio data encoder of the data conversion module to be trained. The audio data encoder of the data conversion module to be trained can be used to first extract the sample original feature vector from the waveform data of the second sample audio data; the dimension of the sample original feature vector can be a preset dimension; then, the sample original feature vector is vector compressed using a preset compression ratio to obtain the sample potential representation vector of the second sample audio data in the third latent space.

[0166] Step S705 : Smoothing the sample latent representation vector to obtain a sample smoothed representation vector of the second sample audio data in the third latent space.

[0167] It should be noted that the smoothing of the sample latent representation vector in step S705 is implemented in the same manner as the smoothing of the first latent representation vector in step S1032 above. The smoothing process in step S705 can be understood by referring to the explanation of step S1032 above. The only difference is that the smoothing process in step S705 is performed on the sample latent representation vector, while the smoothing process in step S1032 is performed on the first latent representation vector.

[0168] In step S706, the sample smoothed representation vector is used as the conversion starting point and the sample potential representation vector is used as the conversion condition. The sample potential representation vector and the sample smoothed representation vector are converted by the vector converter of the data conversion module to be trained to obtain a sample conversion vector.

[0169] It should be noted that in step S706, the vector conversion of the sample latent representation vector and the sample smoothed representation vector is performed by the vector converter of the data conversion module to be trained, using the sample smoothed representation vector as the conversion starting point and the sample latent representation vector as the conversion condition. This is similar to the implementation of the vector conversion of the first latent representation vector and the first smoothed representation vector using the first smoothed representation vector as the conversion starting point and the first latent representation vector as the conversion condition, using the preset first vector conversion path in step S104. The vector conversion process in step S706 can be understood by referring to the relevant explanation of step S104. The only difference is that the objects of the vector conversion in step S706 are the sample latent representation vector and the sample smoothed representation vector, while the objects of the vector conversion in step S104 are the first latent representation vector and the first smoothed representation vector.

[0170] Step S707 : decoding the sample conversion vector by the audio data decoder of the data conversion module to be trained to obtain predicted audio data with a sixth resolution.

[0171] It should be noted that the decoding of the sample conversion vectors by the audio data decoder of the data processing model to be trained in step S707 is implemented in the same manner as the decoding of the conversion vectors in step S101 above. The decoding process in step S707 can be understood by referring to the relevant explanation of step S101 above. The only difference is that the decoding process in step S707 is performed on the sample conversion vectors, while the decoding process in step S101 is performed on the conversion vectors.

[0172] Step S708: performing loss calculation based on the predicted audio data and the reference audio data to obtain a loss result.

[0173] In some embodiments, see Figure 8 , Figure 8It is shown that in step S708, the server performs loss calculation based on the predicted audio data and the reference audio data to obtain a loss result, which can be achieved by executing the following steps S7081 to S7085:

[0174] Step S7081 : In response to the current training process training at least one of the audio data encoder and the audio data decoder, a reconstruction error value is determined based on the predicted audio data and the reference audio data.

[0175] Step S7082: Determine the probability distribution error value based on the mean and variance of the sample potential representation vector.

[0176] Step S7083: Determine a multi-resolution error value based on the transformed data of the predicted audio data and the reference audio data at different resolutions.

[0177] Step S7084: Determine multi-period loss values ​​of the predicted audio data and the reference audio data in different periods.

[0178] Step S5085: Fuse the reconstruction error value, the probability distribution error value, the multi-resolution error value, and the multi-cycle loss value to obtain a loss result.

[0179] In an embodiment of the present application, if the current training process is to train at least one of the audio data encoder and the audio data decoder, the reconstruction error value, the probability distribution error value, the multi-resolution error value and the multi-cycle loss value can be calculated respectively, and then, based on the calculated reconstruction error value, the probability distribution error value, the multi-resolution error value and the multi-cycle loss value, the total error value is determined, and the total error value is determined as the loss result of the training, so that the model parameters in at least one of the audio data encoder and the audio data decoder are updated based on the loss result.

[0180] In other embodiments, the server performs loss calculation based on the predicted audio data and the reference audio data to obtain the loss result, which can also be achieved in the following way: in response to the current training process being after the training of the audio data encoder and the audio data decoder is completed, the vector converter is trained, and the loss result of the vector converter is determined based on the predicted audio data and the reference audio data.

[0181] In an embodiment of the present application, if the current training process is to train a vector converter, after obtaining the loss result, the model parameters in the vector converter can be updated based on the loss result.

[0182] It should be noted that, in the embodiment of the present application, the training process for the data conversion module may be to first train the audio data encoder and the audio data decoder, and then train the vector converter. When training the audio data encoder and the audio data decoder, the audio data encoder and the audio data decoder may be trained simultaneously, or the audio data encoder may be trained first, and then the audio data decoder.

[0183] Here, the method of simultaneously training the audio data encoder and decoder enables end-to-end learning, allowing the entire data conversion module to automatically adjust the parameters of the audio data encoder and decoder to minimize reconstruction error. This training approach ensures coordinated optimization between the audio data encoder and decoder, improving overall performance. Furthermore, end-to-end training optimizes the global performance of the entire data processing model, rather than local optimality. The parameter adjustment of the audio data encoder and decoder is based on the loss function of the entire data conversion module, which can better capture the global characteristics of the input data. Simultaneous training also allows for better utilization of the large amount of training data, reducing the risk of overfitting. The parameter adjustment of the audio data encoder and decoder is based on the entire dataset, rather than a local dataset. Furthermore, end-to-end training reduces training time because the audio data encoder and decoder do not need to be trained separately. For example, in audio enhancement tasks, simultaneous training of the audio data encoder and decoder can ensure that the generated audio has higher fidelity and less noise.

[0184] In an embodiment of the present application, a method of first training the audio data encoder and then training the audio data decoder, by first training the audio data encoder, can ensure that the audio data encoder can effectively extract the core features of the data. Then, based on these features, training the audio data decoder can ensure that the audio data decoder can better reconstruct the data. Training the audio data encoder first can reduce the complexity of the training process. The training of the audio data encoder can be carried out independently without considering the parameter adjustment of the audio data decoder, which can simplify the training process and reduce training time. Training the audio data encoder first can also improve the stability of the training process, because the parameter adjustment of the audio data encoder is based on the entire data set, rather than a local data set, so it can reduce fluctuations in the training process and improve the stability of the training. In addition, by training the audio data encoder first, the expressive power of the features can also be enhanced. The audio data encoder can learn more effective feature representations and provide better input for the audio data decoder.

[0185] In an embodiment of the present application, a method of first training the audio data encoder and the audio data decoder and then training the vector converter can ensure the coordinated optimization between the audio data encoder and the audio data decoder by first training the audio data encoder and the audio data decoder. Then, the vector converter is trained based on these optimized audio data encoders and audio data decoders to ensure that the vector converter can better process the potential representation vector. At the same time, through staged training, the complexity of the training process can be reduced, and the training of the audio data encoder and the audio data decoder can be carried out independently without considering the parameter adjustment of the vector converter, which can simplify the training process and reduce training time. In addition, through staged training, the stability of the training process can also be improved, because the parameter adjustment of the audio data encoder and the audio data decoder is based on the entire data set, rather than a local data set, which can reduce fluctuations in the training process and improve the stability of the training.

[0186] Step S709: Update the model parameters in the data conversion module to be trained based on the loss result to obtain a trained data conversion module.

[0187] In an embodiment of the present application, if the current training process is to train at least one of the audio data encoder and the audio data decoder, the loss result obtained is the loss result obtained by fusing the reconstruction error value, the probability distribution error value, the multi-resolution error value and the multi-cycle loss value, and the model parameters in at least one of the audio data encoder and the audio data decoder can be updated based on the loss result.

[0188] If the current model training process is to train the vector converter, the obtained loss result is the loss result used to train the vector converter, so the model parameters in the vector converter can be updated based on the loss result.

[0189] Through steps 701 to 709, different loss function terms are calculated when training different training objects. For example, when training the audio data encoder and audio data decoder, the loss result is obtained by fusing the reconstruction error value, probability distribution error value, multi-resolution error value, and multi-period loss value. By fusing multiple error values, the data conversion module not only focuses on the global reconstruction error but also considers the distribution of the latent representation, errors at different resolutions, and time periods. This makes the reconstructed data closer to the original data in multiple dimensions. Furthermore, the introduction of multi-resolution error values ​​and multi-period loss values ​​makes the data conversion module more robust when processing data of different scales and time periods, reducing errors caused by scale changes or temporal fluctuations. The introduction of the probability distribution error value makes the latent representation vector closer to the preset distribution, improving the smoothness and continuity of the latent space and facilitating subsequent vector conversion and generation tasks. By comprehensively considering multiple errors, the data conversion module can better generalize to unseen data during training, reducing the risk of overfitting. For another example, when training the vector transformer, the loss result is determined based on the predicted audio data and the reference audio data. In this way, by directly comparing the predicted data (i.e., predicted audio data) and the reference data (i.e., reference audio data), the vector converter can learn a more accurate vector conversion path, and the generated high-resolution data is closer to the real data. And the design of the loss function can ensure that the generated data is semantically consistent with the reference data, avoiding the generation of unreasonable artifacts or distortion. At the same time, the loss function based on the predicted data and the reference data can provide more direct feedback, making the training of the vector converter more efficient and stable. Of course, in some embodiments, different parameter data can also be introduced. In this way, by introducing a variety of reference data during the training process, the vector converter can better adapt to different input conditions and improve the generalization ability of the vector converter.

[0190] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0191] The present embodiment provides an audio data processing method. This method utilizes a multi-stage cascade modeling strategy, decomposing the super-resolution task layer by layer based on sampling rate. High-quality super-resolution is achieved in stages by independently constructing a VAE module and corresponding latent bridge model for each target sampling rate. Specifically, VAEs are constructed at three resolution levels (corresponding to 48kHz, 96kHz, and 192kHz, respectively); a latent bridge model is used to complete super-resolution from any sampling rate to 48kHz; the result is generated at 48kHz, then encoded through a VAE at 96kHz to obtain the target latent representation, constructing the second-stage super-resolution modeling process. Similarly, the modeling process from 96kHz to 192kHz is completed progressively using the same mechanism. Each stage is independently trained and optimized to maximize modeling quality and structural consistency within the frequency band.

[0192] In addition, the audio data processing method also adopts a structured prior enhancement mechanism. By filtering and degrading the waveform generated in the previous stage, the synthesized low-frequency dominant waveform that is more suitable for generation in the second stage and free of high-frequency artifacts is used as the starting point of the sampling path. Specifically: the waveform generated in the previous stage is low-pass filtered to effectively remove potential high-frequency artifacts and incomplete structures; the filtered waveform is encoded using a VAE encoder corresponding to the target sampling rate of the current stage to obtain a structurally degraded but stable latent space representation; during the sampling stage, the latent space representation is Gaussian smoothed to obtain a smoothed representation after Gaussian smoothing. This smoothed representation is used as the starting point, and the structurally degraded but stable latent space representation is used as a condition to guide the Schrödinger bridge model for path generation. This achieves active repair of spectral defects in the previous stage, while enhancing the local consistency and generation robustness of path modeling.

[0193] By erasing the erroneous high-frequency structure, a sampling starting point that is more suitable for the second-stage generation and free of high-frequency artifacts is constructed, reducing the negative impact of the previous stage error on the sampling path. During training, the real samples and the samples generated in the first stage are aligned to narrow the difference between the real samples and the samples generated in the first stage, so as to facilitate better adaptation during sampling. In conjunction with the complete latent space representation, the latent space representation is used as a condition to guide the model to restore the continuity and detail consistency of the mid- and high-frequency bands while retaining the low-frequency information. An automatic repair mechanism is provided for the model to automatically guide the generator to correct and complete the areas where the results in the previous stage were not ideal, thereby alleviating the problem of cascade error accumulation.

[0194] This embodiment provides an audio super-resolution system based on a multi-stage cascaded latent bridge model. The above-mentioned audio data processing method is implemented through the audio super-resolution system. The audio super-resolution system is suitable for super-resolution tasks of various audio signals such as speech and music. The overall architecture of the audio super-resolution system is as follows: Figure 9As shown, it includes an audio input module 801, a 48kHz potential bridge model 802 (corresponding to the above-mentioned vector converter), a linear interpolation module 803, a high-frequency filter 804, a 96kHz VAE 805 (corresponding to the above-mentioned audio data encoder), a Gaussian smoothing module 806, a 96kHz potential bridge model 807, a 192kHz VAE 808, a 192kHz potential bridge model 809 and an audio output module 810.

[0195] The universal audio super-resolution method based on the latent Schrödinger bridge model proposed in the embodiment of the present application introduces Schrödinger bridge modeling in the latent space learned by the variational autoencoder to construct the optimal probabilistic path between low-resolution and high-resolution latent representations, thereby achieving more efficient and higher-quality audio upsampling. Figure 10 This is a schematic diagram of the implementation flow of the audio super-resolution method provided in the embodiment of the present application, see Figure 10 , the method comprises the following steps:

[0196] Step S401: construct an encoder network from waveform space to latent space (i.e., the above-mentioned VAE encoder).

[0197] Here, we can first obtain a pair of low-resolution audio samples in the training dataset And the high-resolution audio corresponding to the low-resolution audio sample Then, the low-resolution audio samples are transformed into and high-resolution audio The waveforms are mapped to the latent space to obtain the latent representation vector of the low-resolution audio sample and the latent representation vector for high-resolution audio , as shown in the following formulas (1) and (2):

[0198] (1);

[0199] (2).

[0200] To improve modeling stability, a scaling factor can be applied here Normalize the latent space features uniformly to obtain the normalized potential representation vector , see the following formula (3) for details:

[0201] (3).

[0202] Here, the latent representation vector The latent representation vector including the above low-resolution audio samples and the latent representation vector for high-resolution audio .

[0203] The embodiment of the present application uses a VAE encoder with a compression ratio of 512 and sets the number of potential channels to 64, which retains key audio information while ensuring a high compression rate, providing a good foundation for subsequent bridging modeling.

[0204] Step S402 : constructing a Schrödinger bridge generation path (corresponding to the above-mentioned vector conversion path) in the latent space.

[0205] Here, first, in the latent space, we construct a (prior value) to (target value), the forward process is defined as a stochastic differential process (SDE), see the following formula (4):

[0206] (4);

[0207] in, represents the hidden state at time t, is the drift function, is the diffusion coefficient, represents standard Wiener noise.

[0208] During the training process, the optimal path starting from the low-resolution latent representation vector and targeting the high-resolution latent representation vector is fitted, so that the sampling process follows the data-to-data transition path rather than the traditional noise-to-data path, thereby improving sampling quality and efficiency.

[0209] Step S403: model training and loss function design.

[0210] In the embodiment of the present application, during the training of the VAE module (the VAE module here includes the above-mentioned VAE encoder and VAE decoder), the following objective function (5) can be used for optimization:

[0211] (5);

[0212] in, represents the total loss function of the VAE module (i.e., the loss result during the training process of at least one of the audio data encoder and the audio data decoder), It consists of several different loss terms, each of which has a weight coefficient. The following is an explanation of each parameter in formula (5): Represents the weight coefficient of the reconstruction loss in the waveform domain; Represents the reconstruction loss (i.e., the reconstruction error value mentioned above), which is used to measure the difference between the output reconstructed by the VAE module and the original input. Represents the weight coefficient of KL divergence loss; Represents the KL divergence loss (i.e., the above-mentioned probability distribution error value), which is used to measure the difference between the potential distribution output by the VAE encoder and the standard normal distribution. It should be noted that the above-mentioned KL divergence loss in the embodiment of the present application is used to constrain the distribution of latent variables to be close to the standard normal distribution. In order to balance the modeling ability and the bridging performance, the embodiment of the present application can be set =1e -7 , and supports arbitrary forms of distance metrics for reconstruction loss, such as L2 loss and perceptual loss. Represents the weight coefficient of the multi-resolution short-time Fourier transform (STFT) loss; Represents the multi-resolution STFT loss (i.e., the multi-resolution error value mentioned above), which is used to measure the difference between audio signals at different frequency resolutions. Represents the weight coefficient of multi-period loss; represents the multi-period loss (i.e., the multi-period loss value mentioned above), which measures the differences in the audio signal at different periods. These loss terms work together to enable the VAE encoder to learn the latent distribution of the input audio signal, thereby obtaining a latent representation vector of the input audio signal in latent space and ultimately generating output audio that is similar to the original input audio. The weight coefficients balance the contributions of the different loss terms and can be adjusted based on the specific task.

[0213] In the embodiment of the present application, the above reconstruction loss It can be calculated by the following formula (6):

[0214] (6)

[0215] Reconstruction loss It measures the difference between the reconstructed output of the VAE module and the original input. represents the original input data (ie, the input audio signal, the original audio data), and the subscript 48 indicates that the resolution of the input audio signal may be the resolution corresponding to the 48 kHz sampling rate. Represents the output data reconstructed by the VAE module (i.e., the audio signal output by the VAE decoder), which has the same sampling rate as the original input data. = indicates that the square of the L2 norm (Euclidean distance) is used to calculate the difference. Therefore, formula (6) represents the original input data With the reconstructed output data The L2 norm square difference between the original input data With the reconstructed output data The square of the Euclidean distance between them. Reconstruction loss The purpose is to enable the VAE module to reconstruct an output that is as similar as possible to the original input.

[0216] In the embodiment of the present application, the potential representation vector z can be expressed by the following formula (7): It can be expressed by the following formula (8):

[0217] (7);

[0218] (8)

[0219] Formula (7) and Formula (8) describe the encoding and decoding process of the VAE module, where Formula (7) represents the role of the VAE encoder. The VAE encoder is a function that converts the original input data into Mapped to a latent representation vector z in a latent space. Represents the parameters of the VAE encoder, which are optimized during the training process so that the input data can be effectively encoded into the latent space. Formula (8) represents the role of the VAE decoder (Decoder), which is also a function that maps the potential representation vector z in the latent space back to the space of the original input data to generate the reconstructed output data . Represents the parameters of the VAE decoder, which are optimized during training to be able to generate output data from the latent space that is similar to the original input data.

[0220] During the training process of the VAE module, the parameters of the VAE encoder and VAE decoder are jointly optimized to minimize the loss function. Through this process, the VAE module is able to learn the latent distribution of the input data and generate output data similar to the original input.

[0221] In the embodiment of the present application, the above KL divergence loss It can be calculated by the following formula (9):

[0222] (9)

[0223] KL divergence loss It measures the difference between the potential distribution of the VAE encoder output (that is, the potential representation vector output by the VAE encoder) and the standard normal distribution. The mean of the latent representation vector z. Denotes the variance of the latent representation vector z. D denotes the dimension of the latent representation vector z. The summation symbol ∑ in formula (9) denotes the summation of the latent representation vectors of all dimensions. KL divergence loss The purpose is to make the distribution of the potential representation vector z as close as possible to the standard normal distribution, that is, a normal distribution with a mean of 0 and a variance of 1. By minimizing the KL divergence loss, the VAE encoder can learn the potential distribution of the original input data and generate outputs similar to the original input data.

[0224] In the embodiment of the present application, the multi-resolution STFT loss It can be calculated by the following formula (10):

[0225] (10)

[0226] Multi-resolution STFT loss It measures the difference between the original input data and the reconstructed output data at different frequency resolutions. Where N represents the number of different resolutions of the STFT. Represents the original input data STFT at the i-th resolution. Represents the reconstructed output data STFT at the i-th resolution. Therefore, Equation (10) represents the original input data and the reconstructed output data The sum of the L2-norm squared differences between STFTs at different resolutions. Multi-resolution STFT loss The purpose is to make the reconstructed output data as close as possible to the original input data at different frequency resolutions, thereby improving the quality of the reconstructed signal.

[0227] In the embodiment of the present application, the multi-cycle loss It can be calculated by the following formula (11):

[0228] (11)

[0229] Multi-period loss It measures the difference between the original input data and the reconstructed output data at different periods. Represents the original input data After the output of the Multi-Period Discriminator (MPD). Represents the reconstructed output data The output of the multi-cycle discriminator. Therefore, formula (11) represents the original input data and the reconstructed output data The L2-norm squared difference between the outputs of the multi-cycle discriminator. Multi-cycle loss The purpose is to make the reconstructed output data as close as possible to the original input data at different periods, thereby improving the quality and authenticity of the reconstructed signal.

[0230] Here, a multi-period discriminator is a discriminator used in audio signal processing. It captures the periodic characteristics of a signal by analyzing it at different periods. In the context of the VAE module, the multi-period discriminator can be used to improve the quality of the generated audio signal, making it closer to the real audio signal.

[0231] In the embodiment of the present application, during the training of the Schrödinger bridge model, a unified generation objective function can be used. , measures the difference between the predicted trajectory and the target potential distribution, where the loss function of the Schrödinger bridge model is It is expressed as the following formulas (12) to (14):

[0232] (12);

[0233] (13);

[0234] in, (14);

[0235] Among them, formula (12) represents the loss function of the Schrödinger bridge model. The loss function of the Schrödinger bridge model is It measures the difference between the predicted trajectory and the target potential distribution. Specifically, Indicates the distribution of data Sampling , and, through the filter (Filter) of the sample The data obtained after filtering Perform expectation calculations. Indicates the expected calculation for time step t. Represents the prediction function of the Schrödinger bridge model. The prediction function of the Schrödinger bridge model uses parameters θ, current state , time step t and initial state to predict the target state. represents the target state. Therefore, the loss function of the Schrödinger bridge model is Indicates the predicted status and target state The purpose of this loss function is to enable the Schrödinger bridge model to accurately predict the target state, thereby improving the prediction performance of the Schrödinger bridge model.

[0236] Formula (13) represents the data Encode through the VAE encoder to obtain the potential representation vector in the latent space Specifically, is a vector in the latent space that represents the data encoded by the VAE encoder. k is an index value that belongs to the set (t, 0, 1) and represents different time steps or data points. Indicates that the data is encoded using VAE In the VAE module, the VAE encoder is a function that converts the original input data into A latent representation vector mapped into a latent space This latent representation vector It can be a mean and a variance that define the normal distribution of the latent variable. Through this encoding process, the VAE encoder is able to learn the latent distribution of the original input data, so that it can generate output data similar to the original input.

[0237] Formula (14) describes the potential representation vector in the Schrödinger bridge model Specifically, and represents the parameters in the Schrödinger bridge model, and Used to define the latent representation vector distribution. Indicates the mean , the covariance matrix is The normal distribution of , where, in formula (14), the mean , the covariance matrix . and are all parameters in the Schrödinger bridge model. and Used to calculate the latent representation vector The mean of . and It is also a parameter in the Schrödinger bridge model. and Used to calculate the latent representation vector The variance of . and represents the initial and target states in the Schrödinger bridge model. It is also a parameter in the Schrödinger bridge model. Used to calculate the latent representation vector Therefore, Formula (14) expresses the potential representation vector The probability distribution of is a normal distribution, whose mean and variance are determined by the parameters of the Schrödinger bridge model and the initial and target states. This normal distribution is used for prediction and inference in the Schrödinger bridge model.

[0238] In some embodiments, the loss function of the Schrödinger bridge model is For LBM, we directly optimize the L2 loss between the noise prediction at trajectory time step t and the actual noise at step t. The goal is to make The sampling process starting from (prior value) naturally converges to (target value, i.e., the latent representation vector of the reference audio data).

[0239] Step S404: sampling and evaluation analysis.

[0240] In the inference phase, the audio super-resolution system first inputs low-resolution audio (corresponding to the original audio data with the first resolution mentioned above) to the VAE encoder, and the low-resolution potential representation vector is obtained by the VAE encoder. , a high-resolution potential representation vector is obtained by sampling the Schrödinger bridge model , expressed as the following formula (15):

[0241] (15);

[0242] in, It represents the sampling process of the Schrödinger bridge model, wherein, in the embodiment of the present application, 50-step sampling can be adopted.

[0243] Then pass the VAE decoder The high-resolution latent representation vector Restore to high-resolution audio waveform , as shown in the following formula (16):

[0244] (16).

[0245] The audio data processing method provided in the embodiment of the present application avoids the lengthy sampling path starting from Gaussian noise by introducing a Schrödinger bridge generation path in the latent space of the VAE module, effectively shortens the generation trajectory, and improves the sampling efficiency and detail performance of the data processing model in high-resolution audio reconstruction.

[0246] when When the waveform is 48kHz (corresponding to the first audio data above), At this time, the 48kHz waveform passes through the linear interpolation module 803 to obtain the 96kHz first waveform (corresponding to the above-mentioned second audio data). The 96kHz first waveform is input into the high-frequency filter 804 for filtering, and then input into the 96kHz VAE module 805 to obtain the encoded waveform (corresponding to the above-mentioned first latent representation vector). The encoded waveform is input into the Gaussian smoothing module 806 to obtain a smoothed waveform (corresponding to the above-mentioned first smoothed representation vector). With the first smoothed representation vector as the conversion starting point and the first latent representation vector as the conversion condition, the first smoothed representation vector and the first latent representation vector are input into the 96kHz latent bridge model 807, and the 96kHz second waveform (corresponding to the above-mentioned target audio data) is obtained through decoding. The determination process of the 96kHz second waveform is specifically expressed as the following formulas (17) to (21):

[0247] (17)

[0248] (18)

[0249] (19)

[0250] (20)

[0251] (twenty one)

[0252] in, Represents the linear interpolation process, Filter represents the filtering process, Represents the waveform after filtering (corresponding to the above filtering data), represents the encoding process of the 96kHz VAE module 805, Represents the encoded 96kHz waveform, represents the smoothing process of the Gaussian smoothing module 806, represents the smoothed 96kHz waveform, represents the 96kHz waveform after vector conversion by the 96kHz potential bridge model 807 (corresponding to the first conversion vector mentioned above), represents the decoding process of the 96kHz VAE module 805, Represents the 96kHz second waveform (corresponding to the target audio data with the third resolution mentioned above).

[0253] Similarly, the modeling process from 96kHz to 192kHz is completed progressively based on the same mechanism mentioned above. The determination process of the second waveform of 192kHz is specifically expressed as the following formulas (22) to (26):

[0254] (twenty two)

[0255] (twenty three)

[0256] (twenty four)

[0257] (25)

[0258] (26)

[0259] in, It represents the 192kHz waveform after filtering. represents the encoding process of the 192kHz VAE module 808, Represents the encoded 192kHz waveform, represents the smoothing process of the Gaussian smoothing module 806, represents the smoothed 192kHz waveform, represents the 192kHz waveform after vector conversion by the 192kHz potential bridge model 809 (corresponding to the second conversion vector mentioned above), represents the decoding process of the 192kHz VAE module 808, Represents the 192kHz second waveform (corresponding to the target audio data with the fourth resolution mentioned above).

[0260] The above-mentioned cascaded multi-stage Schrödinger bridge generation structure C-LBM provided in the embodiment of the present application can adopt any-to-any training method for training each data conversion module in the cascaded multi-stage Schrödinger bridge generation structure C-LBM (i.e., the data processing model).

[0261] Figure 11 This is a comparison diagram of the technical implementation process of LBM provided in the embodiment of the present application and the technical implementation process of AudioSR in the related art. Figure 11 The upper part is the technical implementation process of AudioSR. It can be seen that AudioSR uses a double-cascade compression annotation paradigm; Figure 11 The lower part is the technical implementation process of LBM. It can be seen that LBM directly establishes a Schrödinger bridge mapping between waveform latent spaces.

[0262] Furthermore, based on the training process of each data conversion module in the cascaded multi-stage Schrödinger bridge generation structure C-LBM provided in the above embodiment, the embodiment of the present application further provides an any-to-any training method.

[0263] Because the training dataset (i.e., the sample data described above) contains relatively few high-sampling-rate audio, directly training the 48kHz data conversion module using only 48kHz audio results in poor model training. Using audio of varying sampling rates to train each data conversion module, while increasing the amount of data, significantly reduces training quality (because most datasets contain low-sampling-rate data). Therefore, the present invention proposes an any-to-any training paradigm.

[0264] Specifically, during training, the sampling rate of the low-resolution audio and the actual frequency of the high-resolution audio (sampling rate divided by 2) are both used as additional inputs to the data conversion module to be trained. In this way, the understanding of the frequencies of the two different boundary distributions enables LBM to clearly learn the any-to-any upsampling process of different frequency bands, thereby improving the upsampling performance to the target resolution sampling stage. Therefore, the loss function of each data conversion module during training is It can be expressed as the following formula (27):

[0265] (27).

[0266] in, represents the parameters of the Schrödinger bridge model; Represents the potential representation vector in the latent space of relatively high-resolution audio obtained from any to any training method, Represents the potential representation vector obtained after the relatively low-resolution audio in any-to-any training method is encoded by the VAE encoder; represents the diffusion time step of the Schrödinger bridge model; T represents the maximum value of the diffusion time step of the Schrödinger bridge model; The potential representation vector representing the original input audio at time step t of the Schrödinger bridge trajectory; Indicates the frequency of the input audio, Indicates the frequency of the target audio. and They represent the coefficients related to the offset and diffusion of the random process in the forward process of the Schrödinger bridge model in the data conversion module.

[0267] Through the above data processing process, by introducing the cascaded multi-stage Schrödinger bridge generation structure C-LBM, the 96kHz and 192kHz ultra-high sampling rate audio generation tasks that were originally difficult to model directly are transformed into a staged modeling problem in multiple frequency bands, thereby achieving better modeling quality and generation stability at each frequency band level, effectively breaking through the upper limit of the existing model's high-frequency modeling capabilities, and solving the problems of frequency band breakage and error accumulation in existing methods. In addition, an innovative prior enhancement mechanism is proposed, that is, at each sampling stage, a filtered degraded input waveform is constructed as the starting point, guiding the Schrödinger bridge to sample from a starting point that is more suitable for the second stage generation and free of high-frequency artifacts. While improving the stability of the model, it significantly enhances the ability to repair errors in the previous stage, solves the error accumulation and spectral discontinuity problems existing in the traditional cascade structure, and improves the listening quality and fidelity of the final generated audio.

[0268] Based on the audio data processing method described in the above embodiment, Figure 12 The structural block diagram of an audio data processing device provided in an embodiment of the present application is shown. The audio data processing device 100 can be a device in an electronic device (for example, a server). The audio data processing device can be implemented in software, which can be software in the form of programs and plug-ins, etc., including the following software modules: a conversion module 101, an interpolation processing module 102, a determination module 103, a vector conversion module 104 and a decoding processing module 105. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0269] Among them, the conversion module 101 is used to perform data conversion on the original audio data with a first resolution to obtain first audio data with a second resolution; the interpolation processing module 102 is used to perform interpolation processing on the first audio data to obtain second audio data with a third resolution; the third resolution is greater than the second resolution, and the second resolution is greater than the first resolution; the determination module 103 is used to determine the first latent representation vector of the second audio data in the first latent space, and the first smooth representation vector of the second audio data in the first latent space; the vector conversion module 104 is used to use the first smooth representation vector as the conversion starting point and the first latent representation vector as the conversion condition, and perform vector conversion on the first latent representation vector and the first smooth representation vector through a preset first vector conversion path to obtain a first conversion vector; the decoding processing module 105 is used to decode the first conversion vector to obtain the target audio data with the third resolution.

[0270] In some embodiments, the determination module 103 is further used to: encode the second audio data to obtain a first latent representation vector of the second audio data in a first latent space; and smooth the first latent representation vector to obtain a first smoothed representation vector of the second audio data in the first latent space.

[0271] In some embodiments, the audio data processing device 100 further includes a filtering processing module, and the filtering processing module is configured to perform filtering processing on the second audio data to obtain filtered data.

[0272] In some embodiments, the determining module 103 is further configured to: perform encoding processing on the filtered data to obtain a first latent representation vector of the second audio data in a first latent space.

[0273] In some embodiments, the determination module 103 is further used to: extract a first eigenvector from the filtered data; the dimension of the first eigenvector is a preset dimension; and perform vector compression on the first eigenvector using a preset compression ratio to obtain a first latent representation vector of the second audio data in a first latent space.

[0274] In some embodiments, the interpolation processing module 102 is further configured to perform linear interpolation processing on the waveform data of the first audio data to obtain second audio data having the third resolution.

[0275] In some embodiments, the first vector conversion path includes a vector conversion function; the vector conversion function is a function in a pre-trained vector converter; the vector conversion module 104 is also used to: input the first smooth representation vector and the first latent representation vector into the vector converter; perform vector splicing on the first smooth representation vector and the first latent representation vector through the preprocessing layer of the vector converter to obtain a spliced ​​vector; perform random differentiation processing on the spliced ​​vector through the vector conversion function of the vector converter to obtain the first conversion vector.

[0276] In some embodiments, the audio data processing device 100 also includes a re-execution module, which is used to: perform interpolation processing on the target audio data to obtain third audio data with a fourth resolution; the fourth resolution is greater than the third resolution; determine the second latent representation vector of the third audio data in the second latent space, and the second smooth representation vector of the third audio data in the second latent space; use the second smooth representation vector as a conversion starting point and the second latent representation vector as a conversion condition, perform vector conversion on the second latent representation vector and the second smooth representation vector through a preset second vector conversion path to obtain a second conversion vector; and decode the second conversion vector to obtain the target audio data with the fourth resolution.

[0277] In some embodiments, the audio data processing method is implemented by a data processing model; the data processing model includes a plurality of data conversion modules cascaded in sequence; the data conversion module includes: an audio data encoder, a vector converter and an audio data decoder; the audio data processing device 100 also includes a training module, the training module is used to: obtain sample data; the sample data includes first sample audio data with a fifth resolution and reference audio data with a sixth resolution; interpolate the first sample audio data to obtain second sample audio data with the sixth resolution; input the second sample audio data into the data conversion module to be trained; encode the second sample audio data through the audio data encoder of the data conversion module to be trained, and obtain the second sample audio data in the third potential space The sample potential representation vector between the first and second latent spaces is smoothed to obtain the sample smoothed representation vector of the second sample audio data in the third latent space; with the sample smoothed representation vector as the conversion starting point and the sample potential representation vector as the conversion condition, the sample potential representation vector and the sample smoothed representation vector are vector-converted by the vector converter of the data conversion module to be trained to obtain a sample conversion vector; the sample conversion vector is decoded by the audio data decoder of the data conversion module to be trained to obtain predicted audio data with the sixth resolution; loss calculation is performed based on the predicted audio data and the reference audio data to obtain a loss result; the model parameters in the data conversion module to be trained are updated based on the loss result to obtain a trained data conversion module.

[0278] In some embodiments, the training module is also used to: in response to the current training process being to train at least one of the audio data encoder and the audio data decoder, determine a reconstruction error value based on the predicted audio data and the reference audio data; determine a probability distribution error value based on the mean and variance of the sample potential representation vector; determine a multi-resolution error value based on the transformed data of the predicted audio data and the reference audio data at different resolutions; determine multi-period loss values ​​of the predicted audio data and the reference audio data in different periods; and fuse the reconstruction error value, the probability distribution error value, the multi-resolution error value and the multi-period loss value to obtain the loss result.

[0279] In some embodiments, the training module is further configured to update model parameters in at least one of the audio data encoder and the audio data decoder based on the loss result.

[0280] In some embodiments, the training module is also used to: in response to the current training process being after completing the training of the audio data encoder and the audio data decoder, train the vector converter, and determine the loss result of the vector converter based on the predicted audio data and the reference audio data.

[0281] In some embodiments, the training module is further used to update the model parameters in the vector converter based on the loss result.

[0282] It should be noted that the description of the device embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. For technical details not disclosed in the device embodiment, please refer to the description of the method embodiment of the present application for understanding.

[0283] An embodiment of the present application provides an electronic device, Figure 13 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 13 As shown, the electronic device 130 includes: at least one processor 131 ( Figure 13 Only one is shown in the figure), a memory 132, and computer executable instructions 133 stored in the memory 132 and executable on at least one processor 131. When the processor 131 executes the executable instructions 133, the steps of any of the above-mentioned audio data processing method embodiments are implemented.

[0284] The electronic device may include but is not limited to a processor 131 and a memory 132. It will be understood by those skilled in the art that Figure 13This is merely an example of the electronic device 130 and does not constitute a limitation on the electronic device 130 . The electronic device 130 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0285] The processor 131 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPG), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0286] In some embodiments, the memory 132 may be an internal storage unit of the electronic device 130, such as a hard drive or memory of the electronic device 130. In other embodiments, the memory 132 may also be an external storage device of the electronic device 130, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card, etc. equipped on the electronic device 130. Furthermore, the memory 132 may include both an internal storage unit of the electronic device 130 and an external storage device. The memory 132 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of a computer program. The memory 132 may also be used to temporarily store data that has been output or is about to be output.

[0287] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the audio data processing method described in the present invention.

[0288] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the audio data processing method provided in the embodiment of the present application, for example, Figure 1 The audio data processing method shown.

[0289] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0290] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0291] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0292] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0293] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A method for processing audio data, characterized in that: The method comprises: Performing data conversion on original audio data having a first resolution to obtain first audio data having a second resolution; performing interpolation processing on the first audio data to obtain second audio data having a third resolution; the third resolution is greater than the second resolution, and the second resolution is greater than the first resolution; Determine a first latent representation vector of the second audio data in a first latent space, and a first smoothed representation vector of the second audio data in the first latent space; Taking the first smoothed representation vector as a conversion starting point and the first latent representation vector as a conversion condition, performing vector conversion on the first latent representation vector and the first smoothed representation vector through a preset first vector conversion path to obtain a first conversion vector; The first conversion vector is decoded to obtain target audio data with the third resolution; the first resolution, the second resolution, and the third resolution are three different sampling rates respectively.

2. The method according to claim 1, characterized in that The determining of a first latent representation vector of the second audio data in a first latent space and a first smoothed representation vector of the second audio data in the first latent space comprises: encoding the second audio data to obtain a first latent representation vector of the second audio data in a first latent space; The first latent representation vector is smoothed to obtain a first smoothed representation vector of the second audio data in the first latent space.

3. The method according to claim 2, characterized in that Before encoding the second audio data to obtain a first latent representation vector of the second audio data in a first latent space, the method further includes: performing filtering processing on the second audio data to obtain filtered data; Accordingly, encoding the second audio data to obtain a first latent representation vector of the second audio data in a first latent space includes: The filtered data is encoded to obtain a first latent representation vector of the second audio data in a first latent space.

4. The method according to claim 3, characterized in that The encoding process of the filtered data to obtain a first latent representation vector of the second audio data in a first latent space includes: Extracting a first eigenvector from the filtered data; the dimension of the first eigenvector is a preset dimension; The first feature vector is compressed using a preset compression ratio to obtain a first latent representation vector of the second audio data in a first latent space.

5. The method according to claim 1, wherein The interpolating the first audio data to obtain second audio data having a third resolution includes: Linear interpolation processing is performed on the waveform data of the first audio data to obtain second audio data with the third resolution.

6. The method according to claim 1, characterized in that The first vector conversion path includes a vector conversion function; the vector conversion function is a function in a pre-trained vector converter; The step of performing vector conversion on the first latent representation vector and the first smoothed representation vector using the first smoothed representation vector as a conversion starting point and the first latent representation vector as a conversion condition through a preset first vector conversion path to obtain a first conversion vector includes: Inputting the first smoothed representation vector and the first latent representation vector into the vector converter; Performing vector concatenation on the first smoothed representation vector and the first latent representation vector through a preprocessing layer of the vector converter to obtain a concatenated vector; The first conversion vector is obtained by performing random differentiation processing on the spliced ​​vector using a vector conversion function of the vector converter.

7. The method according to claim 1, characterized in that After decoding the first conversion vector to obtain target audio data with the third resolution, the method further includes: performing interpolation processing on the target audio data to obtain third audio data having a fourth resolution; the fourth resolution being greater than the third resolution; determining a second latent representation vector of the third audio data in a second latent space, and a second smoothed representation vector of the third audio data in the second latent space; Taking the second smoothed representation vector as a conversion starting point and the second latent representation vector as a conversion condition, performing vector conversion on the second latent representation vector and the second smoothed representation vector through a preset second vector conversion path to obtain a second conversion vector; The second conversion vector is decoded to obtain target audio data with the fourth resolution.

8. The method according to any one of claims 1 to 7, characterized in that The audio data processing method is implemented by a data processing model; the data processing model includes a plurality of data conversion modules cascaded in sequence; the data conversion module includes: an audio data encoder, a vector converter and an audio data decoder; The method further comprises: training each of the data conversion modules by the following steps: Acquire sample data; the sample data includes first sample audio data having a fifth resolution and reference audio data having a sixth resolution; performing interpolation processing on the first sample audio data to obtain second sample audio data having the sixth resolution; Inputting the second sample audio data into a data conversion module to be trained; encoding the second sample audio data using the audio data encoder of the data conversion module to be trained to obtain a sample potential representation vector of the second sample audio data in a third latent space; performing smoothing processing on the sample latent representation vector to obtain a sample smoothed representation vector of the second sample audio data in the third latent space; Taking the sample smoothed representation vector as a conversion starting point and the sample potential representation vector as a conversion condition, the vector converter of the data conversion module to be trained performs vector conversion on the sample potential representation vector and the sample smoothed representation vector to obtain a sample conversion vector; Decoding the sample conversion vector by an audio data decoder of the data conversion module to be trained to obtain predicted audio data having the sixth resolution; Performing loss calculation based on the predicted audio data and the reference audio data to obtain a loss result; The model parameters in the data conversion module to be trained are updated based on the loss result to obtain a trained data conversion module.

9. The method according to claim 8, characterized in that The performing loss calculation based on the predicted audio data and the reference audio data to obtain a loss result includes: In response to a current training process training at least one of the audio data encoder and the audio data decoder, determining a reconstruction error value based on the predicted audio data and the reference audio data; Determining a probability distribution error value based on the mean and variance of the sample potential representation vector; determining a multi-resolution error value based on transformation data of the predicted audio data and the reference audio data at different resolutions; determining multi-period loss values ​​of the predicted audio data and the reference audio data in different periods; fusing the reconstruction error value, the probability distribution error value, the multi-resolution error value, and the multi-period loss value to obtain the loss result; Accordingly, updating the model parameters in the data conversion module to be trained based on the loss result includes: Based on the loss result, model parameters in at least one of the audio data encoder and the audio data decoder are updated.

10. The method according to claim 9, characterized in that The performing loss calculation based on the predicted audio data and the reference audio data to obtain a loss result includes: In response to the current training process being after the training of the audio data encoder and the audio data decoder is completed, training the vector converter, and determining a loss result of the vector converter based on the predicted audio data and the reference audio data; Accordingly, updating the model parameters in the data conversion module to be trained based on the loss result includes: Model parameters in the vector converter are updated based on the loss result.

11. An audio data processing device, characterized in that: include: a conversion module, configured to perform data conversion on original audio data having a first resolution to obtain first audio data having a second resolution; an interpolation processing module, configured to perform interpolation processing on the first audio data to obtain second audio data having a third resolution; the third resolution is greater than the second resolution, and the second resolution is greater than the first resolution; a determination module, configured to determine a first latent representation vector of the second audio data in a first latent space, and a first smoothed representation vector of the second audio data in the first latent space; a vector conversion module, configured to perform vector conversion on the first latent representation vector and the first smoothed representation vector via a preset first vector conversion path, using the first smoothed representation vector as a conversion starting point and the first latent representation vector as a conversion condition, to obtain a first conversion vector; A decoding processing module is used to decode the first conversion vector to obtain target audio data with the third resolution; the first resolution, the second resolution and the third resolution are three different sampling rates respectively.

12. An electronic device, characterized in that: include: a memory for storing computer-executable instructions; The processor is configured to implement the audio data processing method according to any one of claims 1 to 10 when executing the computer-executable instructions stored in the memory.

13. A computer-readable storage medium, characterized in that Computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, the audio data processing method according to any one of claims 1 to 10 is implemented.

14. A computer program product, characterized in that The computer program product includes computer-executable instructions stored in a computer-readable storage medium; Wherein, when the processor of the electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, the audio data processing method according to any one of claims 1 to 10 is implemented.

Images