Audio super-resolution generation method and device, equipment and medium

Abstract

The invention relates to the technical field of voice semantics, can be applied to business system platforms such as financial science and technology, medical health and the like, and discloses an audio super-resolution generation method, device, equipment and medium, the method comprises the following steps: obtaining an original audio, performing audio enhancement processing on the original audio to obtain an audio to be processed, performing audio framing on the to-be-processed audio to obtain a spectrum feature sequence; performing audio feature coding on the spectrum feature sequence to obtain a low-resolution audio feature sequence; performing diffusion distillation processing on the low-resolution audio feature sequence to obtain an optimized low-dimensional audio feature sequence; performing deconvolution decoding processing on the optimized low-dimensional audio feature sequence to obtain a target high-dimensional audio feature sequence; and generating a target super-resolution audio of the to-be-processed audio according to the target high-dimensional audio feature sequence. According to the invention, the low-resolution audio quality and the processing efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of speech semantics technology, and in particular to an audio super-resolution generation method, device, equipment and medium. Background Art

[0002] With the acceleration of globalization and the popularization of digital technology, the demand for audio super-resolution processing has increased significantly in scenarios such as healthcare and fintech. For example, in audio restoration, music reconstruction, speech enhancement, and acoustic applications, traditional methods based on interpolation, spectral estimation, or traditional signal processing techniques are difficult to effectively improve the quality of low-resolution audio.

[0003] For example, in telemedicine in healthcare scenarios, doctors use electronic stethoscopes to collect heart sounds, lung sounds, or bowel sounds from patients for diagnosis. However, due to the low sampling rate of the device, environmental noise interference, or network transmission compression, the audio signal may lose high-frequency details or be contaminated by noise, affecting the recognition of pathological features (such as heart murmurs and abnormal respiratory sounds).

[0004] For example, in the field of financial technology, bank telephone customer service needs to verify user identity through voiceprint recognition, but low-quality telephone recordings (narrowband coding, background noise) make the voiceprint features blurred, increasing the risk of fraud. Or in a high-noise environment such as a trading hall, traders give instructions by voice, and background noise may cause the automatic speech recognition system to misread key parameters such as stock codes and prices, leading to a series of problems such as transaction failures.

[0005] Existing technologies for improving the quality of low-resolution audio often use methods based on interpolation, spectral estimation, or traditional signal processing techniques. However, these methods often suffer from limited restoration effects and insufficient noise processing capabilities. In recent years, deep learning methods have been introduced to audio super-resolution, such as models based on convolutional neural networks (CNNs), recurrent neural networks (RNNs), and generative adversarial networks (GANs). However, these models often use multi-stage processes (for example, generating intermediate features before waveform reconstruction), resulting in complex processing, high consumption of computational resources, and limited application in scenarios with high low-latency and real-time requirements.

[0006] Therefore, how to improve the quality and processing efficiency of low-resolution audio has become an urgent problem to be solved. Summary of the Invention

[0007] The present invention provides an audio super-resolution generation method, apparatus, device and medium, the main purpose of which is to solve the problems of low-resolution audio quality and poor processing efficiency.

[0008] In a first aspect, to achieve the above-mentioned objectives, the present invention provides an audio super-resolution generation method, comprising:

[0009] Acquire original audio, perform audio enhancement processing on the original audio to obtain audio to be processed, and perform audio framing on the audio to be processed to obtain a frequency spectrum feature sequence;

[0010] Performing audio feature encoding on the spectral feature sequence to obtain a low-resolution audio feature sequence;

[0011] Performing diffusion distillation processing on the low-resolution audio feature sequence to obtain an optimized low-dimensional audio feature sequence;

[0012] Performing deconvolution decoding on the optimized low-dimensional audio feature sequence to obtain a target high-dimensional audio feature sequence;

[0013] Generate target super-resolution audio of the audio to be processed according to the target high-dimensional audio feature sequence.

[0014] In a second aspect, the present invention further provides an audio super-resolution generation device, comprising:

[0015] An audio framing module is used to obtain original audio, perform audio enhancement processing on the original audio to obtain audio to be processed, and perform audio framing on the audio to be processed to obtain a spectral feature sequence;

[0016] A feature encoding module, configured to perform audio feature encoding on the spectral feature sequence to obtain a low-resolution audio feature sequence;

[0017] a diffusion distillation module, configured to perform diffusion distillation processing on the low-resolution audio feature sequence to obtain an optimized low-dimensional audio feature sequence;

[0018] a deconvolution decoding module, configured to perform deconvolution decoding on the optimized low-dimensional audio feature sequence to obtain a target high-dimensional audio feature sequence;

[0019] A target audio generation module is used to generate target super-resolution audio of the audio to be processed according to the target high-dimensional audio feature sequence.

[0020] In a third aspect, the present invention further provides an electronic device, comprising:

[0021] at least one processor; and,

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can perform the above-mentioned audio super-resolution generation method.

[0024] In a fourth aspect, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned audio super-resolution generation method.

[0025] In the embodiment of the present invention, by performing audio enhancement (such as spectral subtraction, deep learning noise reduction) on the original audio, background noise (such as environmental noise, equipment current noise) can be effectively removed, the noise resistance can be improved, the false recognition rate can be significantly reduced, and the spectrum leakage can be reduced, making the spectrum features (such as Mel spectrum) more stable; by pooling and full connection dimensionality reduction, the spectrum features are compressed to a low-resolution sequence, the subsequent calculation amount is reduced, the robustness of the features to environmental changes is enhanced, and the generalization ability of different audio types (such as speech, music, and environmental sounds) is improved; the Gaussian noise in the diffusion distillation is used to simulate the random disturbance in the audio distribution, and the student model learns the stable feature extraction ability of the teacher model in a noisy environment through distillation, thereby enhancing the actual The robustness to noise greatly improves the accuracy of audio reconstruction, ensures high fidelity, and improves audio quality; the deconvolution operation gradually restores the dimension of the feature sequence through upsampling, supplements the detailed information lost in the low-dimensional features such as high-frequency components and transient changes, improves the audio resolution, and also reduces the amount of calculation, improves the real-time performance of audio reconstruction, and integrates the rhythm parameters through the rhythm control network. The decoded feature sequence can better simulate the timing characteristics of natural audio and reduce the mechanical feel; by extracting and fusing multi-dimensional information such as time domain and frequency domain from the high-dimensional audio feature sequence, the generated audio waveform can more accurately restore the details of the original audio, including high-frequency components, transient changes, etc., thereby significantly improving the resolution and clarity of the audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0027] Figure 1 A schematic diagram of an application environment of an audio super-resolution generation method according to an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of a flow chart of an audio super-resolution generation method provided by one embodiment of the present invention;

[0029] Figure 3 A schematic diagram of a process for performing diffusion distillation processing on the low-resolution audio feature sequence provided by one embodiment of the present invention;

[0030] Figure 4 A schematic diagram of a module of an audio super-resolution generation device provided by one embodiment of the present invention;

[0031] Figure 5 A schematic structural diagram of an electronic device for implementing an audio super-resolution generation method provided by one embodiment of the present invention;

[0032] Figure 6 Another structural schematic diagram of an electronic device for implementing an audio super-resolution generation method provided by one embodiment of the present invention.

[0033] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, and to fully understand and implement how the present disclosure applies technical means to solve technical problems and achieve the corresponding technical effects, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The embodiments of the present disclosure and the various features in the embodiments can be combined with each other without conflict, and the technical solutions formed are all within the scope of protection of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present disclosure.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, apparatus, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] An embodiment of the present application provides an audio super-resolution generation method, and the execution subject of the audio super-resolution generation method includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the device provided by the embodiment of the present application. In other words, the audio super-resolution generation method can be executed by software or hardware installed on a terminal device or a server device. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0037] The present invention provides an audio super-resolution generation method, which can be applied to Figure 1 application environment. Among them, the client communicates with the server through the network. The server can obtain the original audio through the client, and through audio enhancement (such as spectral subtraction, deep learning noise reduction) and other processing of the original audio, it can effectively remove background noise (such as environmental noise, equipment current sound), improve noise resistance, and significantly reduce the false recognition rate. At the same time, it reduces spectrum leakage and makes spectrum features (such as Mel spectrum) more stable; through pooling and full connection dimensionality reduction, the spectrum features are compressed to a low-resolution sequence, which reduces the subsequent calculation amount, enhances the robustness of features to environmental changes, and improves the generalization ability of different audio types (such as speech, music, and environmental sounds); the Gaussian noise in the diffusion distillation simulates the random perturbation in the audio distribution, and the student model learns the stable feature extraction ability of the teacher model in a noisy environment through distillation, thereby enhancing the actual noise. Robustness greatly improves the accuracy of audio reconstruction, ensures high fidelity, and improves audio quality; the deconvolution operation gradually restores the dimension of the feature sequence through upsampling, supplements the detailed information lost in the low-dimensional features such as high-frequency components and transient changes, improves the audio resolution, and also reduces the amount of calculation, improves the real-time performance of audio reconstruction, and integrates the rhythm parameters through the rhythm control network. The decoded feature sequence can better simulate the timing characteristics of natural audio and reduce the mechanical feel; by extracting and integrating multi-dimensional information such as time domain and frequency domain from the high-dimensional audio feature sequence, the generated audio waveform can more accurately restore the details of the original audio, including high-frequency components, transient changes, etc., thereby significantly improving the resolution and clarity of the audio, and finally feeding the target audio output back to the client. Among them, the client can be but is not limited to various personal computers, laptops, smart phones, tablets and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0038] Reference Figure 2 FIG. 1 is a flow chart of a method for generating audio super-resolution according to an embodiment of the present invention. In this embodiment, the method for generating audio super-resolution includes:

[0039] S1. Obtain original audio, perform audio enhancement processing on the original audio to obtain audio to be processed, and perform audio framing on the audio to be processed to obtain a spectrum feature sequence.

[0040] In an embodiment of the present invention, the audio enhancement processing includes processing such as unified sampling rate, denoising and normalization; the audio framing refers to framing the audio to be processed through short-time Fourier transform and using a window function to convert it into a spectrum representation, thereby obtaining a spectrum feature sequence.

[0041] In the embodiment of the present invention, the original audio refers to audio data that has not been processed in any way, and can be a digital record captured directly from the sound source, which retains the waveform characteristics of the original sound wave (such as frequency, amplitude, and time information).

[0042] The present invention can obtain original audio through hardware devices, software generation and other methods. Specifically, human voice or environmental sound can be directly recorded through mobile phones, voice recorders, professional sound cards and other devices, or audio data can be recorded in real time through Internet of Things devices such as smart speakers and wearable devices (such as smart watches). It can also generate synthetic speech as original audio data, such as pre-recorded speech of customer service robots.

[0043] For example, in application examples in the medical and health field, for disease diagnosis and monitoring, coughing and breathing sounds can be used as original audio to identify respiratory diseases. Specifically, an electronic stethoscope can be used to record the patient's lung sounds and extract features such as moist rales and wheezing, thereby assisting in the diagnosis of diseases such as pneumonia and asthma.

[0044] For example, in the application examples in the field of financial technology, voiceprint recognition can replace traditional passwords for identity authentication and anti-fraud. Specifically, the biometric features such as the vocal tract shape and pronunciation habits in the target user's voice can be extracted to create a unique voiceprint template, and spectral analysis can be performed based on the voiceprint template to achieve voiceprint recognition.

[0045] In the embodiment of the present invention, performing audio enhancement processing on the original audio to obtain the audio to be processed includes:

[0046] Performing sampling rate unification processing on the original audio to obtain unified audio;

[0047] De-noising the unified audio to obtain interference-free audio;

[0048] Normalization is performed on the interference-free audio to obtain audio to be processed.

[0049] Specifically, the unified sampling rate processing eliminates sampling rate differences between audio sources to ensure consistency in subsequent feature processing. Different devices may collect audio at different sampling rates, such as a medical stethoscope at 44.1kHz vs. a telephone recording at 8kHz. Unifying the sampling rate can avoid distortion in spectrum analysis.

[0050] Specifically, sampling rate unification can be achieved through upsampling or downsampling. Upsampling refers to increasing the low sampling rate audio in the original audio (such as 8kHz telephone recording) to the target sampling rate (such as 16kHz) through an interpolation algorithm (linear interpolation, polynomial interpolation); downsampling refers to reducing the high sampling rate audio in the original audio (such as 48kHz medical equipment recording) to the target sampling rate through an anti-aliasing filter (such as an FIR filter) to prevent high-frequency signal aliasing.

[0051] For example, in medical and health scenarios, it is often necessary to retain biological signal details such as high-frequency components of heart sounds, so downsampling is usually chosen. In fintech business scenarios, it is necessary to prioritize voice clarity, so upsampling is usually chosen, which can balance quality and storage costs.

[0052] Specifically, denoising separates the target audio signal from background noise, improving the signal-to-noise ratio (SNR). This is particularly crucial for low-quality audio (such as phone recordings and audio collected by wearable devices). Specifically, denoising models such as U-Net and Conv-TasNet can be used to unify the input audio and output interference-free audio.

[0053] In detail, the normalization processing refers to unifying the audio amplitude range to prevent volume fluctuations from affecting subsequent processing such as speech recognition and feature extraction. Among them, peak normalization can be performed to linearly scale the audio amplitude to a maximum peak value equal to the target value (such as -1dBFS) to prevent clipping, making it more consistent with the auditory characteristics of the human ear and suitable for emotional analysis of customer service conversations.

[0054] In the embodiment of the present invention, the step of dividing the audio to be processed into frames to obtain a frequency spectrum feature sequence includes:

[0055] Performing short-time Fourier transform on the audio to be processed to obtain an audio frame sequence of fixed length;

[0056] Performing point-by-point multiplication processing on each audio frame in the audio frame sequence and a preset window function to obtain a windowed audio frame sequence;

[0057] The windowed audio frame sequence is subjected to time-frequency transformation processing to obtain a corresponding frequency spectrum feature sequence.

[0058] In an embodiment of the present invention, the continuous audio signal in the audio to be processed is divided into short time frames. The frame length (Frame Length) can be preset to 20-40ms. For example, at a 16kHz sampling rate, the frame length is 512 points, and the frame shift (Frame Shift) is preset to 50% of the frame length (such as 256 points) to avoid excessive overlap or loss of adjacent frame information. The audio to be processed is slidingly segmented according to the set frame length and frame shift. For example, if the total length of the audio is 3 seconds, the frame length is 512 points, and the frame shift is 256 points, a total of about 117 frames can be segmented (the specific number of frames depends on the total length of the audio).

[0059] In detail, the window function can be a Hamming window, a Hanning window, etc., and each frame of audio signal is multiplied point by point with the selected window function. For example, if the frame length is 512 points, the window function is also a sequence with a length of 512. Each sampling point of each frame is multiplied with the corresponding point of the window function, which can make the signal gradually approach zero at both ends of the frame, thereby reducing the truncation effect to obtain a windowed audio frame sequence.

[0060] The time-frequency transformation converts the time domain signal of the windowed audio frame sequence into a frequency domain complex spectrum including amplitude and phase. The spectrum feature sequence is a low-resolution multidimensional tensor representation including amplitude spectrum and phase spectrum, and the dimension of each spectrum frame corresponds to the fixed length of the audio frame.

[0061] The present invention performs a fast Fourier transform (FFT) on each frame of the windowed audio frame sequence to convert the time domain signal into a frequency domain complex sequence. The number of FFT points is usually consistent with the frame length (such as 512 points), and can also be expanded to a higher resolution (such as 1024 points) by zero padding, which corresponds to the amplitude and phase information of the frequency components respectively.

[0062] In an embodiment of the present invention, audio enhancement (such as spectral subtraction and deep learning noise reduction) can effectively remove background noise (such as environmental noise and equipment current noise), improve noise resistance, and significantly reduce the false recognition rate. At the same time, the enhanced audio is framed to avoid inter-frame fluctuations caused by noise, while reducing spectrum leakage and making spectrum features (such as Mel spectrum) more stable.

[0063] S2. Perform audio feature encoding on the spectrum feature sequence to obtain a low-resolution audio feature sequence.

[0064] In the embodiment of the present invention, multi-layer convolution enhancement, pooling dimensionality reduction, activation processing, etc. are performed on the spectral feature sequence to map it to a low-dimensional latent space to achieve feature optimization and dimensionality compression.

[0065] In the embodiment of the present invention, the step of performing audio feature encoding on the spectral feature sequence to obtain a low-resolution audio feature sequence includes:

[0066] Performing multi-layer convolution on the spectral feature sequence to obtain an audio convolution enhanced feature sequence;

[0067] Performing pooling dimensionality reduction processing on the audio convolution enhanced feature sequence to obtain an audio compression feature sequence;

[0068] Performing activation function processing on the audio compression feature sequence to obtain an audio activation feature sequence;

[0069] Performing multi-layer feature fusion on the audio activation feature sequence to obtain an audio fusion feature sequence;

[0070] Performing self-attention calculation on the audio fusion feature sequence to obtain an attention-weighted feature sequence;

[0071] Performing a fully connected dimensionality reduction process on the attention-weighted feature sequence to obtain a low-resolution audio feature sequence of the audio to be processed.

[0072] In this embodiment, multiple convolution operations are performed on an input spectral feature sequence (e.g., a Mel-spectrogram), using convolution kernels to extract local frequency and temporal patterns (e.g., harmonics and formants) to enhance feature representation. Each convolution layer uses multiple filters to scan local regions of the spectrum, generating higher-level feature maps. For example, the first layer might capture edges or textures, while subsequent layers gradually aggregate into more complex patterns.

[0073] In detail, the feature sequence after convolution enhancement is pooled (such as maximum pooling or average pooling) to reduce the feature dimension, retain significant information while reducing the amount of computation. Specifically, the pooling window is slid in the time or frequency dimension, and the maximum value (or average value) within the window is taken as the output. For example, a 2×2 pooling window can halve the size of the feature map while highlighting the dominant audio features.

[0074] The present invention applies a nonlinear activation function such as ReLU to the feature sequence after pooling, introduces nonlinearity, can set negative values to zero, retain positive values, enhance the sparsity and discriminability of features, fuse the activated features with features from other layers such as skip connections or parallel branches, combine multi-scale information, and improve the comprehensiveness of features. Specifically, features at different levels can be integrated by splicing or summing.

[0075] In detail, the fused feature sequence is processed with a self-attention mechanism to dynamically capture long-term dependencies and highlight the contributions of key time steps or frequency points. Specifically, the similarity of each position to other positions is calculated (by scaling the dot product), and attention weights are generated. The features are weighted and summed according to the weights to generate an attention-weighted feature sequence. For example, in speech, the features corresponding to key phonemes such as vowels or consonants can be enhanced, and the attention-weighted feature sequence is input into the fully connected layer to map high-dimensional features to low-dimensional space to generate the final low-resolution audio feature sequence. Specifically, the fully connected layer compresses the features to the target dimension (such as 64 dimensions) through the weight matrix and bias terms.

[0076] In an embodiment of the present invention, through pooling and full-connection dimensionality reduction, the spectral features are compressed to a low-resolution sequence (such as from 512 dimensions to 64 dimensions), reducing the subsequent calculation amount. At the same time, the compressed feature sequence accelerates the subsequent model processing speed, enhances the robustness of the features to environmental changes, and improves the generalization ability of different audio types (such as speech, music, and ambient sound).

[0077] S3. Perform diffusion distillation processing on the low-resolution audio feature sequence to obtain an optimized low-dimensional audio feature sequence.

[0078] In an embodiment of the present invention, in the diffusion distillation process, a target diffusion model is obtained based on training of a historical low-resolution audio feature sequence, and the low-resolution audio feature sequence is denoised according to the model to generate a noisy audio feature sequence. Subsequently, knowledge distillation is performed, and the target diffusion model is used as a teacher model. A lightweight student model is obtained through distillation technology, and then the student model is used to extract features from the noisy audio feature sequence to obtain a low-dimensional audio feature sequence. This process realizes knowledge transfer from a complex model to a lightweight model, and optimizes feature extraction efficiency and accuracy.

[0079] like Figure 3 As shown, in the embodiment of the present invention, the diffusion distillation process is performed on the low-resolution audio feature sequence to obtain an optimized low-dimensional audio feature sequence, including:

[0080] Acquiring a historical low-resolution audio feature sequence, and performing model training on a preset initial diffusion model based on the historical low-resolution audio feature sequence to obtain a target diffusion model;

[0081] Performing Gaussian noise addition on the low-resolution audio feature sequence according to the target diffusion model to obtain a corresponding noise audio feature sequence;

[0082] Performing knowledge distillation processing on the noise audio feature sequence according to the target diffusion model to obtain a low-dimensional audio feature sequence;

[0083] Function optimization is performed on the low-dimensional audio feature sequence to obtain an optimized low-dimensional audio feature sequence.

[0084] The present invention can collect and organize historical low-resolution audio feature sequences. These feature sequences serve as training data, reflecting the representation of different audio samples in low-dimensional space. An initial diffusion model is preset, which is usually a deep neural network with certain audio feature learning capabilities, but has not yet been optimized for the current task. The initial diffusion model is trained using historical low-resolution audio feature sequences. During the training process, the model learns how to extract effective information from the input features and gradually adjusts the network parameters to minimize the prediction error. After the training is completed, the target diffusion model is obtained, which can more accurately capture the essential structure of the audio features.

[0085] In detail, in order to enhance the robustness and generalization ability of the model, controllable noise interference is introduced into the low-resolution audio feature sequence, and the target diffusion model is used to perform Gaussian noise on the low-resolution audio feature sequence. Specifically, a random noise term that conforms to the Gaussian distribution is added to each eigenvalue to generate the corresponding noisy audio feature sequence. The noisy feature sequence retains the main structure of the original feature, but adds random perturbations, which helps the model learn more stable feature representations.

[0086] Among them, the trained target diffusion model is used as the teacher model. This model has powerful feature extraction capabilities, but may have high computational complexity. In order to reduce computational overhead, a lightweight student model is extracted from the teacher model through knowledge distillation technology.

[0087] In the knowledge distillation stage, the teacher model processes the noisy audio feature sequence to generate intermediate feature representations or prediction results. The student model attempts to imitate the behavior of the teacher model and learns feature extraction capabilities by minimizing the difference between the output and the teacher model. Ultimately, the student model can achieve performance similar to that of the teacher model at a lower computational cost. The trained student model is used to extract features from the noisy audio feature sequence to obtain low-dimensional audio feature sequences. These feature sequences retain the key information of the original audio and have a lower dimension, which is convenient for subsequent processing.

[0088] In detail, the low-dimensional audio feature sequence is further optimized to improve the audio quality. Specific function optimization methods such as nonlinear transformation and regularization are used to process the low-dimensional audio feature sequence, which can adjust the distribution of features, enhance the discriminability of features or reduce redundant information. The optimized low-dimensional audio feature sequence has better representation ability and interpretability.

[0089] In an embodiment of the present invention, performing knowledge distillation on the noise audio feature sequence according to the target diffusion model to obtain a low-dimensional audio feature sequence includes:

[0090] The target diffusion model is used as a teacher model, and knowledge distillation is performed on the teacher model to obtain a corresponding student model;

[0091] The noise audio feature sequence is distilled and extracted according to the student model to obtain a low-dimensional audio feature sequence.

[0092] In an embodiment of the present invention, a trained target diffusion model is used as a teacher model. The teacher model is a complex and powerful neural network. By learning a large number of historical low-resolution audio feature sequences, it has mastered the effective extraction and representation methods of audio features. However, due to its complex structure and large number of parameters, the teacher model may face problems such as high computing resource consumption and poor real-time performance during inference.

[0093] The purpose of knowledge distillation is to extract knowledge from the teacher model and pass it to a lighter-weight student model. The student model will have similar feature extraction capabilities as the teacher model, but with a simpler structure and fewer parameters, making it more suitable for deployment in resource-constrained environments.

[0094] Specifically, the teacher model processes the input noisy audio feature sequence to generate intermediate feature representations or prediction results. The student model attempts to use these intermediate representations or prediction results as supervision signals and adjusts its own network parameters to make its output as close as possible to the output of the teacher model. During the knowledge distillation process, the intermediate layer output of the teacher model, the final prediction result, or a combination of the two can be selected as supervision signals.

[0095] Among them, the output of the intermediate layer can provide richer feature representation information, which helps the student model learn more detailed feature extraction capabilities; and the final prediction result directly reflects the model's completion of the task, which helps the student model to approach the performance of the teacher model as a whole. The noisy audio feature sequence is input into the student model, which processes it and outputs the corresponding low-dimensional audio feature sequence.

[0096] The low-dimensional audio feature sequence not only retains the key information of the original audio (such as pitch, rhythm, timbre, etc.), but also has a lower dimension, which is convenient for audio classification, recognition, generation and other tasks. At the same time, due to the simple structure and few parameters of the student model, the use of low-dimensional feature sequences can also reduce computing resource consumption and improve real-time performance.

[0097] In an embodiment of the present invention, random perturbations in audio distribution are simulated through Gaussian noise addition in diffusion distillation. The student model learns the stable feature extraction capability of the teacher model in a noisy environment through distillation, thereby enhancing the robustness to actual noise, greatly improving the accuracy of audio reconstruction, ensuring high fidelity, and improving audio quality.

[0098] S4. Perform deconvolution decoding on the optimized low-dimensional audio feature sequence to obtain a target high-dimensional audio feature sequence.

[0099] In an embodiment of the present invention, a low-dimensional audio feature sequence is upsampled and optimized through deconvolution kernels to obtain a sampled audio feature sequence, and then its dimension is expanded to a high-resolution scale to form an initial high-dimensional feature. Subsequently, rhythm fusion is performed using the parameters of the rhythm control network to obtain rhythmic high-dimensional features. Finally, the target high-dimensional audio feature sequence is decoded to achieve reconstruction of feature dimensions and rhythmic information.

[0100] In an embodiment of the present invention, performing deconvolution decoding on the optimized low-dimensional audio feature sequence to obtain a target high-dimensional audio feature sequence includes:

[0101] Obtaining deconvolution kernels of a plurality of preset deconvolution layers, and performing upsampling processing on the optimized low-dimensional audio feature sequence according to the deconvolution kernels to obtain a sampled audio feature sequence;

[0102] Expanding the feature dimension of the sampled audio feature sequence to a preset high-resolution scale to obtain an initial high-dimensional audio feature sequence;

[0103] Obtaining rhythmic parameters of a preset rhythmic control network, and performing rhythmic fusion on the initial high-dimensional audio feature sequence according to the rhythmic parameters to obtain a rhythmic high-dimensional audio feature sequence;

[0104] The prosodic high-dimensional audio feature sequence is decoded to obtain a target high-dimensional audio feature sequence.

[0105] In an embodiment of the present invention, multiple preset deconvolution layers and their corresponding deconvolution kernels are obtained. The deconvolution kernel is a parameter set for performing an upsampling operation, which determines how to map low-dimensional features back to a higher-dimensional space. The optimized low-dimensional audio feature sequence is upsampled according to the deconvolution kernel, that is, the deconvolution kernel is used to slide on the feature sequence and generate new feature values, thereby expanding the size of the feature sequence.

[0106] In detail, the feature dimension of the sampled audio feature sequence is expanded to a preset high-resolution scale through an interpolation method to ensure that the feature sequence can be filled to the required dimension. After the dimension expansion, the initial high-dimensional audio feature sequence is obtained.

[0107] The present invention obtains rhythmic parameters of a preset rhythmic control network, which is a specially preset sub-network that predicts rhythmic parameters such as pitch and duration. The rhythmic parameters are key parameters for controlling rhythmic information in audio features, such as pitch, rhythm, and intonation. The rhythmic parameters are used to perform rhythmic fusion on an initial high-dimensional audio feature sequence. The rhythmic parameters are applied to the initial high-dimensional audio feature sequence. The rhythmic information may be integrated into the feature sequence through weighting, modulation or other forms of transformation to make it more consistent with the rhythmic characteristics of natural audio. The rhythmic high-dimensional audio feature sequence is decoded to convert the feature sequence back to the original audio signal or some audio representation that is easier to process. This process may involve inverse transformation, reconstruction or other forms of signal processing.

[0108] In an embodiment of the present invention, the deconvolution operation gradually restores the dimension of the feature sequence through upsampling, supplements the detailed information lost in the low-dimensional features, such as high-frequency components and transient changes, improves the audio resolution, and also reduces the amount of calculation and improves the real-time performance of audio reconstruction; by integrating the rhythm parameters through the rhythm control network, the decoded feature sequence can better simulate the timing characteristics of natural audio and reduce the mechanical feel.

[0109] S5. Generate target super-resolution audio of the audio to be processed according to the target high-dimensional audio feature sequence.

[0110] In an embodiment of the present invention, time domain and frequency domain features are extracted from a high-dimensional audio feature sequence to obtain a corresponding feature sequence, which is then fused across dimensions to form a dimensional fusion feature sequence. This sequence is used to generate an initial super-resolution audio waveform, and finally, the initial audio is activated and normalized to obtain the target super-resolution audio, thereby achieving audio resolution improvement and quality optimization.

[0111] In the embodiment of the present invention, generating the target super-resolution audio of the audio to be processed according to the target high-dimensional audio feature sequence includes:

[0112] Performing multi-dimensional feature extraction on the target high-dimensional audio feature sequence to obtain a time domain feature sequence and a frequency domain feature sequence;

[0113] Performing cross-dimensional feature fusion on the time domain feature sequence and the frequency domain feature sequence to obtain a dimensional fusion feature sequence;

[0114] Generating an audio waveform for the audio to be processed according to the dimensional fusion feature sequence to obtain an initial super-resolution audio;

[0115] The initial super-resolution audio is activated and normalized to obtain a target super-resolution audio.

[0116] In an embodiment of the present invention, the time domain feature extraction refers to extracting time domain features from a high-dimensional audio feature sequence, that is, the changing characteristics of the audio signal on the time axis, using a sliding window or time convolution operation to analyze the amplitude changes of the audio signal at different time points, thereby extracting time domain features such as short-time energy and zero-crossing rate. These features reflect the instantaneous characteristics of the audio signal.

[0117] The frequency domain feature extraction refers to extracting frequency domain features from a high-dimensional audio feature sequence, that is, the distribution characteristics of the audio signal at different frequency components. Fast Fourier transform (FFT) or other frequency domain analysis techniques can be applied to convert the time domain signal into a frequency domain representation to extract frequency domain features, such as spectral energy, harmonic components, etc. These features reflect the frequency distribution of the audio signal.

[0118] In detail, the time domain feature sequence and the frequency domain feature sequence are fused to combine their information and ensure that the time domain feature sequence and the frequency domain feature sequence are aligned in the time or frequency dimension. Weighted summation, splicing or other fusion techniques can be used to combine the time domain and frequency domain features. For example, the time domain features and the frequency domain features can be spliced in the channel dimension to form a feature sequence with a higher dimension.

[0119] Specifically, the dimensional fusion feature sequence is used to generate an audio waveform to restore or enhance the resolution of the audio. The dimensional fusion feature sequence is converted into an audio waveform using a generative adversarial network (GAN) and an autoregressive model. Audio sample points are gradually generated according to the information of the feature sequence. Through an iterative generation process, a complete audio waveform is gradually constructed to obtain the initial super-resolution audio.

[0120] The present invention activates and normalizes the initial super-resolution audio to optimize the quality and dynamic range of the audio, uses an activation function (such as ReLU, Sigmoid) to perform nonlinear transformation on the audio waveform to enhance the dynamic characteristics of the audio. For example, ReLU can remove negative values to make the audio waveform clearer, and perform amplitude normalization on the audio waveform to ensure that the amplitude of the audio signal is within a reasonable range to avoid distortion. For example, maximum and minimum value normalization or mean variance normalization is used to adjust the dynamic range of the audio, thereby obtaining the target super-resolution audio and optimizing the audio quality.

[0121] In an embodiment of the present invention, by extracting and fusing multi-dimensional information such as time domain and frequency domain from a high-dimensional audio feature sequence, the generated audio waveform can more accurately restore the details of the original audio, including high-frequency components, transient changes, etc., thereby significantly improving the resolution and clarity of the audio.

[0122] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0123] like Figure 4 , which is a functional module diagram of an audio super-resolution generation device provided by one embodiment of the present invention.

[0124] In an embodiment of the present disclosure, an audio super-resolution generation device is provided, and the audio super-resolution generation device corresponds to the audio super-resolution generation method in the above embodiment. Figure 5 As shown, the audio super-resolution generation device 100 can be installed in an electronic device. According to the functions to be implemented, the audio super-resolution generation device 100 includes an audio framing module 101, a feature encoding module 102, a diffusion distillation module 103, a deconvolution decoding module 104, and a target audio generation module 105. The functional modules are described in detail as follows:

[0125] An audio framing module is used to obtain original audio, perform audio enhancement processing on the original audio to obtain audio to be processed, and perform audio framing on the audio to be processed to obtain a spectral feature sequence;

[0126] A feature encoding module, configured to perform audio feature encoding on the spectral feature sequence to obtain a low-resolution audio feature sequence;

[0127] a diffusion distillation module, configured to perform diffusion distillation processing on the low-resolution audio feature sequence to obtain an optimized low-dimensional audio feature sequence;

[0128] a deconvolution decoding module, configured to perform deconvolution decoding on the optimized low-dimensional audio feature sequence to obtain a target high-dimensional audio feature sequence;

[0129] A target audio generation module is used to generate target super-resolution audio of the audio to be processed according to the target high-dimensional audio feature sequence.

[0130] In one embodiment, when performing audio enhancement processing on the original audio to obtain the audio to be processed, the audio framing module 101 is configured to:

[0131] Performing sampling rate unification processing on the original audio to obtain unified audio;

[0132] De-noising the unified audio to obtain interference-free audio;

[0133] Normalization is performed on the interference-free audio to obtain audio to be processed.

[0134] In one embodiment, when performing audio framing on the audio to be processed to obtain a spectrum feature sequence, the audio framing module 101 is configured to:

[0135] Performing short-time Fourier transform on the audio to be processed to obtain an audio frame sequence of fixed length;

[0136] Performing point-by-point multiplication processing on each audio frame in the audio frame sequence and a preset window function to obtain a windowed audio frame sequence;

[0137] The windowed audio frame sequence is subjected to time-frequency transformation processing to obtain a corresponding frequency spectrum feature sequence.

[0138] In one embodiment, when performing audio feature encoding on the spectral feature sequence to obtain a low-resolution audio feature sequence, the feature encoding module 102 is configured to:

[0139] Performing multi-layer convolution on the spectral feature sequence to obtain an audio convolution enhanced feature sequence;

[0140] Performing pooling dimensionality reduction processing on the audio convolution enhanced feature sequence to obtain an audio compression feature sequence;

[0141] Performing activation function processing on the audio compression feature sequence to obtain an audio activation feature sequence;

[0142] Performing multi-layer feature fusion on the audio activation feature sequence to obtain an audio fusion feature sequence;

[0143] Performing self-attention calculation on the audio fusion feature sequence to obtain an attention-weighted feature sequence;

[0144] Performing a fully connected dimensionality reduction process on the attention-weighted feature sequence to obtain a low-resolution audio feature sequence of the audio to be processed.

[0145] In one embodiment, when performing the diffusion distillation process on the low-resolution audio feature sequence to obtain the optimized low-dimensional audio feature sequence, the diffusion distillation module 103 is configured to:

[0146] Acquiring a historical low-resolution audio feature sequence, and performing model training on a preset initial diffusion model based on the historical low-resolution audio feature sequence to obtain a target diffusion model;

[0147] Performing Gaussian noise addition on the low-resolution audio feature sequence according to the target diffusion model to obtain a corresponding noise audio feature sequence;

[0148] Performing knowledge distillation processing on the noise audio feature sequence according to the target diffusion model to obtain a low-dimensional audio feature sequence;

[0149] Function optimization is performed on the low-dimensional audio feature sequence to obtain an optimized low-dimensional audio feature sequence.

[0150] In one embodiment, when performing knowledge distillation processing on the noise audio feature sequence according to the target diffusion model to obtain a low-dimensional audio feature sequence, the diffusion distillation module 103 is configured to:

[0151] The target diffusion model is used as a teacher model, and knowledge distillation is performed on the teacher model to obtain a corresponding student model;

[0152] The noise audio feature sequence is distilled and extracted according to the student model to obtain a low-dimensional audio feature sequence.

[0153] In one embodiment, when performing deconvolution decoding on the optimized low-dimensional audio feature sequence to obtain a target high-dimensional audio feature sequence, the deconvolution decoding module 104 is configured to:

[0154] Obtaining deconvolution kernels of a plurality of preset deconvolution layers, and performing upsampling processing on the optimized low-dimensional audio feature sequence according to the deconvolution kernels to obtain a sampled audio feature sequence;

[0155] Expanding the feature dimension of the sampled audio feature sequence to a preset high-resolution scale to obtain an initial high-dimensional audio feature sequence;

[0156] Obtaining rhythmic parameters of a preset rhythmic control network, and performing rhythmic fusion on the initial high-dimensional audio feature sequence according to the rhythmic parameters to obtain a rhythmic high-dimensional audio feature sequence;

[0157] The prosodic high-dimensional audio feature sequence is decoded to obtain a target high-dimensional audio feature sequence.

[0158] In one embodiment, when the target audio generation module 104 generates the target super-resolution audio of the to-be-processed audio according to the target high-dimensional audio feature sequence, it is configured to:

[0159] Performing multi-dimensional feature extraction on the target high-dimensional audio feature sequence to obtain a time domain feature sequence and a frequency domain feature sequence;

[0160] Performing cross-dimensional feature fusion on the time domain feature sequence and the frequency domain feature sequence to obtain a dimensional fusion feature sequence;

[0161] Generating an audio waveform for the audio to be processed according to the dimensional fusion feature sequence to obtain an initial super-resolution audio;

[0162] The initial super-resolution audio is activated and normalized to obtain a target super-resolution audio.

[0163] In the present invention, the specific definition of an audio super-resolution generation device can be found in the definition of an audio super-resolution generation method described above and will not be repeated here. Each module in the above-mentioned audio super-resolution generation device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0164] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of an audio super-resolution generation method.

[0165] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the client side of an audio super-resolution generation method.

[0166] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0167] Acquire original audio, perform audio enhancement processing on the original audio to obtain audio to be processed, and perform audio framing on the audio to be processed to obtain a frequency spectrum feature sequence;

[0168] Performing audio feature encoding on the spectral feature sequence to obtain a low-resolution audio feature sequence;

[0169] Performing diffusion distillation processing on the low-resolution audio feature sequence to obtain an optimized low-dimensional audio feature sequence;

[0170] Performing deconvolution decoding on the optimized low-dimensional audio feature sequence to obtain a target high-dimensional audio feature sequence;

[0171] Generate target super-resolution audio of the audio to be processed according to the target high-dimensional audio feature sequence.

[0172] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and apparatuses can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and actual implementation may employ other division methods.

[0173] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0174] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0175] In some implementations of this embodiment, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the method described in the above embodiment are implemented.

[0176] The readable storage medium of the present invention stores a computer program, which, when executed by a processor of an electronic device, can implement:

[0177] Acquire original audio, perform audio enhancement processing on the original audio to obtain audio to be processed, and perform audio framing on the audio to be processed to obtain a frequency spectrum feature sequence;

[0178] Performing audio feature encoding on the spectral feature sequence to obtain a low-resolution audio feature sequence;

[0179] Performing diffusion distillation processing on the low-resolution audio feature sequence to obtain an optimized low-dimensional audio feature sequence;

[0180] Performing deconvolution decoding on the optimized low-dimensional audio feature sequence to obtain a target high-dimensional audio feature sequence;

[0181] Generate target super-resolution audio of the audio to be processed according to the target high-dimensional audio feature sequence.

[0182] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0183] The computer-readable storage medium may also store at least one computer-executable program / instruction, such as a computer-readable instruction. Computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Computer-readable storage media may include, for example, read-only memory (ROM), a hard disk, a flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above may be performed.

[0184] In addition, the computer device may also include (but is not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (eg, keyboard, mouse, speaker, etc.).

[0185] In one embodiment, the at least one computer executable instruction may also be compiled into or constitute a software product / computer program product, wherein one or more computer executable instructions are executed by a processor to perform the various functions and / or method steps in the embodiments described in the present technology.

[0186] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-described methods. In particular, any reference to memory, storage, database, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0187] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0188] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods may also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, may be implemented with a dedicated hardware-based system that performs the specified function or action, or may be implemented with a combination of dedicated hardware and computer instructions.

[0189] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

[0190] It should be noted that if software tools or components other than those of our company appear in the embodiments of this application, they are only used for illustration and do not represent actual use.

Claims

1. A method for generating audio super-resolution, characterized in that: The method comprises: Acquire original audio, perform audio enhancement processing on the original audio to obtain audio to be processed, and perform audio framing on the audio to be processed to obtain a spectral feature sequence; Performing audio feature encoding on the spectral feature sequence to obtain a low-resolution audio feature sequence; Performing diffusion distillation processing on the low-resolution audio feature sequence to obtain an optimized low-dimensional audio feature sequence; Performing deconvolution decoding on the optimized low-dimensional audio feature sequence to obtain a target high-dimensional audio feature sequence; A target super-resolution audio of the audio to be processed is generated according to the target high-dimensional audio feature sequence.

2. The audio super-resolution generation method according to claim 1, wherein: The performing audio enhancement processing on the original audio to obtain the audio to be processed includes: Performing sampling rate unification processing on the original audio to obtain unified audio; De-noising the unified audio to obtain interference-free audio; Normalization is performed on the interference-free audio to obtain audio to be processed.

3. The audio super-resolution generation method according to claim 1, wherein: The step of dividing the audio to be processed into frames to obtain a frequency spectrum feature sequence includes: Performing short-time Fourier transform on the audio to be processed to obtain an audio frame sequence of fixed length; Performing point-by-point multiplication processing on each audio frame in the audio frame sequence and a preset window function to obtain a windowed audio frame sequence; The windowed audio frame sequence is subjected to time-frequency transformation processing to obtain a corresponding frequency spectrum feature sequence.

4. The audio super-resolution generation method according to claim 1, wherein: The step of performing audio feature encoding on the spectral feature sequence to obtain a low-resolution audio feature sequence includes: Performing multi-layer convolution on the spectral feature sequence to obtain an audio convolution enhanced feature sequence; Performing pooling dimensionality reduction processing on the audio convolution enhanced feature sequence to obtain an audio compression feature sequence; Performing activation function processing on the audio compression feature sequence to obtain an audio activation feature sequence; Performing multi-layer feature fusion on the audio activation feature sequence to obtain an audio fusion feature sequence; Performing self-attention calculation on the audio fusion feature sequence to obtain an attention-weighted feature sequence; Performing a fully connected dimensionality reduction process on the attention-weighted feature sequence to obtain a low-resolution audio feature sequence of the audio to be processed.

5. The audio super-resolution generation method according to claim 1, wherein: The step of performing diffusion distillation processing on the low-resolution audio feature sequence to obtain an optimized low-dimensional audio feature sequence includes: Acquiring a historical low-resolution audio feature sequence, and performing model training on a preset initial diffusion model based on the historical low-resolution audio feature sequence to obtain a target diffusion model; Performing Gaussian noise addition on the low-resolution audio feature sequence according to the target diffusion model to obtain a corresponding noise audio feature sequence; Performing knowledge distillation processing on the noise audio feature sequence according to the target diffusion model to obtain a low-dimensional audio feature sequence; Function optimization is performed on the low-dimensional audio feature sequence to obtain an optimized low-dimensional audio feature sequence.

6. The audio super-resolution generation method according to claim 1, wherein: The performing deconvolution decoding on the optimized low-dimensional audio feature sequence to obtain a target high-dimensional audio feature sequence includes: Obtaining deconvolution kernels of a plurality of preset deconvolution layers, and performing upsampling processing on the optimized low-dimensional audio feature sequence according to the deconvolution kernels to obtain a sampled audio feature sequence; Expanding the feature dimension of the sampled audio feature sequence to a preset high-resolution scale to obtain an initial high-dimensional audio feature sequence; Obtaining rhythmic parameters of a preset rhythmic control network, and performing rhythmic fusion on the initial high-dimensional audio feature sequence according to the rhythmic parameters to obtain a rhythmic high-dimensional audio feature sequence; The prosodic high-dimensional audio feature sequence is decoded to obtain a target high-dimensional audio feature sequence.

7. The audio super-resolution generation method according to claim 1, wherein: Generating the target super-resolution audio of the audio to be processed according to the target high-dimensional audio feature sequence includes: Performing multi-dimensional feature extraction on the target high-dimensional audio feature sequence to obtain a time domain feature sequence and a frequency domain feature sequence; Performing cross-dimensional feature fusion on the time domain feature sequence and the frequency domain feature sequence to obtain a dimensional fusion feature sequence; Generating an audio waveform for the audio to be processed according to the dimensional fusion feature sequence to obtain an initial super-resolution audio; The initial super-resolution audio is activated and normalized to obtain a target super-resolution audio.

8. An audio super-resolution generation device, characterized in that: The device comprises: An audio framing module is used to obtain original audio, perform audio enhancement processing on the original audio to obtain audio to be processed, and perform audio framing on the audio to be processed to obtain a spectral feature sequence; A feature encoding module, configured to perform audio feature encoding on the spectral feature sequence to obtain a low-resolution audio feature sequence; a diffusion distillation module, configured to perform diffusion distillation processing on the low-resolution audio feature sequence to obtain an optimized low-dimensional audio feature sequence; a deconvolution decoding module, configured to perform deconvolution decoding on the optimized low-dimensional audio feature sequence to obtain a target high-dimensional audio feature sequence; A target audio generation module is used to generate target super-resolution audio of the audio to be processed according to the target high-dimensional audio feature sequence.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the audio super-resolution generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the audio super-resolution generation method according to any one of claims 1 to 7 is implemented.

Images