Audio restoration method and device, medium and computing equipment

By obtaining audio feature vectors and description information, generating and denoising noise vectors, the problem of low audio quality is solved, efficient audio repair and consistency of tone semantics is achieved, and it is suitable for a variety of audio data, improving user experience.

CN118942468APending Publication Date: 2024-11-12HANGZHOU NETEASE CLOUD MUSIC TECH CO LTD
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202410994656.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing technology is difficult to effectively repair the problem of low or damaged audio quality, resulting in degradation of audio playback performance and poor user experience. The existing method has limited use scenarios and fewer types of audio repaired.

Method used

By obtaining the feature vector and audio description information of the audio to be repaired, a noise vector is generated based on time steps, and denoising is combined with the adapter module and the audio generation module to perform denoising processing, the target feature vector is obtained to achieve audio repair, ensuring the repaired audio and source audio Consistent in timbre and semantics.

Benefits of technology

It realizes efficient restoration of low-quality audio, is suitable for any kind of audio data, improves the quality of audio repair, expands the usage scenario, and ensures that the sound quality after the audio is repaired is consistent with the source audio, and has a better user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942468A_ABST
    Figure CN118942468A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an audio restoration method and device, a medium and computing equipment. The method comprises the following steps: acquiring a first feature vector of a to-be-restored first audio and audio description information of the first audio; based on set time steps, obtaining audio noise corresponding to each step in the time steps, and generating a noise vector according to each audio noise; denoising the noise vector according to the first feature vector, the time step and the audio description information to obtain a target feature vector; and obtaining a repaired second audio according to the target feature vector. Through the method, the problem that the sound quality of the audio is damaged can be effectively solved, and better experience is brought to the user.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of audio technology, and more specifically, embodiments of the present disclosure relate to an audio repair method, apparatus, medium, and computing device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the present disclosure. No description herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] Currently, there is a huge amount of audio data (such as music and speeches) on the Internet, among which there is a lot of audio data with rich content and high value but low audio quality or damaged. These audio data may have problems such as distortion and noise during recording due to age limitations, low performance of the sound source and recording equipment, or due to repeated lossy encoding and decoding during the transmission process, resulting in a decrease in bit rate and effective frequency, which will affect the audio playback performance and reduce the user experience. Summary of the invention

[0004] The present disclosure provides an audio repair method, apparatus, medium and computing device to solve the problem of damaged sound quality.

[0005] In a first aspect of an embodiment of the present disclosure, an audio repair method is provided, comprising: obtaining a first feature vector of a first audio to be repaired and audio description information of the first audio; based on a set time step, obtaining the audio noise corresponding to each step in the time step, and generating a noise vector according to each of the audio noises; denoising the noise vector according to the first feature vector, the time step and the audio description information to obtain a target feature vector; and obtaining a repaired second audio according to the target feature vector.

[0006] In one embodiment of the present disclosure, denoising the noise vector according to the first eigenvector, the time step and the audio description information to obtain a target eigenvector includes: determining a second eigenvector corresponding to the time step and a third eigenvector corresponding to the audio description information; denoising the noise vector a target number of times according to the first eigenvector, the second eigenvector and the third eigenvector to obtain a target eigenvector, wherein the target number of times is the same as the amount of audio noise.

[0007] In another embodiment of the present disclosure, the noise vector is denoised a target number of times according to the first eigenvector, the second eigenvector and the third eigenvector, including: obtaining the first noise vector after the last denoising, and determining an intermediate vector according to the first eigenvector and the first noise vector; inputting the intermediate vector, the second eigenvector and the third eigenvector into an adapter module; inputting the output parameters of the adapter module, the first noise vector, the second eigenvector and the third eigenvector into an audio generation module to obtain the second noise vector output by the audio generation module, and updating the current denoising times; in response to the denoising times being less than the target times, updating the second noise vector to the first noise vector, and returning to execute the step of determining the intermediate vector according to the first eigenvector and the second eigenvector.

[0008] In another embodiment of the present disclosure, determining the intermediate vector based on the first eigenvector and the first noise vector includes: performing a convolution operation with an initial zero setting on the first eigenvector to obtain a fourth eigenvector; and adding the fourth eigenvector and the first noise vector to obtain the intermediate vector.

[0009] In another embodiment of the present disclosure, the adapter module includes a plurality of control units connected in sequence and a convolutional layer initially set to zero corresponding to each of the control units; the audio generation module includes a plurality of encoder units connected in sequence and a decoder unit corresponding to each of the encoder units, and the decoder units are connected in sequence; wherein the output parameters of a control unit are input to the next control unit, and the output parameters of the control unit are input to the convolutional layer corresponding to the control unit; the output parameters of the convolutional layer are input to the decoder unit corresponding to the convolutional layer; the output parameters of an encoder unit are input to the next encoder unit, and the output parameters of the encoder unit are input to the decoder unit corresponding to the encoder unit; the output parameters of a decoder unit are input to the next decoder unit.

[0010] In another embodiment of the present disclosure, the step of obtaining a first feature vector of a first audio to be repaired and audio description information of the first audio includes: controlling an encoder in a set model to encode valid content in the first audio into a latent vector in a latent space, wherein the valid content is used to indicate audio content other than noise in the first audio; controlling a decoder in the set model to decode the latent vector into an intermediate audio in a latent space; determining the latent vector as the first feature vector, and extracting the audio description information from the intermediate audio; and obtaining a repaired second audio according to the target feature vector includes: controlling the decoder to decode the target feature vector into the second audio in a latent space.

[0011] In another embodiment of the present disclosure, the training steps of the set model are as follows: obtain multiple training samples, the training samples include high-quality audio and low-quality audio, the high-quality audio is used to indicate audio with sound quality higher than or equal to a preset sound quality, and the low-quality audio is used to indicate audio with sound quality lower than a preset sound quality; according to each of the training samples, train the preset model to obtain the set model.

[0012] In another embodiment of the present disclosure, the preset model is trained according to each of the training samples, including: based on the high-quality audio in each of the training samples, the preset model is trained to obtain an intermediate model, and the loss function of the preset model is updated based on the reconstruction loss function, the adversarial loss function, the adversarial feature loss function, the divergence loss function, the high-quality audio and the low-quality audio; according to each of the training samples, the intermediate model is trained to obtain the set model.

[0013] In another embodiment of the present disclosure, the obtaining of multiple training samples includes: obtaining multiple high-quality audios; performing sound quality degradation processing on each of the high-quality audios to obtain multiple low-quality audios; and constructing training samples based on the high-quality audios and the corresponding low-quality audios.

[0014] In yet another embodiment of the present disclosure, the sound quality degradation processing is performed on each of the high-quality audios, including: inputting the high-quality audio into a degradation processor to obtain low-quality audio output by the degradation processor.

[0015] In another embodiment of the present disclosure, the degradation processor includes at least one of the following functions: adding noise to audio, cutting audio, spectral energy mask, adding reverberation, increasing or decreasing volume, modifying signal amplitude, simulating distortion of audio, encoding audio with a lossy encoder, sampling audio with a sampling format with lossy sound quality, and encoding audio with encoding parameters with lossy sound quality.

[0016] In another embodiment of the present disclosure, the obtaining of multiple high-quality audios includes: obtaining audio to be processed with sound quality higher than or equal to preset sound quality from the network, and the sampling rate of the audio to be processed is greater than the preset sampling rate; and resampling each of the audio to be processed into audio with a set sampling rate as high-quality audio.

[0017] In yet another embodiment of the present disclosure, the audio description information includes rhythm parameters, instrument parameters, type, and audio description text of the first audio.

[0018] In yet another embodiment of the present disclosure, the acquiring of the audio description information of the first audio includes: extracting the audio description information from the first audio using a multimodal audio text model.

[0019] According to a second aspect of the disclosed embodiments, an audio repair device is provided, comprising: a first acquisition module, used to acquire a first feature vector of a first audio to be repaired and audio description information of the first audio; a second acquisition module, used to acquire the audio noise corresponding to each step in the time step based on a set time step, and generate a noise vector according to each audio noise; a denoising module, used to denoise the noise vector according to the first feature vector and the audio description information to obtain a target feature vector; and a third acquisition module, used to acquire the repaired second audio according to the target feature vector.

[0020] In one embodiment of the present disclosure, the denoising module includes: a first determination unit, used to determine the second eigenvector corresponding to the time step and the third eigenvector corresponding to the audio description information; a target frequency denoising unit, used to denoise the noise vector a target number of times based on the first eigenvector, the second eigenvector and the third eigenvector to obtain a target eigenvector, wherein the target number is the same as the amount of audio noise.

[0021] In another embodiment of the present disclosure, the target number denoising unit is specifically used to: obtain the first noise vector after the last denoising, and determine the intermediate vector based on the first eigenvector and the first noise vector; input the intermediate vector, the second eigenvector and the third eigenvector into the adapter module; input the output parameters of the adapter module, the first noise vector, the second eigenvector and the third eigenvector into the audio generation module to obtain the second noise vector output by the audio generation module, and update the current denoising number; in response to the denoising number being less than the target number, update the second noise vector to the first noise vector, and return to execute the step of determining the intermediate vector based on the first eigenvector and the second eigenvector.

[0022] In another embodiment of the present disclosure, determining the intermediate vector based on the first eigenvector and the first noise vector includes: performing a convolution operation with an initial zero setting on the first eigenvector to obtain a fourth eigenvector; and adding the fourth eigenvector and the first noise vector to obtain the intermediate vector.

[0023] In another embodiment of the present disclosure, the adapter module includes a plurality of control units connected in sequence and a convolutional layer initially set to zero corresponding to each of the control units; the audio generation module includes a plurality of encoder units connected in sequence and a decoder unit corresponding to each of the encoder units, and the decoder units are connected in sequence; wherein the output parameters of a control unit are input to the next control unit, and the output parameters of the control unit are input to the convolutional layer corresponding to the control unit; the output parameters of the convolutional layer are input to the decoder unit corresponding to the convolutional layer; the output parameters of an encoder unit are input to the next encoder unit, and the output parameters of the encoder unit are input to the decoder unit corresponding to the encoder unit; the output parameters of a decoder unit are input to the next decoder unit.

[0024] In another embodiment of the present disclosure, the first acquisition module includes: a first control unit, used to control the encoder in the set model, to encode the valid content in the first audio into a latent vector in the latent space, and the valid content is used to indicate the audio content other than noise in the first audio; a second control unit, used to control the decoder in the set model, to decode the latent vector into an intermediate audio in the latent space; an extraction unit, used to determine the latent vector as the first feature vector, and extract the audio description information from the intermediate audio; the third acquisition module includes: a third control unit, used to control the decoder, to decode the target feature vector into the second audio in the latent space.

[0025] In another embodiment of the present disclosure, the training steps of the set model are as follows: obtain multiple training samples, the training samples include high-quality audio and low-quality audio, the high-quality audio is used to indicate audio with sound quality higher than or equal to a preset sound quality, and the low-quality audio is used to indicate audio with sound quality lower than a preset sound quality; according to each of the training samples, train the preset model to obtain the set model.

[0026] In another embodiment of the present disclosure, the preset model is trained according to each of the training samples, including: based on the high-quality audio in each of the training samples, the preset model is trained to obtain an intermediate model, and the loss function of the preset model is updated based on the reconstruction loss function, the adversarial loss function, the adversarial feature loss function, the divergence loss function, the high-quality audio and the low-quality audio; according to each of the training samples, the intermediate model is trained to obtain the set model.

[0027] In another embodiment of the present disclosure, the obtaining of multiple training samples includes: obtaining multiple high-quality audios; performing sound quality degradation processing on each of the high-quality audios to obtain multiple low-quality audios; and constructing training samples based on the high-quality audios and the corresponding low-quality audios.

[0028] In yet another embodiment of the present disclosure, the sound quality degradation processing is performed on each of the high-quality audios, including: inputting the high-quality audio into a degradation processor to obtain low-quality audio output by the degradation processor.

[0029] In another embodiment of the present disclosure, the degradation processor includes at least one of the following functions: adding noise to audio, cutting audio, spectral energy mask, adding reverberation, increasing or decreasing volume, modifying signal amplitude, simulating distortion of audio, encoding audio with a lossy encoder, sampling audio with a sampling format with lossy sound quality, and encoding audio with encoding parameters with lossy sound quality.

[0030] In another embodiment of the present disclosure, the obtaining of multiple high-quality audios includes: obtaining audio to be processed with sound quality higher than or equal to preset sound quality from the network, and the sampling rate of the audio to be processed is greater than the preset sampling rate; and resampling each of the audio to be processed into audio with a set sampling rate as high-quality audio.

[0031] In yet another embodiment of the present disclosure, the audio description information includes rhythm parameters, instrument parameters, type, and audio description text of the first audio.

[0032] In yet another embodiment of the present disclosure, the first acquisition module is specifically configured to extract the audio description information from the first audio using a multimodal audio text model.

[0033] According to a third aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method provided in any one of the first aspects is implemented.

[0034] According to a fourth aspect of an embodiment of the present disclosure, a computing device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the computing device executes any one of the methods provided in the first aspect above.

[0035] The audio repair method, apparatus, medium and computing device of the disclosed embodiments obtain the first feature vector of the first audio to be repaired and the audio description information of the first audio, and based on the set time step, obtain the audio noise corresponding to each step in the time step, and generate a noise vector according to each audio noise, denoise the noise vector according to the first feature vector, the time step and the audio description information to obtain the target feature vector, and obtain the repaired second audio according to the target feature vector. In this process, the feature information of the source audio and the text description information of the source audio are combined to guide the audio noise to be denoised to obtain the repaired audio, which can achieve efficient restoration of low-quality audio without causing changes in the timbre or semantics of the audio, and can be applied to any type of audio data, with greater universality. In this way, problems such as impaired audio quality are effectively solved, bringing a better experience to users. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0037] Figure 1 The following schematically shows an application scenario of the audio repair method according to an embodiment of the present disclosure;

[0038] Figure 2 A schematic diagram of a flow chart of an audio repair method provided by an embodiment of the present disclosure is schematically shown;

[0039] Figure 3 The structure diagram of the multimodal audio text model in one embodiment of the present disclosure is schematically shown;

[0040] Figure 4 Schematically shows Figure 2 Schematic diagram of the process of step S203;

[0041] Figure 5a The structure diagram of the audio generation module in one embodiment of the present disclosure is schematically shown;

[0042] Figure 5b The schematic diagram of the structure of the adapter module in one embodiment of the present disclosure is shown;

[0043] Figure 6 One of the flowcharts of another audio repair method provided by an embodiment of the present disclosure is schematically shown;

[0044] Figure 7 The second flowchart of another audio repair method provided by an embodiment of the present disclosure is schematically shown;

[0045] Figure 8 The structure diagram of the degradation processor in one embodiment of the present disclosure is schematically shown;

[0046] Fig. 9 The structure diagram of the setting model in one embodiment of the present disclosure is schematically shown;

[0047] Fig.10 The following is a schematic diagram schematically showing a process flow of another audio repair method provided by an embodiment of the present disclosure;

[0048] Fig.11 The structure diagram of a computer-readable storage medium provided by an embodiment of the present disclosure is schematically shown;

[0049] Fig.12 The structure diagram of an audio repair device provided by an embodiment of the present disclosure is schematically shown;

[0050] Fig.13 A schematic diagram of the structure of a computing device provided by an embodiment of the present disclosure is shown schematically.

[0051] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0052] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0053] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0054] According to an embodiment of the present disclosure, an audio repair method, apparatus, medium and computing device are proposed.

[0055] In this article, it is important to understand that the terms used have the following meanings:

[0056] Diffusion model: A deep learning model architecture that learns the latent structure of a dataset by modeling how data points diffuse in latent space, achieving the goal of generating target data samples from noise.

[0057] Variational Autoencoder (VAE): It consists of two parts: encoder and decoder. The encoder compresses the input data into a latent vector, and the decoder generates new data from this latent vector.

[0058] Hidden vector: The VAE encoder compresses and reduces the dimension of a specific content into a vector, which facilitates calculation and reduces the amount of calculation. It can be restored to specific content such as images and audio through the VAE decoder.

[0059] Audio restoration: refers to upgrading low-quality audio to high-quality audio, including super-resolution high-frequency generation, distortion restoration, noise reduction, etc.

[0060] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0061] In addition, any number of elements in the drawings is for illustration and not limitation, and any naming is only for distinction and does not have any limiting meaning.

[0062] In addition, the data involved in the present disclosure may be data authorized by the user or fully authorized by all parties, and the collection, dissemination, and use of the data shall comply with the requirements of relevant laws and regulations. The implementation methods / examples of the present disclosure may be combined with each other.

[0063] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0065] The inventors have found that users have high demands for audio quality, so the audio quality can be improved by audio repair before audio playback. In particular, for some audios with rich content and high value but low or damaged audio quality, the audio playback performance can be greatly improved after audio repair.

[0066] Regarding audio restoration technology, audio enhancement technology is proposed in the related technology, which uses deep learning technology to perform time domain interpolation on audio to introduce high-frequency details and then reconstruct the lost high-frequency components. Specifically, by using the pre-trained generative adversarial network technology as the algorithm model, during inference, the audio to be enhanced is input into the generative adversarial network model to generate enhanced audio, and then the source audio and the enhanced audio are fused in high frequency and phase to obtain the final audio. The above scheme mainly repairs specific audio problems, such as audio high-frequency restoration, and is mainly based on convolutional neural network models or generative adversarial network models. Due to the limitations of the scheme, audio super-resolution (the process of converting low-resolution audio into high-resolution audio, that is, increasing the sampling rate, or reconstructing a part of the disappeared spectrum) has many restrictions on the scene, such as limiting it to speech or music, or limiting the super-resolution ratio, such as 8k to 16k. Due to these limitations, the existing restoration methods have limited use scenarios on the one hand, and on the other hand, the types of problematic audio that can be repaired are relatively small.

[0067] In addition, related technologies have also proposed the use of artificial intelligence (AI) music generation technology based on diffusion models to generate high-quality audio. For example, large-model technology for music generation based on text descriptions, such as Riffusion (AI tool) fine-tunes the image-generated Stable Diffusion (text-to-image model), and then inputs the text description to obtain the audio spectrum information; another example is AudioLDM (audio generation technology based on latent space diffusion model), which generates the spectrum of music audio based on text, and then uses HiFi-GAN (high-fidelity generative adversarial network) to complete the conversion of spectrum to audio; in addition, StableAudio (AI tool, based on diffusion model and working in latent space), realizes the generation of high-sampling rate stereo music of arbitrary duration by introducing time coding. In the above scheme, the technology of generating music based on text guidance can generate higher quality audio, but its generation link can only ensure that the generated audio is consistent with the input text in macro-semantics. In the audio repair task, it is usually required that the repaired audio is compared with the source audio, except for the damaged or missing parts to be repaired, the song content and timbre, etc. remain unchanged, and the overall audio must be consistent with the source audio at the sampling point level, but the noise, distortion and loss therein are repaired and compensated, which is difficult to achieve for the relevant music generation model.

[0068] It can be seen that the relevant technology is usually implemented by training a convolutional neural network for a specific task, using a training set for a specific task, and then repairing the model after training and optimization. It can only solve the damaged audio of specific factors, and the usage scenarios are more limited. The large model of artificial intelligence to generate music uses a large amount of text-audio data for training and optimization, and realizes the generation of high-quality music audio from input text, which easily leads to changes in audio content and timbre. It should be noted here that although these pre-trained music generation large models are difficult to use directly for audio repair tasks, the rich music prior knowledge contained in them can be used in audio repair tasks, such as training related models (such as the variational autoencoder exemplified in the following text of the technical solution of this disclosure) to combine the audio content information of the source audio to achieve the repair of low-quality audio.

[0069] In view of this, the embodiments of the present disclosure provide an audio repair method, apparatus, medium and computing device, which obtains the first feature vector of the first audio to be repaired and the audio description information of the first audio, and based on the set time step, obtains the audio noise corresponding to each step in the time step, and generates a noise vector according to each audio noise, denoises the noise vector according to the first feature vector, the time step and the audio description information to obtain the target feature vector, and obtains the repaired second audio according to the target feature vector. In this process, the feature information of the source audio and the text description information of the source audio are combined to guide the audio noise to be denoised to obtain the repaired audio, which can achieve the restoration of low-quality audio without causing changes in the timbre or semantics of the audio, thereby improving the quality of audio repair, and can be applied to any type of audio data, with stronger universality, bringing a better experience to users.

[0070] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0071] Application Scenario Overview

[0072] First reference Figure 1 The application scenarios of the solutions provided in the present disclosure are illustrated by examples. Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure, such as Figure 1As shown, in this application scenario, the user opens an audio program on the terminal device 101, such as a music application. The user can initiate an audio repair request to the server 102 through the terminal device 101 according to the needs. The request can carry the first audio requested to be repaired. After receiving the request, the server 102 repairs the first audio (in some scenarios, the server can also automatically trigger the audio repair process, etc.), and sends the repaired second audio to the terminal device 101, and the terminal device 101 plays the repaired second audio in the music application. Optionally, during the audio repair process, the server 102 can obtain the target feature vector based on the feature vector and audio description information of the first audio, and obtain the repaired second audio based on the target feature vector. Optionally, during the audio repair process, the server 102 can also obtain the corresponding model (such as a set model, a variational autoencoder, etc.) from the server 103, and use the corresponding model to repair the audio. These models are trained in the server 103. For example, the music prior knowledge of the music generation large model can be used to train the set model.

[0073] Terminal devices may include, but are not limited to, computers, smart phones, tablet computers, e-book readers, Moving Picture experts group audio layer III (MP3) players, Moving Picture experts group audio layer IV (MP4) players, portable computers, car computers, wearable devices, desktop computers, set-top boxes, smart TVs, etc.

[0074] In this application scenario, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0075] It should be noted that Figure 1 This is only a schematic diagram of an application scenario provided by the embodiment of the present disclosure. Figure 1 does not limit the equipment included in Figure 1 The positional relationship between the devices in the Figure 1In the application scenario shown, a data storage device may also be included. The data storage device may be an external memory relative to the server 102 or an internal memory integrated in the server 102 .

[0076] Exemplary Methods

[0077] Combine the following Figure 1 For application scenarios, refer to Figure 1 To describe the audio repair method according to an exemplary embodiment of the present disclosure. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this regard. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0078] Please refer to Figure 2 , Figure 2 The flowchart of an audio repair method provided by an embodiment of the present disclosure is illustrated by taking the execution subject as the server 102 as an example. The method may include S201-S204:

[0079] Step S201: Obtain a first feature vector of a first audio to be repaired and audio description information of the first audio.

[0080] In this embodiment, the first feature vector may include time domain features, frequency domain features, etc., which may refer to a latent (spatial) vector (i.e., a vector representation after dimensionality reduction of the feature vector, which captures the potential structure and pattern of the data compared to the original feature vector). The audio description information may be a descriptive text information of the audio. Optionally, the audio description information may include any one or a combination of the rhythm parameters, instrument parameters, type, and audio description text of the first audio. The detailed acquisition process of the latent vector may be found in the following text, and will not be elaborated here.

[0081] In some embodiments, the feature vector may also refer to a multidimensional feature vector (i.e., a feature vector that has not been reduced in dimension), wherein the process of obtaining the first feature vector may be to extract the time domain features and frequency features in the first audio, including the extraction of time domain features, frequency domain features, time-frequency domain features, etc., wherein the time domain features of the audio signal are extracted, such as zero crossing rate, energy, short-time energy, etc., or the frequency domain features are extracted using Fourier transform (FFT), such as spectrum, spectrum centroid, spectrum bandwidth, etc.; or the time-frequency domain features are extracted using short-time Fourier transform (STFT) or Mel-frequency cepstral coefficient (MFCC), and the extracted features are combined into a multidimensional vector as the feature vector of the first audio. In addition, the extraction of audio description information may also be performed by extracting metadata of the audio (such as file name, duration, sampling rate, bit rate, etc.), and using natural language processing (NLP) technology to generate a text description of the audio, such as an introduction or label of the audio content.

[0082] It should be noted that in some embodiments, other methods may be used to extract feature vectors and audio description information. For example, a deep learning model (such as VGGish, OpenL3, etc.) may be used to obtain feature vectors.

[0083] Furthermore, in order to improve the extraction accuracy of the audio description information, the above-mentioned acquisition of the audio description information of the first audio may also be carried out in the following manner: using a multimodal audio text model to extract the audio description information from the first audio.

[0084] Wherein, multimodal audio text model is a machine learning model combining audio and text data, and is intended to utilize these two different types of data to improve the performance of tasks, such as speech recognition, sentiment analysis, speech synthesis, etc. In the present embodiment, multimodal audio text model, audio generation module needs the input of text prompt (input text that guides the model to generate specific output), and detailed text description can also input the key information of input audio into the workflow of audio generation module simultaneously, ensures the consistency of generated audio and original audio content. Multimodal audio text description module uses text-music multimodal model, completes extracting description information from music audio. Optionally, the present embodiment can use multimodal basic model (LLARK) model for music as the composition of multimodal audio text model.

[0085] like Figure 3 As shown, the LLARK model inputs the source audio x (i.e., the first audio) and the necessary text description such as: "Detailed description of the provided audio", and the output predicted by the model is a detailed description of the audio: the mode is F minor, the tempo is 121bpm, the instrument is piano, energetic female vocals, etc.

[0086] By extracting audio description information through a multimodal audio-text model, we can obtain more comprehensive and accurate audio description information and provide data support for audio restoration.

[0087] Step S202: based on the set time step, obtain the audio noise corresponding to each step in the time step, and generate a noise vector according to each audio noise.

[0088] In this embodiment, a time step is the minimum time unit for audio processing. Each time step represents a discrete time point or time period, at which corresponding audio features, such as notes, chords, rhythms, etc., can be generated or predicted. In this case, those skilled in the art can adaptively set the step size of the time step in combination with actual applications, and this embodiment does not specifically limit this. By dividing the audio into multiple time steps, the audio data can be processed step by step to ensure that the features in each time period can be accurately extracted and processed.

[0089] In this embodiment, the audio noise at each time step is obtained and synthesized into a large noise vector to facilitate the denoising process in subsequent steps.

[0090] Step S203: De-noise the noise vector according to the first feature vector, the time step and the audio description information to obtain a target feature vector.

[0091] In this embodiment, the noise vector is denoised according to the first feature vector, time step and audio description information. In each time step, the feature information of the source audio and the text content are combined to guide the audio noise denoising to obtain a vector representation that retains the original features of the audio. This can achieve processing for any audio, thereby solving the problem of limited application scenarios in related technologies.

[0092] Next, if Figure 4 As shown, the above step S203 denoises the noise vector according to the first feature vector, the time step and the audio description information to obtain a target feature vector, which may include the following steps:

[0093] S203a, determining a second eigenvector corresponding to the time step and a third eigenvector corresponding to the audio description information.

[0094] In this embodiment, the time step and the audio description information are used as features independent of the audio features (the first feature vector) and as guide vectors for denoising.

[0095] S203b. Perform a target number of denoising on the noise vector according to the first eigenvector, the second eigenvector, and the third eigenvector to obtain a target eigenvector, where the target number is the same as the number of the audio noise.

[0096] In this embodiment, the audio noise may be randomly introduced noise (such as noise introduced in the diffusion process of audio data using a diffusion model), or the noise of the audio data itself. Generally speaking, there is a random noise at each time step. In this embodiment, the noise vector is denoised in accordance with the number of audio noises to achieve denoising for each audio noise, thereby improving the denoising effect.

[0097] In some embodiments, denoising can be performed in a loop traversal manner to improve the denoising effect. Specifically, in the above steps, denoising the noise vector a target number of times according to the first eigenvector, the second eigenvector and the third eigenvector may include the following steps:

[0098] Acquire a first noise vector after the last denoising, and determine an intermediate vector according to the first eigenvector and the first noise vector;

[0099] inputting the intermediate vector, the second feature vector and the third feature vector into an adapter module;

[0100] Input the output parameter of the adapter module, the first noise vector, the second feature vector and the third feature vector into the audio generation module, obtain the second noise vector output by the audio generation module, and update the current denoising times;

[0101] In response to the denoising times being less than the target times, the second noise vector is updated to the first noise vector, and the step of determining the intermediate vector according to the first eigenvector and the second eigenvector is returned to be executed.

[0102] In this embodiment, for each denoising, an intermediate vector is determined by the denoised noise vector and the first eigenvector. The intermediate vector is the eigenvector of the noise that has undergone the previous denoising. The intermediate vector and the second eigenvector corresponding to the time step and the third eigenvector corresponding to the audio description information are input into the adapter module. The adapter module performs synchronous reasoning with the audio generation module and incorporates the reasoning results into the steps of the audio generation module to embed the source audio semantics and content, guide the module to generate latent vectors of high-quality audio, and finally restore the repaired high-quality audio through the audio latent space decoder.

[0103] In this embodiment, the adapter module inputs the latent space vector (i.e., the first eigenvector) of the source audio, outputs the sampling level guide of the audio generation module, and fuses the content information of the source audio into the generation link in each time step generated by the music generation module (i.e., the audio generation module). The audio generation module can be implemented based on a pre-trained music generation large model, such as a text-generated audio model (StableAudio) or other models, which brings musical prior knowledge to the audio repair task. It inputs a given time step and text description (i.e., audio description information), gradually generates audio from noise, and superimposes the content information of the adapter module therein, so that high-quality audio can be generated after repair.

[0104] Next, the determination of the intermediate vector according to the first feature vector and the first noise vector in the above step is further described:

[0105] Performing a convolution operation with an initial zero setting on the first eigenvector to obtain a fourth eigenvector;

[0106] The fourth eigenvector and the first noise vector are added to obtain the intermediate vector.

[0107] In this embodiment, for each intermediate vector (or first eigenvector) input into the adapter, the eigenvector may be preprocessed by an initial zero convolution operation, which may be to eliminate some unnecessary features or to perform feature standardization. An intermediate vector is obtained by convolving the fourth eigenvector after the initial zero convolution operation with the first noise vector. The intermediate vector contains mixed information of the original features and noise, which may be more suitable for subsequent denoising processing steps, and will not introduce other mixed information.

[0108] Through the above method, the feature vector of the audio and the noise vector can be superimposed and then input into the adapter for processing, and the feature information of the source audio can be retained, and the introduction of other error terms can be avoided.

[0109] In some embodiments, the adapter module may include a plurality of control units connected in sequence and a convolutional layer initially set to zero corresponding to each of the control units; the audio generation module includes a plurality of encoder units connected in sequence and a decoder unit corresponding to each of the encoder units, and each of the decoder units is connected in sequence;

[0110] Wherein, the output parameter of a control unit is input to the next control unit, and the output parameter of the control unit is input to the convolution layer corresponding to the control unit;

[0111] The output parameters of the convolutional layer are input to the decoder unit corresponding to the convolutional layer;

[0112] The output parameter of one encoder unit is input to the next encoder unit, and the output parameter of the one encoder unit is input to the decoder unit corresponding to the one encoder unit;

[0113] The output parameters of one decoder unit are input to the next decoder unit.

[0114] In this embodiment, in order to enable the audio generation module (such as StableAudio) to generate high-quality audio after recovery from low-quality audio, the generation process of StableAudio is guided by an adapter module. The adapter is mainly used to identify the content in the low-quality audio and achieve fine control generation at the sampling point level.

[0115] Exemplarily, the audio generation module can be a large text music generation model with the ability to generate music. By inputting the desired audio text description and noise, high-quality complete music is generated. The audio generation module can be generated based on models such as riffusion, stable audio, and suno. These pre-trained text music generation models can all serve as the main body of the audio generation module in the embodiments of the present disclosure. Take stableaudio as an example for explanation. Figure 5a As shown in the figure, the audio generation module is implemented using the Diffusion architecture, which consists of 4 levels of symmetrical (downsampling) encoder blocks (EncoderBlock) and (upsampling) decoder blocks (DecoderBlock). The downsampling encoder blocks and upsampling decoder blocks use jump connections. Each block consists of 2 convolutional residual layers, followed by a series of self-attention and cross-attention layers. During inference, the audio description information (prompt) and time step information (t) are converted into feature vectors and noise vectors (Zt) and input into the diffusion network (DiffusionUnet), where Zt is initially a random noise (hidden vector) calculated based on T time steps, and each subsequent time step is output by DiffusionUnet Zt-1. After T time steps, the audio latent vector Z is obtained. 0 , and converted to audio representation through the VAE decoder. In other words, each time the input is input, the noise is removed once, and the input includes the description text and the noise input Z. For example, when the first input is Z t and audio description information, and finally input Z1 and audio description information to get the output Z 0 .

[0116] Among them, the above-mentioned audio generation module can be trained using massive audio data, and the model contains rich audio prior data, which can help to achieve audio restoration tasks efficiently. Optionally, the audio generation module can fix the pre-trained weights of the model, and obtain audio prior data by reasoning according to the conventional reasoning link, and can add semantic information extracted from the original audio by the multimodal audio text description module and content information extracted from the source audio by the adapter module during the reasoning process, guiding the audio generation module to generate reasoning towards the content of the source audio.

[0117] By way of further example, the structure of the adapter module (connected to the audio generation module) is as follows: Figure 5b As shown in the figure, the adapter module can include a control module (ControlBlock) with the same structure as the encoder module in the diffusion network (DIffusionUnet) and a zero convolution (ZeroConv) with all weights initialized to 0. The latent vector (Zx, i.e., the first eigenvector) of the source audio is processed by zero convolution (1*1 convolution with initial zero) and added to the noise vector (Zt), and input into the four cascaded control modules (ControlBlock) of the control network (ControlNeT) together with the audio description information (prompt) and the time step t. The output of each layer of the control module is added to the encoder block of the corresponding diffusion network after zero convolution processing as the input of the decoder block, realizing the embedding of the content information of the source audio into the generation link of the audio generation module. Similar to the diffusion network, the control network (ControlNeT) will act on each of the T time steps.

[0118] During the training process, the weights in DiffusionUnet can be fixed, and only the weights in the control network (ControlNeT) need to be optimized. Using the generated training data, low-quality audio is input into the audio latent space encoder to generate Zx, and high-quality audio is used as the training target, and the Controlnet weights are optimized in the same way as the diffusion model training.

[0119] Through the above technical solution, the above structure of the adapter can achieve fine control generation at the sampling point level of audio data.

[0120] Continue to refer to Figure 2 , step S204, obtaining the repaired second audio according to the target feature vector.

[0121] In this embodiment, after obtaining the target feature vector, a deep learning method, such as a variational autoencoder (VAE) or a generative adversarial network (GAN), can be used to obtain the corresponding audio signal according to the feature vector.

[0122] In some embodiments, the second audio corresponding to the target feature vector can also be obtained by other inversion methods of extracting the first feature vector from the source audio (first audio), such as short-time Fourier transform (STFT), Mel spectrum coefficient (MFCC), etc. to extract the first feature vector, and MFCC can be used to reversely convert it back to Mel spectrum, and then converted back to linear spectrum. Alternatively, the spectrum data can be converted back to a time domain signal using an inverse Fourier transform (such as inverse STFT). The present disclosure does not specifically limit the process of obtaining the second audio.

[0123] In the above technical solution, the text content, audio features, time steps and other information of the audio to be repaired are used to guide the audio noise to be denoised to obtain the repaired audio. Any type of audio can be repaired, which effectively solves the problems of limited usage scenarios and fewer types of repaired audio in related technologies, and is more in line with the content, timbre, etc. of the source audio.

[0124] Please refer to Figure 6 , Figure 6 It is a flow chart of another audio repair method provided in an embodiment of the present application. On the basis of the above embodiment, a codec is used to extract feature vectors and restore audio. The above step S201 obtains the first feature vector of the first audio to be repaired and the audio description information of the first audio, which is further divided into steps S201a-S201c, and the above step S204 is further divided into step S204a.

[0125] Step S201a: Control an encoder in the setting model to encode valid content in the first audio into a latent vector in a latent space, where the valid content is used to indicate audio content other than noise in the first audio.

[0126] In this embodiment, the setting model can be any neural network model with an encoder and a decoder. By controlling the encoder in the setting model, the first audio to be repaired is input into the encoder, and in the latent space, the encoder encodes the effective content in the first audio into a latent vector, wherein the effective content is the pure audio content after removing the noise.

[0127] It can be understood that latent space is a low-dimensional representation space that can be used to capture the potential structure and features of input data. This embodiment uses latent space to encode latent vectors in the set model to achieve compression and abstraction of the original audio, extract key features of the audio, and remove noise.

[0128] Step S201b: Control the decoder in the set model to decode the latent vector into intermediate audio in the latent space.

[0129] In this embodiment, the decoder can decode the latent vector back to high-dimensional audio data. By decoding the latent vector into intermediate audio, it is convenient to extract the audio description information later. The structure of the decoder is usually the reverse process of the encoder.

[0130] Step S201c: determine the latent vector as the first feature vector, and extract the audio description information from the intermediate audio.

[0131] In this embodiment, the latent vector is a compression and dimensionality reduction of the effective content of the first audio. By decoding it, a denoised intermediate audio can be obtained. That is, the intermediate audio is equivalent to a denoised version of the first audio. The audio description information is extracted based on the intermediate audio, which has less noise and more accurate audio description information.

[0132] Step S204a: Control the decoder to decode the target feature vector into the second audio in a latent space.

[0133] The target feature vector is a clearer audio feature obtained by denoising the denoised vector by combining the time step, the first feature vector, and the audio description information. The decoder decodes it in the source space to obtain the repaired second audio. During the repair process, the overall audio is consistent with the source audio sampling point level. Compared with the source audio, the noise, distortion and loss of the source audio are repaired. By introducing the audio description information, the audio content and timbre remain unchanged.

[0134] The above-mentioned setting model can be trained in other servers (such as server 103) and obtained from other servers, or it can be trained and generated in the server. The following describes the training process of the setting model, taking the training in the server as an example. Figure 7 As shown, it also includes step S701 and step S702 of setting the model training. The specific steps of setting the model training are as follows:

[0135] S701. Acquire multiple training samples, where the training samples include high-quality audio and low-quality audio, where the high-quality audio is used to indicate audio with sound quality higher than or equal to a preset sound quality, and the low-quality audio is used to indicate audio with sound quality lower than a preset sound quality.

[0136] In this embodiment, audio data may be collected from at least one audio data source to obtain training samples. For example, multiple training samples may be obtained from an online audio library, a recording device (using high-quality and low-quality recording devices to record audio respectively), or an existing data set (using an existing annotated audio data set).

[0137] It should be noted that those skilled in the art can adaptively determine the preset sound quality in combination with actual applications and prior art. The preset sound quality can be determined according to the requirements of a specific application scenario, for example, by measuring parameters such as signal-to-noise ratio (SNR), sampling rate, bit rate, etc.

[0138] In some embodiments, the above-mentioned step S701 obtains multiple training samples, which may include the following steps: obtaining multiple high-quality audios; performing sound quality degradation processing on each of the high-quality audios to obtain multiple low-quality audios; and constructing training samples based on the high-quality audios and the corresponding low-quality audios.

[0139] In one example, high-quality audio may be obtained by other means, such as obtaining an annotated high-quality audio dataset from the platform.

[0140] In another example, in order to ensure that high-quality audio data is not affected by other factors (for example, audio data obtained from the platform is damaged due to network delays or other reasons), the above steps can be used to obtain multiple high-quality audios in the following manner: obtain audio to be processed with sound quality higher than or equal to a preset sound quality from the network, and the sampling rate of the audio to be processed is greater than the preset sampling rate; resample each of the audio to be processed into audio with a set sampling rate as high-quality audio.

[0141] The network may refer to an audio platform. In one implementation, the sampling rate on the platform may be greater than 48 khz (preset sound quality), and the saved format may be a lossless format such as wav, flac, etc. (to reduce audio loss), and the audio may be uniformly resampled to 44.1 khz audio to avoid the loss of ultra-high frequencies caused by the sound source, recording, etc., which may affect the repair ability of the model. The collected audio is used as high-quality audio for the training set.

[0142] Next, the audio degradation process in the above high-quality audio acquisition process is further introduced. Specifically, in the above steps, the sound quality degradation process is performed on each of the high-quality audios, which may include the following steps: inputting the high-quality audio into a degradation processor to obtain low-quality audio output by the degradation processor.

[0143] The degradation processor may be a pre-trained tool for degrading high-quality audio, and may be obtained by training with a large amount of prior audio data.

[0144] It is understandable that low-quality audio can be audio of poor quality due to age, storage, transcoding, transcription, etc., which is manifested in sound distortion, noise, low bit rate, frequency band damage, etc. compared to high-quality audio, and has poor listening experience, and is an object that needs to be repaired by the technical solution of the present disclosure. In the embodiment of the present disclosure, the relevant algorithm modules need to be trained and optimized with pairs of high-quality audio and low-quality audio. In order to generate low-quality audio corresponding to the above high-quality audio, a low-quality audio generation pipeline is constructed, and high-quality audio is used as input to simulate the generation of low-quality audio, so that the correspondence between high-quality audio and low-quality audio can be achieved. Among them, the degradation processor may include one or more degradation tools, and each time the input audio is randomly degraded using 0 or more tools in different degradation tool groups to obtain low-quality audio.

[0145] Exemplarily, the above-mentioned degradation processor includes at least one of the following functions: adding noise to audio, cutting audio, spectral energy mask, adding reverberation, increasing or decreasing volume, modifying signal amplitude, simulating distortion of audio, encoding audio with a lossy encoder, sampling audio with a lossy sound quality sampling format, and encoding audio with lossy sound quality encoding parameters.

[0146] In this example, the degradation processor can be configured to include three degradation tool groups, each of which can include multiple degradation tools, such as a noise tool group, a sound effect tool group, and a codec tool group. Each time the input audio is randomly degraded using 0 or more tools in different degradation tool groups to obtain low-quality audio. Figure 8 As shown, after high-quality audio passes through these three tool groups, low-quality audio is obtained.

[0147] Among them, the noise tool group can include randomly adding noise of different degrees, randomly cutting and interrupting audio, random spectral energy masking, etc.; the sound effect tool group can include random strong reverberation, randomly increasing the volume and using a limiter, random simulated distortion, etc.; the codec group can include different lossy encoders such as mp3, aac, Ogg, etc., and cooperate with different encoding parameters such as bit rate 96kbps, 128kbps, 320kbps and sampling format signed16, signed32, float, etc.

[0148] In some examples, the collected high-quality audio can be repeatedly input into a degradation pipeline (i.e., multiple tool groups of a degradation processor) n times to obtain training data pairs (high-quality audio, low-quality audio) that are n times the number of high-quality audio.

[0149] In some examples, in addition to using the pipeline degradation method of the above-mentioned multiple degradation tool groups, any degradation tool in the degradation tool group can also be used to perform single degradation or combined degradation, which is not particularly limited in this embodiment.

[0150] Continue to refer to Figure 7 , step S702, training the preset model according to each of the training samples to obtain the set model.

[0151] In this embodiment, the set model to be trained can be a variational autoencoder, including an encoder and a decoder. Fig. 9 As shown, the encoder converts the audio signal x (first audio) to the latent space representation z (first eigenvector). After the audio is repaired in the latent space, the decoder converts the latent space z representation back to the audio signal x' (second audio), which can significantly improve the efficiency of audio repair. The encoder and decoder are a predefined deep learning model, whose input and output can be time domain audio signals or Fourier transformed audio spectra according to different implementations. The parameters of the encoder and decoder need to be trained in advance. By training them with the training samples obtained above, the model converges to obtain the final variational autoencoder.

[0152] Exemplarily, the above-mentioned step S602 trains the preset model according to each of the training samples, and may include the following steps: training the preset model based on the high-quality audio in each of the training samples to obtain an intermediate model, and the loss function of the preset model is updated based on the reconstruction loss function, the adversarial loss function, the adversarial feature loss function, the divergence loss function, the high-quality audio and the low-quality audio; training the intermediate model according to each of the training samples to obtain the set model.

[0153] In the text-generated audio scheme of the related art, the training goal is to keep the input audio x and the output audio x' consistent. In the disclosed embodiment, the input audio x is low-quality audio, and the output audio x' is high-quality audio, so the encoder needs to adapt to the low-quality audio. For this reason, during the training process, this embodiment proposes a two-step optimization scheme.

[0154] In the first step, using high-quality audio data, you can train the variational autoencoder according to the following training scheme. The training loss includes: reconstruction loss L recon , adversarial loss L D , adversarial feature loss L DF and KL divergence loss L klThe generated adversarial loss is a discriminator based on full convolution. When x and x' are input, the discriminator judges the input audio and the reconstructed audio, and extracts the features of the middle layer of the discriminator to calculate the adversarial feature loss. Aggregate all losses and construct L according to the following formula VAE loss, complete the first step of training until convergence, where λ 1 ,λ 2 ,λ 3 ,λ 4 , are weight coefficients, which can be adjusted based on experience.

[0155] Lrecon=||xx′||1

[0156] LDF=||D(x)-D(x′)||1

[0157] LVAE=λ1Lrecon+λ2LD+λ3LDF+λ4LKL

[0158] In the second step, based on the encoder and decoder (i.e., the intermediate model) trained in the first step, we use the constructed pairs of high-quality audio data and low-quality audio data to fine-tune the encoder using the following formula to adapt it to the low-quality audio and prevent the encoder from mistaking impurities in the low-quality audio as audio content and subsequent repair of the sound effect. We construct the training loss function L according to the following formula to complete the fine-tuning training of the encoder: where Decoder refers to the decoder, Encoder refers to the encoder, and x hq Refers to high quality audio, x lq Refers to low quality audio.

[0159] L=||Decoder(Encoder(xhq))-Decoder(Encoder(xlq))||1

[0160] When the above loss L meets the loss condition, the set model can be obtained, otherwise the parameter adjustment training continues. Optionally, in this example, if the large model of text music generated based on pre-training in the relevant technology is implemented, the weight of the existing model can be directly used in the above first step without further training, which can improve the training efficiency.

[0161] Through the above technical solution, the model is trained on the server side, and based on the audio prior knowledge in the pre-trained music generation model, an adaptive optimization solution for the VAE encoder for low-quality audio repair is provided, which can effectively improve the quality of low-quality music audio after repair. In addition, the music audio repair solution based on the above model can meet the needs of low-quality audio repair in different usage scenarios with different degradation factors.

[0162] To facilitate understanding of the embodiments of the present disclosure, an exemplary embodiment is provided below for illustration:

[0163] The goal of the disclosed embodiments is to repair low-quality audio and generate corresponding high-quality audio. The rich music prior knowledge of the pre-trained music generation model can be used to repair low-quality audio, extract semantic information from low-quality audio, enhance the ability to repair highly damaged audio, fully integrate the content information of the source audio, and ensure the consistency of the content details of the audio before and after repair.

[0164] Optionally, the embodiments of the present disclosure may implement the steps of audio restoration in the above embodiments through the following modules: an audio latent space encoding and decoding module, a multimodal audio text description module, an adapter module and an audio generation module. Among them, the audio latent space encoding and decoding module uses the encoder to compress low-quality audio and reduce its dimension to a latent vector (Embedding vector in deep learning). Subsequent operations will complete the audio repair in the latent space (a space that can provide users with private file storage). Finally, the repaired latent vector is converted back by the decoder to obtain high-quality audio; the multimodal audio text description module can be used to extract the semantic information of music from the source audio. Specifically, it inputs audio and outputs description text. The description text will serve as the semantic support for subsequent audio repair to ensure that semantic support is still provided when the audio is severely damaged; the adapter module inputs the latent space vector of the source audio and outputs the sampling-level guidance of the audio generation module, and integrates the content information of the source audio into the generation link in each time step generated by the music generation module; the audio generation module can be implemented based on a pre-trained music generation large model to provide prior knowledge of musicality for the audio repair task. It inputs a given time step and text description, gradually generates audio from noise, and superimposes the content information of the adapter module on it to generate high-quality audio after repair.

[0165] For example, in combination Fig.10As shown, by inputting a low-quality audio, first using the audio latent space encoding module, the audio is converted into the latent vector space to obtain the latent vector Zx (i.e., the first eigenvector), and then using the audio latent space decoding module to restore it to the preprocessed audio. At this time, the restored audio is only roughly repaired (i.e., the intermediate audio). The preprocessed audio is input into the multimodal audio text description module to obtain the text description information prompt (i.e., audio description information) of the current audio. The description information may include the following two aspects: 1. Metadata information of the song: beat, key signature, BPM, time signature, instrument, style, etc.; 2. Description of the content of the song: "A string quartet is playing lively and energetically. The song begins with a fast and artistic melody played by the violin in the high range. The notes played include F6, a#6, C7 and F5, creating an engaging and energetic opening theme. […] The interaction between the violin, cello and viola creates a rich and dynamic sound." On the one hand, the text description information prompt and the time step information t are used as the input of the audio generation module, and reasoning is performed according to the conventional generation link. On the other hand, the text description information prompt, the time step information t and the latent vector Zx of the source audio are used as the input of the adapter module. The adapter module synchronously reasoned with the audio generation module and integrated the reasoning results into the steps of the audio generation module to embed the semantics and content of the source audio, guide the module to generate the latent vector of high-quality audio, and finally restore the repaired high-quality audio through the audio latent space decoder. Optionally, the above-mentioned multimodal audio text description module and audio generation module can be directly implemented based on the pre-trained model, and the audio latent space codec module and adapter module can collect and produce data for audio repair tasks and train and optimize.

[0166] In summary, the embodiments of the present disclosure obtain semantic information of the source audio and embed the semantic information into the audio repair process, thereby meeting the need to repair the audio based on semantics when it is severely damaged, and can extract detailed content information of the source audio through the adapter module to ensure that the audio generation model generates high-quality audio consistent with the source audio content; in addition, an adaptation and optimization method for the VAE encoder for low-quality audio repair is provided, which can effectively improve the quality of low-quality music audio after repair. At the same time, the audio prior data of the large music generation model is combined for model training, which can meet the needs of low-quality audio repair in different usage scenarios with different degradation factors, and improve the universality of audio repair solutions.

[0167] Exemplary Media

[0168] After introducing the method of the exemplary embodiment of the present disclosure, next, refer to Fig.11 A storage medium according to an exemplary embodiment of the present disclosure is described.

[0169] refer to Fig.11 As shown, the storage medium 30 stores a program product for implementing the above method according to an embodiment of the present disclosure, which can be a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0170] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0171] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable signal medium may also be any readable medium other than a readable storage medium.

[0172] Program code for performing the disclosed operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN).

[0173] Exemplary Devices

[0174] After introducing the medium of the exemplary embodiment of the present disclosure, next, reference is made to Fig.12 An audio repair device according to an exemplary embodiment of the present disclosure is described, which is used to implement a method in any of the above method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0175] Fig.12FIG. 4 shows a schematic diagram of the structure of an audio repair device provided by an embodiment of the present disclosure. Fig.12 As shown, the device 120 may include a first acquisition module 121, a second acquisition module 122, a denoising module 123 and a third acquisition module 124, wherein:

[0176] A first acquisition module 121, configured to acquire a first feature vector of a first audio to be restored and audio description information of the first audio;

[0177] A second acquisition module 122 is used to acquire the audio noise corresponding to each time step based on the set time step, and generate a noise vector according to each audio noise;

[0178] A denoising module 123, configured to denoise the noise vector according to the first feature vector and the audio description information to obtain a target feature vector;

[0179] The third acquisition module 124 is used to acquire the restored second audio according to the target feature vector.

[0180] In one embodiment of the present disclosure, the denoising module 123 includes:

[0181] A first determining unit, configured to determine a second eigenvector corresponding to the time step and a third eigenvector corresponding to the audio description information;

[0182] A target frequency denoising unit is used to denoise the noise vector by a target frequency according to the first eigenvector, the second eigenvector and the third eigenvector to obtain a target eigenvector, where the target frequency is the same as the number of the audio noise.

[0183] In another embodiment of the present disclosure, the target frequency denoising unit is specifically used to:

[0184] Acquire a first noise vector after the last denoising, and determine an intermediate vector according to the first eigenvector and the first noise vector;

[0185] inputting the intermediate vector, the second feature vector and the third feature vector into an adapter module;

[0186] Input the output parameter of the adapter module, the first noise vector, the second feature vector and the third feature vector into the audio generation module, obtain the second noise vector output by the audio generation module, and update the current denoising times;

[0187] In response to the denoising times being less than the target times, the second noise vector is updated to the first noise vector, and the step of determining the intermediate vector according to the first eigenvector and the second eigenvector is returned to be executed.

[0188] In yet another embodiment of the present disclosure, determining an intermediate vector according to the first feature vector and the first noise vector includes:

[0189] Performing a convolution operation with an initial zero setting on the first eigenvector to obtain a fourth eigenvector;

[0190] The fourth eigenvector and the first noise vector are added to obtain the intermediate vector.

[0191] In yet another embodiment of the present disclosure, the adapter module includes a plurality of control units connected in sequence and a convolutional layer initially set to zero corresponding to each of the control units; the audio generation module includes a plurality of encoder units connected in sequence and a decoder unit corresponding to each of the encoder units, and each of the decoder units is connected in sequence;

[0192] Wherein, the output parameter of a control unit is input to the next control unit, and the output parameter of the control unit is input to the convolution layer corresponding to the control unit;

[0193] The output parameters of the convolutional layer are input to the decoder unit corresponding to the convolutional layer;

[0194] The output parameter of one encoder unit is input to the next encoder unit, and the output parameter of the one encoder unit is input to the decoder unit corresponding to the one encoder unit;

[0195] The output parameters of one decoder unit are input to the next decoder unit.

[0196] In yet another embodiment of the present disclosure, the first acquisition module 121 includes:

[0197] A first control unit, configured to control an encoder in a set model to encode valid content in the first audio into a latent vector in a latent space, wherein the valid content is used to indicate audio content other than noise in the first audio;

[0198] A second control unit is used to control a decoder in the set model to decode the latent vector into an intermediate audio in a latent space;

[0199] an extraction unit, configured to determine the latent vector as the first feature vector, and extract the audio description information from the intermediate audio;

[0200] The third acquisition module 124 includes:

[0201] The third control unit is used to control the decoder to decode the target feature vector into the second audio in a latent space.

[0202] In yet another embodiment of the present disclosure, the training steps of the setting model are specifically as follows:

[0203] Acquire a plurality of training samples, wherein the training samples include high-quality audio and low-quality audio, wherein the high-quality audio is used to indicate audio with sound quality higher than or equal to a preset sound quality, and the low-quality audio is used to indicate audio with sound quality lower than the preset sound quality;

[0204] The preset model is trained according to each of the training samples to obtain the set model.

[0205] In yet another embodiment of the present disclosure, the training of a preset model according to each of the training samples includes:

[0206] Based on the high-quality audio in each of the training samples, a preset model is trained to obtain an intermediate model, and the loss function of the preset model is updated based on the reconstruction loss function, the adversarial loss function, the adversarial feature loss function, the divergence loss function, the high-quality audio, and the low-quality audio;

[0207] The intermediate model is trained according to each of the training samples to obtain the set model.

[0208] In yet another embodiment of the present disclosure, the obtaining of a plurality of training samples includes:

[0209] Get multiple high-quality audios;

[0210] Performing sound quality degradation processing on each of the high-quality audios to obtain multiple low-quality audios;

[0211] A training sample is constructed according to the high-quality audio and the corresponding low-quality audio.

[0212] In yet another embodiment of the present disclosure, the step of performing sound quality degradation processing on each high-quality audio comprises:

[0213] The high-quality audio is input into a degradation processor to obtain low-quality audio output by the degradation processor.

[0214] In yet another embodiment of the present disclosure, the degradation processor includes at least one of the following functions:

[0215] Add noise to audio, cut audio, spectral energy mask, add reverberation, increase or decrease volume, modify signal amplitude, simulate distortion to audio, encode audio using a lossy encoder, sample audio using a lossy sampling format, and encode audio using lossy encoding parameters.

[0216] In yet another embodiment of the present disclosure, the obtaining of a plurality of high-quality audios includes:

[0217] Acquire, from a network, an audio to be processed whose sound quality is higher than or equal to a preset sound quality, wherein a sampling rate of the audio to be processed is higher than a preset sampling rate;

[0218] Each of the to-be-processed audios is resampled into audio with a set sampling rate as high-quality audio.

[0219] In yet another embodiment of the present disclosure, the audio description information includes rhythm parameters, instrument parameters, type, and audio description text of the first audio.

[0220] In yet another embodiment of the present disclosure, the first acquisition module 121 is specifically configured to:

[0221] The audio description information is extracted from the first audio using a multimodal audio-text model.

[0222] Exemplary Computing Devices

[0223] After introducing the method, medium and apparatus of the exemplary embodiments of the present disclosure, next, reference is made to Fig.13 A computing device according to an exemplary embodiment of the present disclosure is described.

[0224] Fig.13 The computing device 40 shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present disclosure.

[0225] like Fig.13 As shown, the computing device 40 is in the form of a general computing device. The components of the computing device 40 may include, but are not limited to: at least one processing unit 401, at least one storage unit 402, and a bus 403 connecting different system components (including the processing unit 401 and the storage unit 402). Among them, at least one storage unit 402 stores computer-executable instructions; at least one processing unit 401 includes a processor, and the processor executes the computer-executable instructions to implement the method described above.

[0226] The bus 403 includes a data bus, a control bus, and an address bus.

[0227] The storage unit 402 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 4021 and / or a cache memory 4022 , and may further include a readable medium in the form of a non-volatile memory, such as a read-only memory (ROM) 4023 .

[0228] The storage unit 402 may also include a program / utility 4025 having a set (at least one) of program modules 4024, such program modules 4024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0229] The computing device 40 may also communicate with one or more external devices 404 (e.g., a keyboard, a pointing device, etc.). Such communication may be performed via an input / output (I / O) interface 405. Furthermore, the computing device 40 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 406. Fig.13 As shown, network adapter 406 communicates with other modules of computing device 40 via bus 403. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with computing device 40, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0230] It should be noted that although several units / modules or sub-units / modules of the audio repair device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided to be embodied by multiple units / modules.

[0231] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0232] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. An audio repair method, characterized in that: include: Obtaining a first feature vector of a first audio to be restored and audio description information of the first audio; Based on a set time step, obtaining audio noise corresponding to each step in the time step, and generating a noise vector according to each audio noise; Denoising the noise vector according to the first feature vector, the time step, and the audio description information to obtain a target feature vector; Acquire a restored second audio according to the target feature vector.

2. The audio repair method according to claim 1, characterized in that: The denoising the noise vector according to the first feature vector, the time step, and the audio description information to obtain a target feature vector includes: Determining a second eigenvector corresponding to the time step and a third eigenvector corresponding to the audio description information; The noise vector is denoised a target number of times according to the first eigenvector, the second eigenvector, and the third eigenvector to obtain a target eigenvector, where the target number is the same as the amount of the audio noise.

3. The audio repair method according to claim 2, characterized in that: Performing a target number of denoising operations on the noise vector according to the first eigenvector, the second eigenvector, and the third eigenvector includes: Obtaining a first noise vector after the last denoising, and determining an intermediate vector according to the first eigenvector and the first noise vector; inputting the intermediate vector, the second eigenvector, and the third eigenvector into an adapter module; Input the output parameter of the adapter module, the first noise vector, the second eigenvector, and the third eigenvector into the audio generation module to obtain the second noise vector output by the audio generation module, and update the current denoising times; In response to the denoising number being less than the target number, the second noise vector is updated to the first noise vector, and the step of determining the intermediate vector based on the first eigenvector and the second eigenvector is returned to be executed.

4. The audio repair method according to claim 3, characterized in that: The determining of the intermediate vector according to the first eigenvector and the first noise vector includes: Performing a convolution operation with an initial zero setting on the first eigenvector to obtain a fourth eigenvector; The fourth eigenvector and the first noise vector are added to obtain the intermediate vector.

5. The audio repair method according to claim 3, characterized in that: The adapter module includes a plurality of control units connected in sequence and a convolutional layer initially set to zero corresponding to each control unit; the audio generation module includes a plurality of encoder units connected in sequence and a decoder unit corresponding to each encoder unit, and each decoder unit is connected in sequence; The output parameters of a control unit are input to the next control unit, and the output parameters of the control unit are input to the convolution layer corresponding to the control unit; The output parameters of the convolutional layer are input to the decoder unit corresponding to the convolutional layer; The output parameter of one encoder unit is input to the next encoder unit, and the output parameter of the encoder unit is input to the decoder unit corresponding to the encoder unit; The output parameters of one decoder unit are input to the next decoder unit.

6. The audio repair method according to claim 1, characterized in that: The obtaining of a first feature vector of the first audio to be repaired and audio description information of the first audio includes: Controlling an encoder in the setting model to encode valid content in the first audio into a latent vector in a latent space, where the valid content is used to indicate audio content other than noise in the first audio; Controlling a decoder in the set model to decode the latent vector into intermediate audio in a latent space; Determine the latent vector as the first feature vector, and extract the audio description information from the intermediate audio; The step of obtaining the restored second audio according to the target feature vector includes: The decoder is controlled to decode the target feature vector into the second audio in a latent space.

7. The audio repair method according to claim 6, characterized in that: The training steps of the setting model are as follows: Acquire a plurality of training samples, the training samples comprising high-quality audio and low-quality audio, the high-quality audio being used to indicate audio having a sound quality higher than or equal to a preset sound quality, and the low-quality audio being used to indicate audio having a sound quality lower than the preset sound quality; The preset model is trained according to each of the training samples to obtain the set model.

8. An audio repair device, characterized in that: include: A first acquisition module, configured to acquire a first feature vector of a first audio to be restored and audio description information of the first audio; A second acquisition module is used to acquire audio noise corresponding to each time step based on a set time step, and generate a noise vector according to each audio noise; a denoising module, configured to denoise the noise vector according to the first feature vector and the audio description information to obtain a target feature vector; The third acquisition module is configured to acquire the restored second audio according to the target feature vector.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method according to any one of claims 1 to 7 is implemented.

10. A computing device, characterized in that include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the computing device to perform the method according to any one of claims 1 to 7.

Citation Information

Cited By

  • Speaker confirmation method and device, equipment and storage medium

    CN119495306A

  • Speaker confirmation method, device, equipment and storage medium

    CN119495306B

  • Audio distortion repairing method, audio playing method and system

    CN120015056A

  • Audio super-resolution processing method and system

    CN120032654A

  • Method for repairing sound quality, electronic equipment, storage medium and program product

    CN120510857A