Artificial intelligence-based audio optimization methods, devices, computer equipment, and media

Abstract

This invention relates to the field of audio processing technology, and more particularly to an audio optimization method, apparatus, computer device, and medium based on artificial intelligence. The method uses a linear layer to map the spectral envelope of the audio to be optimized to obtain envelope features; uses an embedding layer to embed standard audio parameters as parametric features; uses a prediction model to predict the fusion features of the envelope features and parametric features to obtain a predicted pitch curve; uses a noise-adding model to add noise to the Mel spectrum of the audio to be optimized to obtain a noise-adding result; uses a noise estimation model to calculate the noise of the noise-adding result to obtain predicted noise; updates the noise estimation model based on the predicted noise, the actual noise, and the predicted pitch curve; uses the updated noise estimation model to calculate the reference noise of the noise-adding result; and denoises the noise-adding result based on the reference noise to obtain the optimized Mel spectrum. Combining pitch information with the optimized noise estimation model ensures that the denoising process meets pitch requirements, thus improving the audio optimization effect.

Description

Technical Field

[0001] This invention relates to the field of audio processing technology, and in particular to an audio optimization method, apparatus, computer device, and medium based on artificial intelligence. Background Technology

[0002] Audio optimization tasks refer to optimizing the pitch curve of the audio to be optimized so that the optimized pitch curve can meet the preset pitch standard. Audio optimization tasks can be applied to various application scenarios such as intelligent pitch correction, audio enhancement, and speech synthesis. For example, in the intelligent pitch correction scenario, it can optimize the audio to be optimized that is out of tune, so as to obtain optimized audio that is more in line with the musical score.

[0003] However, existing audio optimization tasks typically employ generative models. These models require both low-quality and high-quality audio from the same person to ensure accuracy. In real-world applications, however, only low-quality audio is often available. This means generative models can only further improve the quality of high-quality audio, failing to achieve good optimization results for low-quality audio. Consequently, audio optimization exhibits poor generalization, resulting in suboptimal optimization performance. Therefore, improving the accuracy of audio optimization is a pressing issue. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an audio optimization method, apparatus, computer device, and medium based on artificial intelligence to solve the problem of poor audio optimization results.

[0005] In a first aspect, embodiments of the present invention provide an audio optimization method based on artificial intelligence, the audio optimization method comprising:

[0006] Obtain the spectral envelope of the audio to be optimized, perform feature mapping on the spectral envelope using a trained linear layer to obtain envelope features, determine standard audio parameters based on the audio to be optimized, and perform feature embedding on the standard audio parameters through a trained embedding layer to obtain parameter features;

[0007] The envelope features and the parameter features are fused to obtain fused features. The trained prediction model is then used to predict the pitch of the fused features to obtain the predicted pitch curve.

[0008] Obtain the Mel spectrum of the audio to be optimized, and add noise to the Mel spectrum using a noise model to obtain the noise-added result;

[0009] The noise estimation model is used to calculate the noise in the noise-added result to obtain the predicted noise. Based on the predicted noise, the real noise corresponding to the noise-added model and the predicted pitch curve, the prediction loss is calculated. Based on the prediction loss, the noise estimation model is updated to obtain the updated noise estimation model.

[0010] The noise is calculated on the noise-added result using the updated noise estimation model to obtain reference noise. The noise-added result is then denoised based on the reference noise to obtain a denoised result, which is used as the optimized Mel spectrum.

[0011] Secondly, embodiments of the present invention provide an audio optimization device based on artificial intelligence, the audio optimization device comprising:

[0012] The feature extraction module is used to obtain the spectral envelope of the audio to be optimized, perform feature mapping on the spectral envelope using a trained linear layer to obtain envelope features, determine standard audio parameters based on the audio to be optimized, and perform feature embedding on the standard audio parameters through a trained embedding layer to obtain parameter features.

[0013] The pitch prediction module is used to fuse the envelope features and the parameter features to obtain fused features, and use a trained prediction model to predict the pitch of the fused features to obtain a predicted pitch curve.

[0014] The spectrum noise-adding module is used to acquire the Mel spectrum of the audio to be optimized, and to add noise to the Mel spectrum using a noise-adding model to obtain the noise-adding result;

[0015] The model update module is used to perform noise calculation on the noise-added result using a noise estimation model to obtain predicted noise, calculate the prediction loss based on the predicted noise, the real noise corresponding to the noise-added model and the predicted pitch curve, and update the noise estimation model based on the prediction loss to obtain an updated noise estimation model.

[0016] The spectrum optimization module is used to perform noise calculation on the noise-added result using the updated noise estimation model to obtain reference noise, and to denoise the noise-added result based on the reference noise to obtain a denoised result, which is then used as the optimized Mel spectrum.

[0017] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the audio optimization method as described in the first aspect.

[0018] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the audio optimization method as described in the first aspect.

[0019] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows:

[0020] The process involves obtaining the spectral envelope of the audio to be optimized, performing feature mapping on the spectral envelope using a trained linear layer to obtain envelope features, determining standard audio parameters based on the audio to be optimized, embedding features of the standard audio parameters using a trained embedding layer to obtain parametric features, fusing the envelope features and parametric features to obtain fused features, using a trained prediction model to predict pitch based on the fused features to obtain a predicted pitch curve, obtaining the Mel spectrum of the audio to be optimized, adding noise to the Mel spectrum using a noise addition model to obtain the noise addition result, calculating the noise using a noise estimation model to obtain the predicted noise, and then combining the predicted noise and the noise addition... The model is compared with the actual noise and the predicted pitch curve. The prediction loss is calculated, and the noise estimation model is updated based on the prediction loss. The updated noise estimation model is then used to calculate the noise in the noise-added result to obtain the reference noise. The noise-added result is then denoised based on the reference noise to obtain the denoised result. The denoised result is used as the optimized Mel spectrum and combined with the predicted pitch curve to predict the noise. This ensures that the predicted noise meets the pitch requirements of the denoising process. Furthermore, the audio optimization is performed through noise addition and denoising, resulting in higher audio quality than conventional methods and improving the audio optimization effect. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of an application environment for an audio optimization method based on artificial intelligence provided in Embodiment 1 of the present invention;

[0023] Figure 2 This is a flowchart illustrating an audio optimization method based on artificial intelligence provided in Embodiment 1 of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of an audio optimization device based on artificial intelligence provided in Embodiment 2 of the present invention;

[0025] Figure 4This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0027] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0028] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0030] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0032] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0033] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0034] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0035] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0036] The first embodiment of this invention provides an artificial intelligence-based audio optimization method, which can be applied to applications such as... Figure 1 In this application environment, the client communicates with the server. The client includes, but is not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0037] The server can be deployed in various application scenarios such as intelligent audio editing, audio enhancement, and speech synthesis. Taking the intelligent audio editing scenario as an example, the user enters the audio to be optimized through the client. At this time, the audio to be optimized is usually singing audio. The intelligent audio editing task is to beautify the singing audio, so that the beautified singing audio is more pleasant and more professional while retaining the timbre of the singing audio.

[0038] See Figure 2This is a flowchart illustrating an artificial intelligence-based audio optimization method provided in Embodiment 1 of the present invention. The audio optimization method described above can be applied to... Figure 1 In this context, the server-side computer connects to the client to obtain the spectral envelope and Mel spectrum of the audio to be optimized. The server-side computer can be equipped with a pre-trained embedding layer, a pre-trained linear layer, a pre-trained prediction model, a noise-adding model, and a noise estimation model. The pre-trained embedding layer, linear layer, and prediction model can be used to extract features from the audio to be optimized, and pitch prediction is performed based on the extracted features. The noise-adding model and noise estimation model are used for audio optimization. Figure 2 As shown, the audio optimization method may include the following steps:

[0039] Step S201: Obtain the spectral envelope of the audio to be optimized, perform feature mapping on the spectral envelope using a trained linear layer to obtain envelope features, determine standard audio parameters based on the audio to be optimized, and embed the standard audio parameters into features using a trained embedding layer to obtain parameter features.

[0040] Among them, the audio to be optimized can refer to the audio that needs to be optimized by the user through the client. The spectral envelope can represent the envelope feature information of the audio to be optimized. The standard audio parameters can refer to MIDI, which can refer to the standardized code that represents audio parameters. Audio parameters can include pitch, dynamics, rhythm, etc. The standard audio parameters can refer to the audio parameters under the standard pitch corresponding to the audio to be optimized, that is, the audio parameters under the professional pitch corresponding to the audio to be optimized.

[0041] The trained linear layer can be used to map the spectral envelope into envelope features, and the envelope features can be used to characterize the feature information of the spectral envelope. The trained embedding layer can be used to embed standard audio parameters into parametric features, and the parametric features can be used to characterize the features of the standard audio parameters.

[0042] Specifically, the spectral envelope can be extracted using audio information extraction tools, such as the pyworld library in Python, and the corresponding function for spectral envelope extraction is the pw.cheaptrick() function.

[0043] Standard audio parameters can be obtained using open-source software such as tony and Melodyne. After the spectral envelope passes through a trained linear layer, it needs to be processed by an activation function to obtain the envelope features. In this embodiment, the ReLU function can be used as the activation function.

[0044] Both the envelope features and the parametric features are represented in matrix form, and the matrix corresponding to the envelope feature has the same size as the matrix corresponding to the reference feature, so as to facilitate the subsequent fusion of the envelope features and the parametric features.

[0045] The steps described above—obtaining the spectral envelope of the audio to be optimized, using a trained linear layer to perform feature mapping on the spectral envelope to obtain envelope features, determining standard audio parameters based on the audio to be optimized, and embedding the standard audio parameters into features through a trained embedding layer to obtain parameter features—can fully extract the audio feature information of the audio to be optimized, so that subsequent pitch prediction has rich feature input, thereby improving the accuracy of pitch prediction.

[0046] Step S202: The envelope features and parameter features are fused to obtain fused features. The trained prediction model is used to predict the pitch of the fused features to obtain the predicted pitch curve.

[0047] The fusion features can include envelope features and parametric features. The trained prediction model can be used to predict pitch based on the fusion features, and the predicted pitch curve can be used to represent the pitch information of the audio to be optimized.

[0048] Specifically, fusion can be achieved through a connection method. Since the matrix corresponding to the envelope feature has the same matrix size as the matrix corresponding to the reference feature, the fused feature can be obtained by direct connection.

[0049] A trained prediction model can consist of the encoder structure of a Transformer model and a pre-defined linear prediction layer. The encoder structure of a Transformer model can include four encoding layers, and the pre-defined linear prediction layer can refer to a fully connected layer.

[0050] Optionally, the training process for the embedding layer, linear layer, and prediction model includes:

[0051] The sample audio spectrum envelope, sample audio parameters, and pitch labels are obtained. A linear layer is used to perform feature mapping on the sample spectrum envelope to obtain the sample envelope features. An embedding layer is used to embed the sample audio parameters to obtain the sample parameter features.

[0052] The sample envelope features and sample parameter features are fused to obtain the sample fused features. The pitch is then predicted using a prediction model to obtain the sample pitch curve.

[0053] Based on the sample pitch curves and pitch labels, the embedding layer, linear layer, and prediction model are trained to obtain the trained embedding layer, trained linear layer, and trained prediction model.

[0054] Here, sample audio can refer to the audio used to construct the training dataset. The training dataset includes the sample spectral envelope and sample audio parameters corresponding to several sample audios. The sample spectral envelope and sample audio parameters corresponding to a single sample audio are used as a training sample, and the pitch label of the corresponding sample audio is used as the label data of the training sample of the corresponding sample audio.

[0055] Sample envelope features can be used to characterize the feature information of sample spectral envelope, sample parameter features can be used to characterize the feature information of sample audio parameters, and sample fusion features can include the feature information of sample envelope features and sample parameter features.

[0056] Specifically, the sample audio can refer to audio data provided by non-professional users. Correspondingly, the pitch label of the sample audio can be non-professional pitch information, that is, the pitch information corresponding to the sample audio itself. Based on the sample pitch curve and pitch label, the difference loss is calculated through the difference loss function. Based on the difference loss, the embedding layer, linear layer and prediction model are trained until the difference loss converges, and the trained embedding layer, trained linear layer and trained prediction model are obtained. In this embodiment, the difference loss function can be the mean squared error loss function.

[0057] The trained embedding layer, trained linear layer, and trained prediction model can be input with professional audio parameters and non-professional envelope spectra during the inference stage. The professional audio parameters are also known as standard audio parameters, and the output is a professional pitch curve, which can be represented as a two-dimensional matrix.

[0058] In this embodiment, the embedding layer, linear layer, and prediction model are jointly trained, enabling the model to learn the correlation information between the spectral envelope and pitch. This allows the model to predict professional pitch curves based on professional audio parameters and non-professional spectral envelopes during the inference stage. Compared to existing methods that require training with both non-professional and professional audio from the same person, this method greatly reduces the difficulty of dataset construction. Training can be completed using only non-professional audio, improving the generalization and practicality of the embedding layer, linear layer, and prediction model.

[0059] Step S203: Obtain the Mel spectrum of the audio to be optimized, and use a noise-adding model to add noise to the Mel spectrum to obtain the noise-adding result.

[0060] Among them, the Mel spectrum can be used to characterize the audio features of the audio to be optimized, and the noise model can be used to add Gaussian noise to the Mel spectrum. The noise result can refer to the Mel spectrum with Gaussian noise added.

[0061] Specifically, the Mel spectrum can be obtained by sequentially performing pre-emphasis, frame segmentation, windowing, Fourier transform, and Mel filter bank filtering on the audio to be optimized. The Mel spectrum can be represented in the form of a two-dimensional matrix.

[0062] Optionally, a noise-adding model can be used to add noise to the Mel spectrum to obtain the noise-adding result, including:

[0063] Random numbers are randomly generated within a preset range, and the generated random numbers are used as the number of iterations. Gaussian noise is randomly generated as the real noise.

[0064] The iteration round is initialized to zero. The Mel spectrum is used as the data to be denoised. Based on the real noise, the preset adjustment parameters and the data to be denoised, the denoised data is calculated using the denoising model. The denoised data is used as the data to be denoised, and the iteration round is increased by one.

[0065] Return to the execution step, which calculates the noise-added data using the noise-adding model based on the randomly generated Gaussian noise, preset adjustment parameters, and the data to be noise-added, and uses the noise-added data as the data to be noise-added, incrementing the iteration round by one, until the iteration round and the number of iterations are the same, obtaining the noise-added data for the corresponding iteration round, and determining the noise-added data with the largest iteration round as the noise-adding result.

[0066] Among them, the preset range can refer to the range of values ​​of random numbers, the number of iterations can refer to the number of times noise is added during the noise addition process, and the real noise can refer to the noise data used in the noise addition process.

[0067] The iteration round can refer to the current iteration round, the data to be denoised can refer to the data that needs to be denoised in the current iteration, the adjustment parameter can refer to the hyperparameter in the denoising process, and the data that has been denoised can refer to the data that has been denoised in the current iteration.

[0068] Specifically, with Representing the Mel spectrum, the corresponding noisy data when the iteration round is one. It can be represented as ,in, This can represent the adjustment parameter when the iteration round is one. If we can represent true noise with an iteration number of one, then the iteration number is... At that time, the corresponding noisy data It can be represented as:

[0069]

[0070] in, It can refer to the number of iteration rounds. Adjustment parameters at time, All are preset adjustment parameters. The range of values ​​is , It can refer to the number of iteration rounds. The data to be noisy at that time, i.e., the iteration round is Noisy data at that time It can refer to the number of iteration rounds. Real noise at that time In other words, the actual noise is Gaussian noise. It should be noted that... It is also represented in two-dimensional matrix form, with the same size as the Mel spectrum.

[0071] Combining the reparameter technique, the above equation can be iteratively derived to obtain:

[0072]

[0073] in, , , Representing real noise, this formula can be expressed directly from... Noise addition The process.

[0074] In this embodiment, the audio to be optimized is noise-added through iterative noise addition to obtain latent variables, i.e., the noise-added result, which serves as the input to the subsequent noise estimation model, so as to generate optimized audio based on the noise-added result.

[0075] Optionally, after using the generated random number as the number of iterations, the following is also included:

[0076] A preset mapping table is used to map the number of iterations to adjustment parameters. The mapping table includes the mapping relationship between the number of iterations and the adjustment parameters.

[0077] The mapping table contains several iteration counts and adjustment parameter groups. Each iteration count corresponds to one adjustment parameter group, and each adjustment parameter group contains several adjustment parameters. For example, when the iteration count is 5, the corresponding adjustment parameter group contains... , , , and .

[0078] In this embodiment, the adjustment parameters are stored in a mapping table, which allows the adjustment parameters to be quickly determined based on the number of iterations, thus improving the efficiency of parameter setting.

[0079] The steps described above—obtaining the Mel spectrum of the audio to be optimized, adding noise to the Mel spectrum using a noise-adding model, and obtaining the noise-adding result—provide the noise-adding result as input for subsequent noise estimation. This allows for accurate estimation of noise information based on the noise-adding result, facilitating accurate noise reduction in the subsequent denoising process. It also avoids optimizing audio generation without losing audio features, thereby improving the accuracy of audio optimization.

[0080] Step S204: Use the noise estimation model to calculate the noise in the noise-added result to obtain the predicted noise. Calculate the prediction loss based on the predicted noise, the actual noise corresponding to the noise-added model, and the predicted pitch curve. Update the noise estimation model based on the prediction loss to obtain the updated noise estimation model.

[0081] Among them, the noise estimation model can be used to calculate noise based on the noise addition result, the predicted noise can refer to the estimated noise data, the predicted loss can be used to guide the update of the noise estimation model, and the updated noise estimation model can be used to estimate noise data that is close to the real noise.

[0082] Optionally, a noise estimation model will be used to calculate the noise in the noise-added result to obtain the predicted noise, including:

[0083] Determine the number of iterations corresponding to the noise-added result to obtain the target number of iterations;

[0084] The target number of iterations is mapped through a pre-defined fully connected layer to obtain the mapping result. A noise estimation model is then used to predict the noise in both the mapping result and the noise-added result to obtain the predicted noise.

[0085] The target iteration number can refer to the number of iterations the noise-added result undergoes, and the mapping result can be used to provide iteration number information for the noise estimation process.

[0086] Specifically, the noise estimation model can be a WaveNet model, a U-Net model, etc. The noise estimation model can include several encoders and their corresponding decoders. For any set of encoders and decoders that have a corresponding relationship, the input size of the encoder is consistent with the output size of the decoder, and the output size of the encoder is consistent with the input size of the decoder. The mapping result can be added as embedding information to the input of each encoder and each decoder. The addition method can be concatenation, point-by-point addition, etc.

[0087] In this embodiment, by adding the target iteration number as additional information to the noise estimation model, reference information is provided for the noise estimation process, guiding the noise estimation process to estimate the noise of each noise addition step by step, thereby improving the accuracy of noise estimation.

[0088] Optionally, based on the predicted noise, the actual noise corresponding to the noise-added model, and the predicted pitch curve, the prediction loss is calculated. Based on the prediction loss, the noise estimation model is updated to obtain an updated noise estimation model, including:

[0089] The prediction loss is calculated based on the real noise, the predicted noise, the predicted pitch curve, and the preset prediction loss function.

[0090] Based on the predicted loss, the gradient descent method is used to update the parameters of the noise estimation model until the predicted loss converges, thus obtaining a better noise estimation model.

[0091] Among them, the prediction loss function can be the mean squared error loss function, Euclidean distance or other loss functions used to measure data differences, and the gradient descent method can be stochastic gradient descent or batch gradient descent.

[0092] Specifically, the prediction loss function It can be represented as:

[0093]

[0094] in, It can represent the input noise-added result. , It can represent the predicted pitch curve. It can represent a noise estimation model. These can represent the parameters of the noise estimation model. It can represent real noise.

[0095] In this embodiment, the training of the noise estimation model is supervised by the difference between the real noise and the estimated noise, and the predicted pitch curve is introduced into the noise estimation model as input, so that the noise estimation process can take into account the pitch information, thereby ensuring that the pitch information can be preserved when the predicted noise is used in the subsequent denoising process, and achieving high-quality audio optimization.

[0096] Step S205: Use the updated noise estimation model to calculate the noise in the noise-added result to obtain the reference noise. Then, denoise the noise-added result based on the reference noise to obtain the denoised result. Use the denoised result as the optimized Mel spectrum.

[0097] Here, reference noise can refer to noise data estimated by an updated noise estimation model, denoising result can refer to the result after restoring the noise-added result based on the reference noise, and optimized Mel spectrum can refer to the target output of audio optimization.

[0098] Optionally, the noise-added result is denoised based on the reference noise to obtain the denoised result, including:

[0099] A preset mapping table is used to map the number of iterations to adjustment parameters. The noise-added result is used as the data to be denoised. The number of iterations is initialized to zero. Based on the adjustment parameters, reference noise, noise-added result and preset denoising function, the denoised data is calculated. The denoised data is used as the data to be denoised, and the number of iterations is increased by one.

[0100] Return to the execution step and calculate the denoised data based on the adjustment parameters, reference noise, noise addition result and preset denoising function. Use the denoised data as the data to be denoised and increase the iteration round by one until the iteration round and the number of iterations are the same. Obtain the denoised data of the corresponding iteration round and determine the denoised data with the largest iteration round as the denoising result.

[0101] Among them, the adjustment parameter is a preset value, the mapping table includes the mapping relationship between the number of iterations and the adjustment parameter, the denoising function can be a formula for denoising, the data to be denoised can be the data that needs to be denoised in the current iteration, and the denoised data can be the data that has been denoised in the current iteration.

[0102] Specifically, the denoising function can be expressed as:

[0103]

[0104] in, This can represent the result of adding noise, that is, the data to be denoised when the iteration number is zero. It can represent the denoised data when the iteration round is zero. This can represent the reference noise when the iteration number is zero, which is also the result with added noise. The output of the input noise estimation model. This can represent the adjustment parameter when the iteration number is zero. This could refer to the standard deviation of the reference noise, which is Gaussian noise, and its standard deviation can be obtained directly. This could refer to random noise, ensuring that the final denoising result differs somewhat from the Mel spectrum, thus achieving an optimization rather than reconstruction effect. After iteration, the final recovered [sample / sample] can be obtained. As a result of noise reduction.

[0105] In this embodiment, an iterative process is used for gradual denoising. While preserving pitch information, the Mel spectrum is regenerated to obtain an optimized Mel spectrum. This optimized Mel spectrum is generated by adding and removing noise, which can produce higher quality and more diverse optimized results.

[0106] The above steps involve using an updated noise estimation model to calculate the noise in the noise-added result, obtaining a reference noise, denoising the noise-added result based on the reference noise, and using the denoised result as the optimized Mel spectrum. By using reference noise with pitch information for denoising, the denoising process is ensured to meet pitch requirements, thus improving the audio optimization effect.

[0107] In this embodiment, noise is predicted by combining the predicted pitch curve, so that the predicted noise can make the denoising process meet the pitch requirements. Furthermore, audio optimization is performed by adding and removing noise, resulting in higher audio quality than conventional methods and improving the audio optimization effect.

[0108] Corresponding to the AI-based audio optimization method in the above embodiments, Figure 3 A structural block diagram of an AI-based audio optimization device according to Embodiment 2 of the present invention is shown. This audio optimization device is applied to a server. The computer device corresponding to the server is connected to the client to obtain the spectral envelope and Mel spectrum of the audio to be optimized from the client. The computer device corresponding to the server may be deployed with a trained embedding layer, a trained linear layer, a trained prediction model, a noise-adding model, and a noise estimation model. The trained embedding layer, the trained linear layer, and the trained prediction model can be used to extract features of the audio to be optimized, and to predict pitch based on the extracted features. The noise-adding model and the noise estimation model can be used for audio optimization. For ease of explanation, only the parts relevant to the embodiments of the present invention are shown.

[0109] See Figure 3 The audio optimization device includes:

[0110] The feature extraction module 31 is used to obtain the spectral envelope of the audio to be optimized, perform feature mapping on the spectral envelope using a trained linear layer to obtain envelope features, determine standard audio parameters based on the audio to be optimized, and perform feature embedding on the standard audio parameters through a trained embedding layer to obtain parameter features.

[0111] The pitch prediction module 32 is used to fuse envelope features and parametric features to obtain fused features, and to use the trained prediction model to predict pitch using the fused features to obtain the predicted pitch curve.

[0112] The spectrum noise addition module 33 is used to obtain the Mel spectrum of the audio to be optimized, and to add noise to the Mel spectrum using a noise addition model to obtain the noise addition result;

[0113] The model update module 34 is used to calculate the noise of the noise-added result using the noise estimation model to obtain the predicted noise. Based on the predicted noise, the real noise corresponding to the noise-added model and the predicted pitch curve, the predicted loss is calculated. Based on the predicted loss, the noise estimation model is updated to obtain the updated noise estimation model.

[0114] The spectrum optimization module 35 is used to calculate the noise of the noise-added result using the updated noise estimation model, obtain the reference noise, denoise the noise-added result based on the reference noise, obtain the denoised result, and use the denoised result as the optimized Mel spectrum.

[0115] Optionally, the aforementioned spectrum noise-adding module 33 includes:

[0116] An initialization unit is used to randomly generate random numbers within a preset range, using the generated random numbers as the number of iterations, and randomly generating Gaussian noise as the real noise.

[0117] The initial noise-adding unit is used to initialize the iteration round to zero, using the Mel spectrum as the data to be noise-adding. Based on the real noise, preset adjustment parameters, and the data to be noise-adding, the noise-adding model is used to calculate the noise-adding data, and the noise-adding data is used as the data to be noise-adding, increasing the iteration round by one.

[0118] The iterative noise-adding unit is used to return to the execution step of calculating the noise-adding data using a noise-adding model based on randomly generated Gaussian noise, preset adjustment parameters, and data to be noise-added, using the noise-adding data as the data to be noise-added, and increasing the iteration round by one until the iteration round and the number of iterations are the same, obtaining the noise-adding data of the corresponding iteration round, and determining the noise-adding data with the largest iteration round as the noise-adding result.

[0119] Optionally, the aforementioned spectrum noise-adding module 33 further includes:

[0120] The parameter mapping unit is used to map the number of iterations to adjustment parameters using a preset mapping table. The mapping table includes the mapping relationship between the number of iterations and the adjustment parameters.

[0121] Optionally, the above model update module 34 includes:

[0122] The iteration number determination unit is used to determine the number of iterations corresponding to the noise addition result, thereby obtaining the target number of iterations;

[0123] The noise prediction unit is used to map the target iteration number through a preset fully connected layer to obtain the mapping result. Then, using a noise estimation model, it performs noise prediction on the mapping result and the noise-added result to obtain the predicted noise.

[0124] Optionally, the above model update module 34 includes:

[0125] The loss calculation unit is used to calculate the predicted loss based on the real noise, the predicted noise, the predicted pitch curve, and the preset prediction loss function.

[0126] The parameter update unit is used to update the parameters of the noise estimation model based on the prediction loss using the gradient descent method until the prediction loss converges, thus obtaining the updated noise estimation model.

[0127] Optionally, the aforementioned spectrum optimization module 35 includes:

[0128] The initial denoising unit is used to map the number of iterations to adjustment parameters using a preset mapping table, using the noise addition result as the data to be denoised, initializing the iteration round to zero, calculating the denoised data based on the adjustment parameters, reference noise, noise addition result and preset denoising function, and using the denoised data as the data to be denoised, increasing the iteration round by one;

[0129] The iterative denoising unit is used to return to the step of calculating the denoised data based on the adjustment parameters, reference noise, noise addition result and preset denoising function, using the denoised data as the data to be denoised, and increasing the iteration round by one until the iteration round and the number of iterations are the same, to obtain the denoised data of the corresponding iteration round, and to determine the denoised data with the largest iteration round as the denoising result.

[0130] Optionally, the aforementioned audio optimization device also includes:

[0131] The sample feature extraction module is used to obtain the sample spectral envelope, sample audio parameters, and pitch labels of the sample audio. It uses a linear layer to perform feature mapping on the sample spectral envelope to obtain the sample envelope features, and uses an embedding layer to embed the sample audio parameters to obtain the sample parameter features.

[0132] The sample pitch prediction module is used to fuse sample envelope features and sample parameter features to obtain sample fusion features. The prediction model is then used to predict the pitch of the sample fusion features to obtain the sample pitch curve.

[0133] The model training module is used to train the embedding layer, linear layer, and prediction model based on the sample pitch curves and pitch labels, resulting in a trained embedding layer, a trained linear layer, and a trained prediction model.

[0134] It should be noted that the information interaction and execution process between the above modules and units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0135] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 4As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the diagram), a memory, and a computer program stored in the memory and executable on at least one processor, which, when executed by the processor, implements the steps in any of the above-described audio optimization method embodiments.

[0136] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0137] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0138] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of the computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0139] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0140] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0141] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0142] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0143] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0144] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0145] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An audio optimization method based on artificial intelligence, characterized in that, The audio optimization method includes: Obtain the spectral envelope of the audio to be optimized, perform feature mapping on the spectral envelope using a trained linear layer to obtain envelope features, determine standard audio parameters based on the audio to be optimized, and perform feature embedding on the standard audio parameters using a trained embedding layer to obtain parameter features; The envelope features and the parameter features are fused to obtain fused features. The trained prediction model is then used to predict the pitch of the fused features to obtain the predicted pitch curve. Obtain the Mel spectrum of the audio to be optimized, and add noise to the Mel spectrum using a noise model to obtain the noise-added result; The noise estimation model is used to calculate the noise in the noise-added result to obtain the predicted noise. Based on the predicted noise, the real noise corresponding to the noise-added model and the predicted pitch curve, the prediction loss is calculated. Based on the prediction loss, the noise estimation model is updated to obtain the updated noise estimation model. The noise is calculated on the noise-added result using the updated noise estimation model to obtain reference noise. The noise-added result is then denoised based on the reference noise to obtain a denoised result, which is used as the optimized Mel spectrum.

2. The audio optimization method according to claim 1, characterized in that, The step of adding noise to the Mel spectrum using a noise-adding model to obtain the noise-adding result includes: Random numbers are randomly generated within a preset range, and the generated random numbers are used as the number of iterations. Gaussian noise is randomly generated as the real noise. The iteration round is initialized to zero. The Mel spectrum is used as the data to be denoised. Based on the real noise, the preset adjustment parameters and the data to be denoised, the denoised data is calculated using the denoising model. The denoised data is used as the data to be denoised, and the iteration round is incremented by one. Return to the execution step of calculating the noise-added data using the noise-adding model based on the randomly generated Gaussian noise, preset adjustment parameters, and the data to be noise-added, using the noise-added data as the data to be noise-added, and increasing the iteration round by one until the iteration round and the number of iterations are the same, to obtain the noise-added data of the corresponding iteration round, and determine the noise-added data with the largest iteration round as the noise-adding result.

3. The audio optimization method according to claim 2, characterized in that, After using the generated random number as the number of iterations, the method further includes: A preset mapping table is used to map the number of iterations to the adjustment parameter, and the mapping table includes the mapping relationship between the number of iterations and the adjustment parameter.

4. The audio optimization method according to claim 1, characterized in that, The step of using a noise estimation model to calculate the noise in the noise-added result to obtain the predicted noise includes: The number of iterations corresponding to the noise-adding result is determined to obtain the target number of iterations; The target number of iterations is mapped through a preset fully connected layer to obtain a mapping result. The noise estimation model is then used to predict the noise from the mapping result and the noise-added result to obtain the predicted noise.

5. The audio optimization method according to claim 1, characterized in that, The step of calculating the prediction loss based on the predicted noise, the actual noise corresponding to the noise-added model, and the predicted pitch curve, and updating the noise estimation model based on the prediction loss to obtain an updated noise estimation model includes: The prediction loss is calculated based on the actual noise, the predicted noise, the predicted pitch curve, and the preset prediction loss function. Based on the predicted loss, the parameters of the noise estimation model are updated using the gradient descent method until the predicted loss converges, thus obtaining the updated noise estimation model.

6. The audio optimization method according to claim 2, characterized in that, The step of denoising the noise result based on the reference noise to obtain the denoised result includes: The iteration number is mapped to the adjustment parameter using a preset mapping table. The noise addition result is used as the data to be denoised. The iteration round is initialized to zero. The denoised data is calculated based on the adjustment parameter, the reference noise, the noise addition result and the preset denoising function. The denoised data is used as the data to be denoised, and the iteration round is increased by one. Return to the step of calculating the denoised data based on the adjustment parameters, the reference noise, the noise addition result, and the preset denoising function, using the denoised data as the data to be denoised, and increasing the iteration round by one until the iteration round and the number of iterations are the same, to obtain the denoised data of the corresponding iteration round, and determine the denoised data with the largest iteration round as the denoising result.

7. The audio optimization method according to any one of claims 1 to 6, characterized in that, The training process for the embedding layer, the linear layer, and the prediction model includes: The sample audio spectrum envelope, sample audio parameters, and pitch labels are obtained. The linear layer is used to perform feature mapping on the sample spectrum envelope to obtain sample envelope features. The embedding layer is used to perform feature embedding on the sample audio parameters to obtain sample parameter features. The sample envelope features and the sample parameter features are fused to obtain sample fusion features. The prediction model is then used to predict the pitch of the sample fusion features to obtain the sample pitch curve. Based on the sample pitch curve and the pitch label, the embedding layer, the linear layer, and the prediction model are trained to obtain the trained embedding layer, the trained linear layer, and the trained prediction model.

8. An audio optimization device based on artificial intelligence, characterized in that, The audio optimization device includes: The feature extraction module is used to obtain the spectral envelope of the audio to be optimized, perform feature mapping on the spectral envelope using a trained linear layer to obtain envelope features, determine standard audio parameters based on the audio to be optimized, and perform feature embedding on the standard audio parameters through a trained embedding layer to obtain parameter features. The pitch prediction module is used to fuse the envelope features and the parameter features to obtain fused features, and use a trained prediction model to predict the pitch of the fused features to obtain a predicted pitch curve. The spectrum noise-adding module is used to acquire the Mel spectrum of the audio to be optimized, and to add noise to the Mel spectrum using a noise-adding model to obtain the noise-adding result; The model update module is used to perform noise calculation on the noise-added result using a noise estimation model to obtain predicted noise, calculate the prediction loss based on the predicted noise, the real noise corresponding to the noise-added model and the predicted pitch curve, and update the noise estimation model based on the prediction loss to obtain an updated noise estimation model. The spectrum optimization module is used to perform noise calculation on the noise-added result using the updated noise estimation model to obtain reference noise, and to denoise the noise-added result based on the reference noise to obtain a denoised result, which is then used as the optimized Mel spectrum.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the audio optimization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the audio optimization method as described in any one of claims 1 to 7.

Images