Audio optimization method and device based on artificial intelligence, computer equipment and medium

Abstract

The invention relates to the technical field of audio processing, in particular to an audio optimization method and device based on artificial intelligence, computer equipment and a medium. According to the method, a linear layer is used for mapping a spectrum envelope of a to-be-optimized audio to obtain an envelope feature, an embedding layer is used for embedding a standard audio parameter into a parameter feature, a prediction model is used for predicting a fusion feature of the envelope feature and the parameter feature to obtain a prediction pitch curve, and a noise adding model is used for adding noise to a Mel spectrum of the to-be-optimized audio to obtain a predicted pitch curve. Calculating the noise of the noise addition result by using a noise estimation model to obtain predicted noise, updating the noise estimation model according to the predicted noise, the real noise and the predicted pitch curve, calculating the reference noise of the noise addition result by using the updated noise estimation model, and de-noising the noise addition result according to the reference noise. The noise estimation model is optimized in combination with pitch information, so that the denoising process meets the pitch requirement, and the audio optimization effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of audio processing technology, and in particular to an audio optimization method, device, computer equipment and medium based on artificial intelligence. Background Art

[0002] The audio optimization task is to optimize the pitch curve of the audio to be optimized so that the optimized pitch curve can meet the preset pitch standard. The audio optimization task can be applied to various application scenarios such as intelligent tuning, audio enhancement, and speech synthesis. For example, in the intelligent tuning scenario, the audio to be optimized that is out of tune can be optimized to obtain optimized audio that is more in line with the musical score.

[0003] However, existing audio optimization tasks are usually implemented using generative models, and generative models require both poor-quality and good-quality audio of the same person to ensure the accuracy of audio optimization. However, in actual application scenarios, only poor-quality audio can be obtained, resulting in generative models only being able to further improve the quality of audio to be optimized with good audio quality, while it is difficult to obtain good optimization results for audio to be optimized with poor audio quality. The generalization of audio optimization is poor, that is, the effect of audio optimization is poor. Therefore, how to improve the accuracy of audio optimization has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide an audio optimization method, apparatus, computer equipment and medium based on artificial intelligence to solve the problem of poor audio optimization effect.

[0005] In a first aspect, an embodiment of the present invention provides an audio optimization method based on artificial intelligence, the audio optimization method comprising: Obtaining the spectrum envelope of the audio to be optimized, performing feature mapping on the spectrum envelope using a trained linear layer to obtain envelope features, determining standard audio parameters according to the audio to be optimized, and performing feature embedding on the standard audio parameters using a trained embedding layer to obtain parameter features; The envelope feature and the parameter feature are fused to obtain a fused feature, and a trained prediction model is used to perform pitch prediction on the fused feature to obtain a predicted pitch curve; Obtaining the Mel spectrum of the audio to be optimized, and adding noise to the Mel spectrum using a noise adding model to obtain a noise adding result; Use a noise estimation model to perform noise calculation on the noise addition result to obtain predicted noise, calculate a predicted loss based on the predicted noise, the real noise corresponding to the noise addition model and the predicted pitch curve, and update the noise estimation model based on the predicted loss to obtain an updated noise estimation model; The updated noise estimation model is used to perform noise calculation on the noisy result to obtain a reference noise, and the noisy result is denoised according to the reference noise to obtain a denoised result, and the denoised result is used as the optimized Mel spectrum.

[0006] In a second aspect, an embodiment of the present invention provides an audio optimization device based on artificial intelligence, the audio optimization device comprising: A feature extraction module is used to obtain the spectrum envelope of the audio to be optimized, perform feature mapping on the spectrum envelope using a trained linear layer to obtain envelope features, determine standard audio parameters based on the audio to be optimized, and perform feature embedding on the standard audio parameters using a trained embedding layer to obtain parameter features; A pitch prediction module, used for fusing the envelope feature and the parameter feature to obtain a fused feature, and using a trained prediction model to perform pitch prediction on the fused feature to obtain a predicted pitch curve; A spectrum noise adding module, used for obtaining the Mel spectrum of the audio to be optimized, and adding noise to the Mel spectrum using a noise adding model to obtain a noise adding result; A model updating module, configured to use a noise estimation model to perform noise calculation on the noise addition result to obtain predicted noise, calculate a predicted loss based on the predicted noise, the real noise corresponding to the noise addition model and the predicted pitch curve, and update the noise estimation model based on the predicted loss to obtain an updated noise estimation model; The spectrum optimization module is used to use the updated noise estimation model to perform noise calculation on the noisy result to obtain a reference noise, denoise the noisy result according to the reference noise to obtain a denoised result, and use the denoised result as the optimized Mel spectrum.

[0007] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the audio optimization method as described in the first aspect when executing the computer program.

[0008] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the audio optimization method as described in the first aspect is implemented.

[0009] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: Get the spectrum envelope of the audio to be optimized, use the trained linear layer to perform feature mapping on the spectrum envelope to obtain envelope features, determine standard audio parameters according to the audio to be optimized, embed features of the standard audio parameters through the trained embedding layer to obtain parameter features, fuse the envelope features and parameter features to obtain fused features, use the trained prediction model to predict the pitch of the fused features to obtain a predicted pitch curve, get the Mel spectrum of the audio to be optimized, use the noise addition model to add noise to the Mel spectrum to obtain a noise addition result, use the noise estimation model to calculate the noise on the noise addition result to obtain predicted noise, and calculate the predicted noise based on the predicted noise and noise addition. The real noise and predicted pitch curve corresponding to the model are used to calculate the prediction loss, and the noise estimation model is updated according to the prediction loss to obtain an updated noise estimation model. The updated noise estimation model is used to calculate the noise of the noisy result to obtain the reference noise, and the noisy result is denoised according to the reference noise to obtain the denoised result. The denoised result is used as the optimized Mel spectrum and combined with the predicted pitch curve to predict the noise, so that the predicted noise can make the denoising process meet the pitch requirements, and audio optimization is performed by adding noise and denoising, so that the audio optimization result has higher audio quality than the conventional method, thereby improving the effect of audio optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0011] Figure 1 This is a schematic diagram of an application environment of an audio optimization method based on artificial intelligence provided in Embodiment 1 of the present invention; Figure 2 It is a flowchart of an audio optimization method based on artificial intelligence provided in Embodiment 1 of the present invention; Figure 3 is a structural diagram of an audio optimization device based on artificial intelligence provided in Embodiment 2 of the present invention; Figure 4 It is a structural diagram of a computer device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0012] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0013] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0014] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0015] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0016] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0017] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0018] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0019] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0020] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0021] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0022] The audio optimization method based on artificial intelligence provided by the first embodiment of the present invention can be applied to Figure 1 In the application environment, the client communicates with the server. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, personal digital assistants (PDAs) and other computer devices. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0023] The server can be deployed in various application scenarios such as intelligent audio editing, audio enhancement, and speech synthesis. Taking the intelligent audio editing scenario as an example, the user enters the audio to be optimized through the client. At this time, the audio to be optimized is usually singing audio. The intelligent audio editing task is to beautify the singing audio so that the beautified singing audio is more pleasant and professional while retaining the timbre of the singing audio.

[0024] See also Figure 2, is a flowchart of an audio optimization method based on artificial intelligence provided by Embodiment 1 of the present invention. The above audio optimization method can be applied to Figure 1 The server in the embodiment of the present invention is connected to the client to obtain the spectrum envelope and Mel spectrum of the audio to be optimized from the client. The computer device corresponding to the server can be deployed with a trained embedding layer, a trained linear layer, a trained prediction model, a noise addition model and a noise estimation model. The trained embedding layer, the trained linear layer and the trained prediction model can be used to extract the features of the audio to be optimized, and perform pitch prediction based on the extracted features. The noise addition model and the noise estimation model can be used to perform audio optimization. Figure 2 As shown, the audio optimization method may include the following steps: Step S201, obtain the spectrum envelope of the audio to be optimized, use the trained linear layer to perform feature mapping on the spectrum envelope to obtain envelope features, determine standard audio parameters according to the audio to be optimized, and perform feature embedding on the standard audio parameters through the trained embedding layer to obtain parameter features.

[0025] Among them, the audio to be optimized may refer to the audio that needs to be optimized and input by the user through the client, the spectrum envelope may represent the envelope feature information of the audio to be optimized, the standard audio parameters may refer to MIDI, MIDI may refer to the standardized code representing audio parameters, and the audio parameters may include pitch, dynamics, rhythm, etc. The standard audio parameters may refer to the audio parameters under the standard pitch corresponding to the audio to be optimized, that is, the audio parameters under the professional pitch corresponding to the audio to be optimized.

[0026] The trained linear layer can be used to map the spectrum envelope into envelope features, which can be used to characterize the characteristic information of the spectrum envelope. The trained embedding layer can be used to embed the standard audio parameters into parameter features, which can be used to characterize the characteristics of the standard audio parameters.

[0027] Specifically, the spectrum envelope may be extracted by using an audio information extraction tool, such as the pyworld library in Python, and the function corresponding to the spectrum envelope extraction may be the pw.cheaptrick() function.

[0028] The standard audio parameters can be obtained through open source software such as Tony and Melodyne. After the spectrum envelope passes through the trained linear layer, it needs to be processed by an activation function to obtain the envelope features. In this embodiment, the activation function can use the Relu function.

[0029] The envelope features and parameter features are both expressed in matrix form, and the matrix corresponding to the envelope feature has the same size as the matrix corresponding to the reference feature, so as to facilitate the subsequent fusion of the envelope features and the parameter features.

[0030] The above steps of obtaining the spectral envelope of the audio to be optimized, performing feature mapping on the spectral envelope using a trained linear layer to obtain envelope features, determining standard audio parameters according to the audio to be optimized, and performing feature embedding on the standard audio parameters through a trained embedding layer to obtain parameter features can fully extract the audio feature information of the audio to be optimized, so that subsequent pitch prediction can obtain rich feature input, thereby improving the accuracy of pitch prediction.

[0031] Step S202: The envelope feature and the parameter feature are fused to obtain a fused feature, and the trained prediction model is used to perform pitch prediction on the fused feature to obtain a predicted pitch curve.

[0032] The fused features may include envelope features and parameter features, the trained prediction model may be used to predict pitch based on the fused features, and the predicted pitch curve may be used to characterize the pitch information of the audio to be optimized.

[0033] Specifically, the fusion can be performed in a concatenated manner. Since the matrix corresponding to the envelope feature and the matrix corresponding to the reference feature have the same size, they can be directly concatenated to obtain the fused feature.

[0034] The trained prediction model may be composed of an encoder structure of a Transformer model and a preset linear prediction layer. The encoder structure of the Transformer model may include four encoding layers, and the preset linear prediction layer may refer to a fully connected layer.

[0035] Optionally, the training process of the embedding layer, linear layer, and prediction model includes: Obtain the sample spectrum envelope, sample audio parameters and pitch label of the sample audio, use the linear layer to perform feature mapping on the sample spectrum envelope to obtain the sample envelope feature, and use the embedding layer to perform feature embedding on the sample audio parameters to obtain the sample parameter feature; The sample envelope feature and the sample parameter feature are fused to obtain the sample fusion feature, and the prediction model is used to predict the pitch of the sample fusion feature to obtain the sample pitch curve; According to the sample pitch curve and the pitch label, the embedding layer, the linear layer and the prediction model are trained to obtain the trained embedding layer, the trained linear layer and the trained prediction model.

[0036] Among them, the sample audio may refer to the audio used to construct a training data set, the training data set includes sample spectrum envelopes and sample audio parameters corresponding to several sample audios, the sample spectrum envelope and sample audio parameters corresponding to a single sample audio are used as a training sample, and the pitch label corresponding to the sample audio is used as the label data of the training sample corresponding to the sample audio.

[0037] The sample envelope feature can be used to characterize the feature information of the sample spectrum envelope, the sample parameter feature can be used to characterize the feature information of the sample audio parameter, and the sample fusion feature can include the feature information of the sample envelope feature and the sample parameter feature.

[0038] Specifically, the sample audio may refer to audio data provided by non-professional users. Accordingly, the pitch label of the sample audio may be non-professional pitch information, that is, the pitch information corresponding to the sample audio itself. The difference loss is calculated through the difference loss function according to the sample pitch curve and the pitch label. Based on the difference loss, the embedding layer, the linear layer and the prediction model are trained until the difference loss converges to obtain a trained embedding layer, a trained linear layer and a trained prediction model. The difference loss function in this embodiment may use a mean square error loss function.

[0039] The trained embedding layer, the trained linear layer and the trained prediction model can input professional audio parameters and non-professional envelope spectra during the inference stage. The professional audio parameters are also standard audio parameters, and output professional pitch curves. The pitch curve can be represented in a two-dimensional matrix.

[0040] In this embodiment, the embedding layer, linear layer and prediction model are jointly trained, so that the model can learn the correlation information between the spectral envelope and pitch, and then predict the professional pitch curve based on professional audio parameters and non-professional spectral envelope in the inference stage. Compared with the existing method that requires the non-professional audio and professional audio of the same person to be trained, the difficulty of constructing the data set is greatly reduced, and the training can be completed only by relying on non-professional audio, which improves the generalization and practicality of the embedding layer, linear layer and prediction model.

[0041] Step S203, obtaining the Mel spectrum of the audio to be optimized, and adding noise to the Mel spectrum using a noise adding model to obtain a noise adding result.

[0042] The Mel spectrum may be used to characterize audio features of the audio to be optimized, the noise adding model may be used to add Gaussian noise to the Mel spectrum, and the noise adding result may refer to the Mel spectrum with Gaussian noise added.

[0043] Specifically, the Mel spectrum can be obtained by sequentially performing pre-emphasis, framing, windowing, Fourier transform and Mel filter bank filtering on the audio to be optimized. The Mel spectrum can be represented in the form of a two-dimensional matrix.

[0044] Optionally, the mel spectrum is denoised using a denoising model to obtain a denoising result, including: Randomly generate a random number within a preset range, use the generated random number as the number of iterations, and randomly generate Gaussian noise as the real noise; Initialize the iteration round to zero, use the Mel spectrum as the data to be noisy, use the noisy model to calculate the noisy data according to the real noise, the preset adjustment parameters and the data to be noisy, use the noisy data as the data to be noisy, and increase the iteration round by one; Return to execute the step of calculating the noisy data using the noisy model according to the randomly generated Gaussian noise, the preset adjustment parameters and the data to be noisy, and use the noisy data as the data to be noisy, increase the iteration round by one, until the iteration round and the number of iterations are the same, obtain the noisy data of the corresponding iteration round, and determine the noisy data with the largest iteration round as the noisy result.

[0045] The preset range may refer to the value range of the random number, the number of iterations may refer to the number of noise additions in the noise addition process, and the real noise may refer to the noise data used in the noise addition process.

[0046] The iteration round may refer to the round of the current iteration process, the data to be denoised may refer to the data that needs to be denoised in the current iteration process, the adjustment parameter may refer to the hyperparameter in the denoising process, and the noisy data may refer to the data that has been denoised in the current iteration process.

[0047] Specifically, represents the Mel spectrum, then when the iteration round is one, the corresponding noised data It can be expressed as ,in, It can represent the adjustment parameter when the iteration round is one, can represent the real noise with one iteration round, then the iteration round is When the corresponding noise data It can be expressed as: in, It can be referred to as the iteration round The adjustment parameters are These are all preset adjustment parameters. The value range is , It can be referred to as the iteration round The data to be noised at the time, that is, the iteration round is The noisy data at time , It can be referred to as the iteration round The real noise when , that is, the real noise belongs to Gaussian noise. It should be noted that It is also represented in a two-dimensional matrix, which is consistent with the size of the Mel spectrum.

[0048] Combining the heavy parameter technique, iterative derivation from the above formula yields: in, , , represents the real noise, which can be expressed directly from Add noise to get process.

[0049] In this embodiment, the audio to be optimized is denoised by iterative denoising to obtain a latent variable, that is, a denoising result, which is used as an input of a subsequent noise estimation model to facilitate generation of optimized audio based on the denoising result.

[0050] Optionally, after using the generated random number as the number of iterations, it also includes: The number of iterations is mapped to the adjustment parameters using a preset mapping table, and the mapping table includes a mapping relationship between the number of iterations and the adjustment parameters.

[0051] The mapping table includes several iteration times and adjustment parameter groups. One iteration number corresponds to one adjustment parameter group, and one adjustment parameter group includes several adjustment parameters. For example, when the iteration number is 5, the corresponding adjustment parameter group includes , , , and .

[0052] In this embodiment, the adjustment parameters are stored in a mapping table, so that the adjustment parameters can be quickly determined according to the number of iterations, thereby improving the efficiency of parameter setting.

[0053] The above steps of obtaining the Mel spectrum of the audio to be optimized, using the denoising model to denoise the Mel spectrum, and obtaining the denoising result can provide the input denoising result for the subsequent noise estimation, so that the noise information can be accurately estimated according to the denoising result, which is convenient for the subsequent denoising process to obtain accurate denoising results, avoid optimizing audio generation when the audio features are lost, and improve the accuracy of audio optimization.

[0054] Step S204, use the noise estimation model to perform noise calculation on the noisy result to obtain predicted noise, calculate the predicted loss based on the predicted noise, the real noise corresponding to the noisy model and the predicted pitch curve, and update the noise estimation model based on the predicted loss to obtain an updated noise estimation model.

[0055] Among them, the noise estimation model can be used to calculate noise based on the noise addition result, the predicted noise can refer to the estimated noise data, the predicted loss can be used to guide the update of the noise estimation model, and the updated noise estimation model can be used to estimate noise data close to the actual noise.

[0056] Optionally, a noise estimation model is used to perform noise calculation on the noise-added result to obtain predicted noise, including: Determine the number of iterations corresponding to the noise addition result and obtain the target number of iterations; The target number of iterations is mapped through a preset fully connected layer to obtain a mapping result. The noise estimation model is used to perform noise prediction on the mapping result and the noise addition result to obtain the predicted noise.

[0057] The target number of iterations may refer to the number of iterations experienced by the noise addition result, and the mapping result may be used to provide iteration number information for the noise estimation process.

[0058] Specifically, the noise estimation model can select the wavenet model, the U-Net model, etc. The noise estimation model can include several encoders and their corresponding decoders. For any group of encoders and decoders that have a corresponding relationship, the input size of the encoder is consistent with the output size of the decoder, and the output size of the encoder is consistent with the input size of the decoder. The mapping result can be added to the input of each encoder and each decoder as embedded information, and the adding method can be concatenation, point-by-point addition, etc.

[0059] In this embodiment, by adding the target number of iterations as additional information to the noise estimation model, reference information is provided for the noise estimation process, and the noise estimation process is guided to gradually estimate the noise of each noise addition process, thereby improving the accuracy of noise estimation.

[0060] Optionally, the prediction loss is calculated according to the predicted noise, the real noise corresponding to the noise-added model, and the predicted pitch curve, and the noise estimation model is updated according to the predicted loss to obtain an updated noise estimation model, including: The prediction loss is calculated according to the real noise, the predicted noise, the predicted pitch curve and the preset prediction loss function; Based on the prediction loss, the gradient descent method is used to update the parameters of the noise estimation model until the prediction loss converges to obtain an updated noise estimation model.

[0061] Among them, the prediction loss function can adopt the mean square error loss function, Euclidean distance and other loss functions used to measure data differences, and the gradient descent method can adopt the stochastic gradient descent method, the batch gradient descent method and the like.

[0062] Specifically, the prediction loss function It can be expressed as: in, It can represent the noise result of the input , can represent the predicted pitch curve, The noise estimation model can be expressed as can represent the parameters of the noise estimation model, can represent real noise.

[0063] In this embodiment, the training of the noise estimation model is supervised by the difference between the real noise and the estimated noise, and the predicted pitch curve is introduced as input into the noise estimation model, so that the noise estimation process can take the pitch information into account, thereby ensuring that the subsequent denoising process can retain the pitch information when using the predicted noise, thereby achieving high-quality audio optimization.

[0064] Step S205, using the updated noise estimation model to perform noise calculation on the noisy result to obtain reference noise, denoising the noisy result according to the reference noise to obtain a denoised result, and using the denoised result as the optimized Mel spectrum.

[0065] The reference noise may refer to noise data estimated by an updated noise estimation model, the denoising result may refer to the result after restoring the denoising result according to the reference noise, and the optimized Mel spectrum may refer to the target output of audio optimization.

[0066] Optionally, the noise-added result is denoised according to the reference noise to obtain a denoised result, including: The number of iterations is mapped to the adjustment parameters using a preset mapping table, and the denoising result is used as the data to be denoised. The number of iterations is initialized to zero. The denoised data is calculated based on the adjustment parameters, reference noise, denoising result and the preset denoising function. The denoised data is used as the data to be denoised, and the number of iterations is increased by one. Return to execute the denoised data calculated according to the adjustment parameters, reference noise, denoising result and preset denoising function, use the denoised data as the data to be denoised, increase the iteration round by one, until the iteration round and the number of iterations are the same, obtain the denoised data of the corresponding iteration round, and determine the denoised data with the largest iteration round as the denoising result.

[0067] Among them, the adjustment parameter is a preset value, the mapping table includes a mapping relationship between the number of iterations and the adjustment parameter, the denoising function can refer to the formula for denoising processing, the data to be denoised can refer to the data that needs to be denoised in the current iteration round, and the denoised data can refer to the data that has been denoised in the current iteration round.

[0068] Specifically, the denoising function can be expressed as: in, It can represent the noise addition result, that is, the data to be denoised when the iteration round is zero. can represent the denoised data when the iteration round is zero, It can represent the reference noise when the iteration round is zero, that is, the noise result Output result after inputting the noise estimation model, It can represent the adjustment parameter when the iteration round is zero, It can refer to the standard deviation corresponding to the reference noise. The reference noise is Gaussian noise, and its standard deviation can be obtained directly. It can refer to random noise to ensure that there is a certain difference between the final denoising result and the Mel spectrum, so as to achieve the effect of optimization rather than reconstruction. After the iteration is completed, the final restored As the denoising result.

[0069] In this embodiment, an iterative process is used for step-by-step denoising. While retaining the pitch information, the Mel spectrum is regenerated to obtain an optimized Mel spectrum, which is generated in a noise-adding and denoising manner, and can generate an optimization result with higher quality and more diverse styles.

[0070] In the above step, the updated noise estimation model is used to calculate the noise of the noisy result to obtain the reference noise, the noisy result is denoised according to the reference noise to obtain the denoised result, the denoised result is used as the optimized Mel spectrum, and the reference noise that introduces the pitch information is used for denoising, so as to ensure that the denoising process meets the pitch requirements and improve the effect of audio optimization.

[0071] In this embodiment, the noise is predicted in combination with the predicted pitch curve, so that the predicted noise can make the denoising process meet the pitch requirements, and the audio is optimized by adding noise and denoising, so that the audio optimization result has higher audio quality than the conventional method, thereby improving the effect of audio optimization.

[0072] Corresponding to the audio optimization method based on artificial intelligence in the above embodiment, Figure 3 The structural block diagram of the audio optimization device based on artificial intelligence provided by the second embodiment of the present invention is shown. The above-mentioned audio optimization device is applied to the server. The computer device corresponding to the server is connected to the client to obtain the spectrum envelope and Mel spectrum of the audio to be optimized from the client. The computer device corresponding to the server can be deployed with a trained embedding layer, a trained linear layer, a trained prediction model, a noise addition model and a noise estimation model. The trained embedding layer, the trained linear layer and the trained prediction model can be used to extract the features of the audio to be optimized, and perform pitch prediction based on the extracted features. The noise addition model and the noise estimation model can be used to perform audio optimization. For ease of explanation, only the parts related to the embodiment of the present invention are shown.

[0073] See also Figure 3 , the audio optimization device comprises: The feature extraction module 31 is used to obtain the spectrum envelope of the audio to be optimized, perform feature mapping on the spectrum envelope using the trained linear layer to obtain envelope features, determine standard audio parameters according to the audio to be optimized, and perform feature embedding on the standard audio parameters using the trained embedding layer to obtain parameter features; A pitch prediction module 32 is used to fuse the envelope feature and the parameter feature to obtain a fused feature, and use the trained prediction model to perform pitch prediction on the fused feature to obtain a predicted pitch curve; The spectrum noise adding module 33 is used to obtain the Mel spectrum of the audio to be optimized, and to add noise to the Mel spectrum using the noise adding model to obtain the noise adding result; The model updating module 34 is used to perform noise calculation on the noise addition result using the noise estimation model to obtain the predicted noise, calculate the predicted loss according to the predicted noise, the real noise corresponding to the noise addition model and the predicted pitch curve, and update the noise estimation model according to the predicted loss to obtain an updated noise estimation model; The spectrum optimization module 35 is used to use the updated noise estimation model to perform noise calculation on the noisy result to obtain reference noise, denoise the noisy result according to the reference noise to obtain a denoised result, and use the denoised result as the optimized Mel spectrum.

[0074] Optionally, the spectrum noise adding module 33 includes: An initialization unit, used to randomly generate a random number within a preset range, use the generated random number as the number of iterations, and randomly generate Gaussian noise as real noise; The initial noise adding unit is used to initialize the iteration round to zero, use the Mel spectrum as the data to be noised, calculate the noisy data using the noise adding model according to the real noise, the preset adjustment parameters and the data to be noised, use the noisy data as the data to be noised, and increase the iteration round by one; The iterative noise adding unit is used to return to the execution of the step of calculating the noisy data using the noise adding model according to the randomly generated Gaussian noise, the preset adjustment parameters and the data to be noisy, using the noisy data as the data to be noisy, increasing the iteration round by one until the iteration round and the number of iterations are the same, obtaining the noisy data of the corresponding iteration round, and determining the noisy data with the largest iteration round as the noisy result.

[0075] Optionally, the spectrum noise adding module 33 further includes: The parameter mapping unit is used to map the number of iterations to the adjustment parameters using a preset mapping table, wherein the mapping table includes a mapping relationship between the number of iterations and the adjustment parameters.

[0076] Optionally, the model updating module 34 includes: A number determination unit is used to determine the number of iterations corresponding to the noise addition result to obtain a target number of iterations; The noise prediction unit is used to map the target number of iterations through a preset fully connected layer to obtain a mapping result, and use a noise estimation model to perform noise prediction on the mapping result and the noise addition result to obtain a predicted noise.

[0077] Optionally, the model updating module 34 includes: A loss calculation unit, used for calculating the predicted loss according to the real noise, the predicted noise, the predicted pitch curve and the preset predicted loss function; The parameter updating unit is used to update the parameters of the noise estimation model based on the prediction loss by using the gradient descent method until the prediction loss converges to obtain an updated noise estimation model.

[0078] Optionally, the spectrum optimization module 35 includes: An initial denoising unit is used to map the number of iterations to an adjustment parameter using a preset mapping table, use the denoising result as the data to be denoised, initialize the iteration round to zero, calculate the denoised data according to the adjustment parameter, reference noise, the denoising result and a preset denoising function, use the denoised data as the data to be denoised, and increase the iteration round by one; The iterative denoising unit is used to return to execute the denoising function according to the adjustment parameters, reference noise, denoising results and preset denoising functions, calculate the denoised data, use the denoised data as the data to be denoised, increase the iteration round by one, until the iteration round and the number of iterations are the same, obtain the denoised data of the corresponding iteration round, and determine the denoised data with the largest iteration round as the denoising result.

[0079] Optionally, the audio optimization device further includes: A sample feature extraction module is used to obtain the sample spectrum envelope, sample audio parameters and pitch labels of the sample audio, use a linear layer to perform feature mapping on the sample spectrum envelope to obtain sample envelope features, and use an embedding layer to perform feature embedding on the sample audio parameters to obtain sample parameter features; The sample pitch prediction module is used to fuse the sample envelope feature and the sample parameter feature to obtain the sample fusion feature, and use the prediction model to predict the pitch of the sample fusion feature to obtain the sample pitch curve; The model training module is used to train the embedding layer, the linear layer and the prediction model according to the sample pitch curve and the pitch label to obtain the trained embedding layer, the trained linear layer and the trained prediction model.

[0080] It should be noted that the information interaction, execution process, etc. between the above-mentioned modules and units are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0081] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned audio optimization method embodiments are implemented.

[0082] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 4 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0083] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0084] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0085] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0086] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing.

[0087] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0088] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0089] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0090] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0091] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. An audio optimization method based on artificial intelligence, characterized in that: The audio optimization method comprises: Obtaining the spectrum envelope of the audio to be optimized, performing feature mapping on the spectrum envelope using a trained linear layer to obtain envelope features, determining standard audio parameters according to the audio to be optimized, and performing feature embedding on the standard audio parameters using a trained embedding layer to obtain parameter features; The envelope feature and the parameter feature are fused to obtain a fused feature, and a trained prediction model is used to perform pitch prediction on the fused feature to obtain a predicted pitch curve; Obtaining the Mel spectrum of the audio to be optimized, and adding noise to the Mel spectrum using a noise adding model to obtain a noise adding result; Use a noise estimation model to perform noise calculation on the noise addition result to obtain predicted noise, calculate a predicted loss based on the predicted noise, the real noise corresponding to the noise addition model and the predicted pitch curve, and update the noise estimation model based on the predicted loss to obtain an updated noise estimation model; The updated noise estimation model is used to perform noise calculation on the noisy result to obtain a reference noise, and the noisy result is denoised according to the reference noise to obtain a denoised result, and the denoised result is used as the optimized Mel spectrum.

2. The audio optimization method according to claim 1, characterized in that: The step of adding noise to the Mel spectrum using the noise adding model to obtain a noise adding result includes: Randomly generate a random number within a preset range, use the generated random number as the number of iterations, and randomly generate Gaussian noise as the real noise; Initialize the iteration round to zero, use the Mel spectrum as the data to be noisy, calculate the noisy data using the noisy model according to the real noise, the preset adjustment parameters and the data to be noisy, use the noisy data as the data to be noisy, and increase the iteration round by one; Return to executing the step of calculating the noisy data using the noisy model according to the randomly generated Gaussian noise, the preset adjustment parameters and the data to be noisy, taking the noisy data as the data to be noisy, increasing the iteration round by one until the iteration round and the number of iterations are the same, obtaining the noisy data of the corresponding iteration round, and determining the noisy data with the largest iteration round as the noisy result.

3. The audio optimization method according to claim 2, characterized in that: After taking the generated random number as the number of iterations, the method further comprises: The number of iterations is mapped to the adjustment parameter using a preset mapping table, wherein the mapping table includes a mapping relationship between the number of iterations and the adjustment parameter.

4. The audio optimization method according to claim 1, characterized in that: The step of using a noise estimation model to perform noise calculation on the noise addition result to obtain predicted noise includes: Determine the number of iterations corresponding to the noise addition result to obtain a target number of iterations; The target number of iterations is mapped through a preset fully connected layer to obtain a mapping result, and the noise estimation model is used to perform noise prediction on the mapping result and the noise addition result to obtain the predicted noise.

5. The audio optimization method according to claim 1, characterized in that: The step of calculating a prediction loss according to the predicted noise, the real noise corresponding to the noise-added model, and the predicted pitch curve, and updating the noise estimation model according to the predicted loss to obtain an updated noise estimation model comprises: Calculate the prediction loss according to the real noise, the predicted noise, the predicted pitch curve and a preset prediction loss function; Based on the prediction loss, the parameters of the noise estimation model are updated using the gradient descent method until the prediction loss converges, thereby obtaining the updated noise estimation model.

6. The audio optimization method according to claim 1, characterized in that: The denoising the denoised result according to the reference noise to obtain a denoised result includes: The number of iterations is mapped to an adjustment parameter using a preset mapping table, the denoising result is used as the data to be denoised, the iteration round is initialized to zero, the denoised data is calculated according to the adjustment parameter, the reference noise, the denoising result and a preset denoising function, the denoised data is used as the data to be denoised, and the iteration round is increased by one; Return to executing the step of calculating the denoised data according to the adjustment parameter, the reference noise, the denoising result and the preset denoising function, taking the denoised data as the data to be denoised, increasing the iteration round by one, until the iteration round and the number of iterations are the same, obtaining the denoised data of the corresponding iteration round, and determining the denoised data with the largest iteration round as the denoising result.

7. The audio optimization method according to any one of claims 1 to 6, characterized in that: The training process of the embedding layer, the linear layer and the prediction model includes: Acquire a sample spectrum envelope, a sample audio parameter, and a pitch label of the sample audio, use the linear layer to perform feature mapping on the sample spectrum envelope to obtain a sample envelope feature, and use the embedding layer to perform feature embedding on the sample audio parameter to obtain a sample parameter feature; The sample envelope feature and the sample parameter feature are fused to obtain a sample fusion feature, and the prediction model is used to perform pitch prediction on the sample fusion feature to obtain a sample pitch curve; The embedding layer, the linear layer and the prediction model are trained according to the sample pitch curve and the pitch label to obtain the trained embedding layer, the trained linear layer and the trained prediction model.

8. An audio optimization device based on artificial intelligence, characterized in that: The audio optimization device comprises: A feature extraction module is used to obtain the spectrum envelope of the audio to be optimized, perform feature mapping on the spectrum envelope using a trained linear layer to obtain envelope features, determine standard audio parameters based on the audio to be optimized, and perform feature embedding on the standard audio parameters using a trained embedding layer to obtain parameter features; A pitch prediction module, used for fusing the envelope feature and the parameter feature to obtain a fused feature, and using a trained prediction model to perform pitch prediction on the fused feature to obtain a predicted pitch curve; A spectrum noise adding module, used for obtaining the Mel spectrum of the audio to be optimized, and adding noise to the Mel spectrum using a noise adding model to obtain a noise adding result; A model updating module, configured to use a noise estimation model to perform noise calculation on the noise addition result to obtain predicted noise, calculate a predicted loss based on the predicted noise, the real noise corresponding to the noise addition model and the predicted pitch curve, and update the noise estimation model based on the predicted loss to obtain an updated noise estimation model; The spectrum optimization module is used to use the updated noise estimation model to perform noise calculation on the noisy result to obtain a reference noise, denoise the noisy result according to the reference noise to obtain a denoised result, and use the denoised result as the optimized Mel spectrum.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the audio optimization method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the audio optimization method according to any one of claims 1 to 7 is implemented.

Images