Audio noise reduction model training method, singing work processing method, device, and medium

Abstract

The application relates to an audio noise reduction model training method, a singing work processing method, a computer device and a storage medium. The audio noise reduction model training method comprises the following steps: obtaining target clean audio of a first sample user and interference clean audio of a second sample user; constructing first noise audio by using the target clean audio, and constructing second noise audio by using the target clean audio and the interference clean audio; inputting the first noise audio into a to-be-trained audio noise reduction model to obtain first noise reduction audio, and obtaining an initial audio noise reduction model trained by using a difference degree between the first noise reduction audio and the target clean audio; inputting the second noise audio into the initial audio noise reduction model to obtain second noise reduction audio, and obtaining an audio noise reduction model trained by using a difference degree between the second noise reduction audio and the target clean audio. The method can improve the noise reduction precision of singing audio.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an audio noise reduction model training method, a singing work processing method, computer equipment, and storage medium. Background Technology

[0002] With the development of computer technology, a technique has emerged that uses music software to create singing works. Users can record human vocal audio through a microphone, and then mix the recorded human vocal audio with musical accompaniment to generate the final singing work.

[0003] Because recording vocals with a microphone usually captures ambient noise and musical accompaniment, directly using the recorded vocal audio to generate the final song might affect its quality. Therefore, before generating the final song, a trained audio noise reduction model is typically used to denoise the recorded vocal audio, and then the denoised vocal audio is used to generate the final song.

[0004] However, current audio noise reduction models can typically only remove environmental noise and musical accompaniment from singing audio, resulting in low noise reduction accuracy for singing audio. Summary of the Invention

[0005] Therefore, it is necessary to provide an audio noise reduction model training method, a singing work processing method, a device, a computer equipment, a computer-readable storage medium, and a computer program product that can improve the noise reduction accuracy in response to the above-mentioned technical problems.

[0006] In a first aspect, this application provides a method for training an audio noise reduction model, the method comprising:

[0007] Obtain the target clean audio of the first sample user and the interference clean audio of the second sample user;

[0008] A first noisy audio is constructed using the target clean audio, and a second noisy audio is constructed using the target clean audio and the interfering clean audio;

[0009] The first noisy audio is input into the audio denoising model to be trained. The audio denoising model is used to denoise the first noisy audio to obtain the first denoised frequency. The audio denoising model is trained by using the degree of difference between the first denoised frequency and the target clean audio to obtain the initial audio denoising model after training.

[0010] The second noisy audio is input into the initial audio denoising model, and the second noisy audio is denoised by the initial audio denoising model to obtain the second denoised frequency. The initial audio denoising model is then trained using the degree of difference between the second denoised frequency and the target clean audio to obtain the trained audio denoising model.

[0011] In one embodiment, the audio denoising model includes: a first sub-model and a second sub-model; the step of inputting the first noisy audio into the audio denoising model to be trained, performing denoising processing on the first noisy audio through the audio denoising model to obtain a first denoised frequency, and training the audio denoising model using the degree of difference between the first denoised frequency and the target clean audio to obtain a trained initial audio denoising model includes: inputting the first noisy audio into the first sub-model of the audio denoising model to be trained, performing a first denoising processing on the first noisy audio through the first sub-model to obtain a first denoised frequency; obtaining the degree of difference between the first denoised frequency and the target clean audio, training the first sub-model based on the degree of difference between the first denoised frequency and the target clean audio to obtain a trained first sub-model; and obtaining the initial audio denoising model based on the trained first sub-model and the second sub-model.

[0012] In one embodiment, the step of inputting the second noisy audio into the initial audio denoising model, performing denoising processing on the second noisy audio through the initial audio denoising model to obtain a second denoised frequency, and training the initial audio denoising model using the degree of difference between the second denoised frequency and the target clean audio to obtain a trained audio denoising model includes: inputting the second noisy audio into a trained first sub-model of the initial audio denoising model, performing a first denoising process on the second noisy audio through the trained first sub-model to obtain an initial denoised frequency; inputting the initial denoised frequency into a second sub-model of the initial audio denoising model, performing a second denoising process on the initial denoised frequency through the second sub-model to obtain the second denoised frequency; obtaining the degree of difference between the second denoised frequency and the target clean audio, training the second sub-model based on the degree of difference between the second denoised frequency and the target clean audio to obtain a trained second sub-model; and obtaining the trained audio denoising model based on the trained first sub-model and the trained second sub-model.

[0013] In one embodiment, the second sub-model includes a timbre feature extraction module; the step of inputting the initial noise-reduced frequency into the second sub-model of the initial audio noise reduction model, and performing a second noise reduction process on the initial noise-reduced frequency through the second sub-model to obtain the second noise-reduced frequency includes: inputting the initial noise-reduced frequency into the timbre feature extraction module to obtain the timbre features corresponding to the initial noise-reduced frequency; and performing a second noise reduction process on the initial noise-reduced frequency based on the timbre features to obtain the second noise-reduced frequency.

[0014] In one embodiment, the step of inputting the first noisy audio into an audio denoising model to be trained, and performing denoising processing on the first noisy audio through the audio denoising model to obtain a first denoised frequency includes: inputting the first noisy audio into the audio denoising model to be trained, performing sub-band decomposition processing on the first noisy audio through the audio denoising model to obtain multiple noise sub-bands of the first noisy audio; obtaining the spectral features of each noise sub-band, and performing denoising processing on the spectral features using the denoising module of the audio denoising model to obtain the denoised spectral features of each noise sub-band; obtaining the corresponding denoised sub-band based on the denoised spectral features of each noise sub-band, and using each denoised sub-band to obtain the first denoised frequency.

[0015] In one embodiment, the step of constructing a first noisy audio using the target clean audio and constructing a second noisy audio using the target clean audio and interfering clean audio includes: acquiring sample noise data; the sample noise data includes: sample environmental noise and / or sample accompaniment audio; constructing the first noisy audio using the target clean audio and the sample noise data according to a preset first signal-to-noise ratio; and constructing the second noisy audio using the target clean audio, interfering clean audio, and the sample noise data according to a preset second signal-to-noise ratio.

[0016] Secondly, this application also provides a method for processing singing works, including:

[0017] Obtain the accompaniment audio and vocal audio used to generate the singing piece;

[0018] The singing audio is input into the trained audio denoising model, and the audio denoising audio is processed by the audio denoising model to obtain the denoised singing audio; the audio denoising model is trained by the audio denoising model training method as described in any embodiment of the first aspect.

[0019] A singing piece is generated using the accompanying audio and the noise-reduced vocal audio.

[0020] In one embodiment, before inputting the singing audio into the trained audio denoising model, the method further includes: obtaining clean audio of the target user corresponding to the singing audio, and pre-storing the clean audio into the timbre feature extraction module of the trained audio denoising model; the timbre feature extraction module is used to extract the timbre features of the clean audio as the timbre features of the target user; the step of inputting the singing audio into the trained audio denoising model and performing denoising processing on the singing audio through the audio denoising model to obtain the denoised singing audio includes: inputting the singing audio into the trained audio denoising model, performing denoising processing on the singing audio based on the timbre features of the target user, and obtaining the denoised singing audio.

[0021] Thirdly, this application also provides an audio noise reduction model training device, the device comprising:

[0022] The clean audio acquisition module is used to acquire the target clean audio of the first sample user and the interference clean audio of the second sample user.

[0023] The sample environmental noise acquisition module is used to construct a first noise audio using the target clean audio, and to construct a second noise audio using the target clean audio and the interfering clean audio.

[0024] The initial noise reduction model training module is used to input the first noisy audio into the audio noise reduction model to be trained, perform noise reduction processing on the first noisy audio through the audio noise reduction model to obtain the first noise-reduced frequency, and use the degree of difference between the first noise-reduced frequency and the target clean audio to train the audio noise reduction model to obtain the trained initial audio noise reduction model.

[0025] The audio noise reduction model training module is used to input the second noisy audio into the initial audio noise reduction model, perform noise reduction processing on the second noisy audio through the initial audio noise reduction model to obtain a second noise-reduced frequency, and use the degree of difference between the second noise-reduced frequency and the target clean audio to train the initial audio noise reduction model to obtain a trained audio noise reduction model.

[0026] Fourthly, this application also provides a vocal work processing device, the device comprising:

[0027] The accompaniment and vocal acquisition module is used to acquire the accompaniment audio and vocal audio used to generate the singing work;

[0028] The noise-reduced vocal acquisition module is used to input the vocal audio into a trained audio noise reduction model, and to perform noise reduction processing on the vocal audio through the audio noise reduction model to obtain the noise-reduced vocal audio; the audio noise reduction model is trained by the audio noise reduction model training method as described in any embodiment of the first aspect.

[0029] The singing composition generation module is used to generate singing compositions using the accompaniment audio and the noise-reduced vocal audio.

[0030] Fifthly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0031] Obtain the target clean audio of the first sample user and the interference clean audio of the second sample user;

[0032] A first noisy audio is constructed using the target clean audio, and a second noisy audio is constructed using the target clean audio and the interfering clean audio;

[0033] The first noisy audio is input into the audio denoising model to be trained. The audio denoising model is used to denoise the first noisy audio to obtain the first denoised frequency. The audio denoising model is trained by using the degree of difference between the first denoised frequency and the target clean audio to obtain the initial audio denoising model after training.

[0034] The second noisy audio is input into the initial audio denoising model, and the second noisy audio is denoised by the initial audio denoising model to obtain the second denoised frequency. The initial audio denoising model is trained by using the degree of difference between the second denoised frequency and the target clean audio to obtain the trained audio denoising model.

[0035] When the processor executes the computer program, it also performs the following steps:

[0036] Obtain the accompaniment audio and vocal audio used to generate the singing piece;

[0037] The singing audio is input into the trained audio denoising model, and the audio denoising audio is processed by the audio denoising model to obtain the denoised singing audio; the audio denoising model is trained by the audio denoising model training method as described in any embodiment of the first aspect.

[0038] A singing piece is generated using the accompanying audio and the noise-reduced vocal audio.

[0039] Sixthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0040] Obtain the target clean audio of the first sample user and the interference clean audio of the second sample user;

[0041] A first noisy audio is constructed using the target clean audio, and a second noisy audio is constructed using the target clean audio and the interfering clean audio;

[0042] The first noisy audio is input into the audio denoising model to be trained. The audio denoising model is used to denoise the first noisy audio to obtain the first denoised frequency. The audio denoising model is trained by using the degree of difference between the first denoised frequency and the target clean audio to obtain the initial audio denoising model after training.

[0043] The second noisy audio is input into the initial audio denoising model, and the second noisy audio is denoised by the initial audio denoising model to obtain the second denoised frequency. The initial audio denoising model is trained by using the degree of difference between the second denoised frequency and the target clean audio to obtain the trained audio denoising model.

[0044] When the computer program is executed by the processor, it also performs the following steps:

[0045] Obtain the accompaniment audio and vocal audio used to generate the singing piece;

[0046] The singing audio is input into the trained audio denoising model, and the audio denoising audio is processed by the audio denoising model to obtain the denoised singing audio; the audio denoising model is trained by the audio denoising model training method as described in any embodiment of the first aspect.

[0047] A singing piece is generated using the accompanying audio and the noise-reduced vocal audio.

[0048] Seventhly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0049] Obtain the target clean audio of the first sample user and the interference clean audio of the second sample user;

[0050] A first noisy audio is constructed using the target clean audio, and a second noisy audio is constructed using the target clean audio and the interfering clean audio;

[0051] The first noisy audio is input into the audio denoising model to be trained. The audio denoising model is used to denoise the first noisy audio to obtain the first denoised frequency. The audio denoising model is trained by using the degree of difference between the first denoised frequency and the target clean audio to obtain the initial audio denoising model after training.

[0052] The second noisy audio is input into the initial audio denoising model, and the second noisy audio is denoised by the initial audio denoising model to obtain the second denoised frequency. The initial audio denoising model is trained by using the degree of difference between the second denoised frequency and the target clean audio to obtain the trained audio denoising model.

[0053] When the computer program is executed by the processor, it also performs the following steps:

[0054] Obtain the accompaniment audio and vocal audio used to generate the singing piece;

[0055] The singing audio is input into the trained audio denoising model, and the audio denoising audio is processed by the audio denoising model to obtain the denoised singing audio; the audio denoising model is trained by the audio denoising model training method as described in any embodiment of the first aspect.

[0056] A singing piece is generated using the accompanying audio and the noise-reduced vocal audio.

[0057] The aforementioned audio denoising model training method, singing work processing method, device, computer equipment, storage medium, and computer program product acquire the target clean audio from a first sample user and the interference clean audio from a second sample user; construct a first noisy audio using the target clean audio, and construct a second noisy audio using the target clean audio and the interference clean audio; input the first noisy audio into the audio denoising model to be trained, perform denoising processing on the first noisy audio through the audio denoising model to obtain a first denoised frequency, and train the audio denoising model using the degree of difference between the first denoised frequency and the target clean audio to obtain a trained initial audio denoising model; input the second noisy audio into the initial audio denoising model, perform denoising processing on the second noisy audio through the initial audio denoising model to obtain a second denoised frequency, and train the initial audio denoising model using the degree of difference between the second denoised frequency and the target clean audio to obtain a trained audio denoising model. This application constructs a first noisy audio using clean audio from a first sample user, and constructs a second noisy audio using clean audio from a second sample user and the clean audio from the first sample user. The initial audio denoising model is first trained using the first noisy audio, and then further trained using the second noisy audio. This results in an audio denoising model that not only removes environmental noise and musical accompaniment, but also removes interference audio from other users affecting the target user, thereby improving the denoising accuracy of singing audio. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the audio noise reduction model training method in one embodiment;

[0059] Figure 2 This is a schematic diagram of the process of training the initial audio noise reduction model in one embodiment;

[0060] Figure 3 This is a schematic diagram of the process of training an audio noise reduction model in one embodiment;

[0061] Figure 4 This is a schematic diagram of the process of training the initial audio noise reduction model in one embodiment;

[0062] Figure 5 This is a schematic diagram of the process of constructing noisy audio in one embodiment;

[0063] Figure 6 This is a flowchart illustrating a method for processing singing works in one embodiment;

[0064] Figure 7 This is a flowchart illustrating a method for generating a singing piece in one embodiment;

[0065] Figure 8 This is a flowchart illustrating a method for generating a singing piece in another embodiment;

[0066] Figure 9 This is a block diagram of a neural network-based audio noise reduction system in one embodiment;

[0067] Figure 10 This is a flowchart illustrating the training process of a noise reduction model in one embodiment;

[0068] Figure 11 This is a schematic diagram of audio framing in one embodiment;

[0069] Figure 12 This is a schematic diagram of the noise reduction network structure in one embodiment;

[0070] Figure 13 This is a flowchart illustrating the prediction process of a noise reduction model in one embodiment;

[0071] Figure 14 This is a structural block diagram of an audio noise reduction model training device in one embodiment;

[0072] Figure 15 This is a structural block diagram of a singing work processing device in one embodiment;

[0073] Figure 16 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0075] In one embodiment, such as Figure 1 As shown, an audio noise reduction model training method is provided. This embodiment illustrates the method's application to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0076] Step S101: Obtain the target clean audio of the first sample user and the interference clean audio of the second sample user.

[0077] The target clean audio refers to the clean audio of the first sample user, such as a singing or human voice signal recorded by the first sample user in a quiet environment. The first sample user is the sample user corresponding to the audio signal that needs to be retained in the end. The second sample user is a sample user that is different from the first sample user. The interference clean audio is the clean audio provided by other users. This clean audio can be used to interfere with the clean audio of the first sample user. It can be a singing or human voice signal recorded by the second sample user in a quiet environment.

[0078] In this embodiment, the terminal can obtain the singing or human voice signals recorded by the first sample user and the second sample user in a quiet environment, and use them as the target clean audio and the interference clean audio, respectively.

[0079] Step S102: Construct a first noisy audio using the target clean audio, and construct a second noisy audio using the target clean audio and the interference clean audio.

[0080] The first noisy audio is constructed based on the target clean audio, for example, by adding ambient noise to the target clean audio. The second noisy audio is constructed using the target clean audio and interfering clean audio, which can be obtained by simultaneously adding interfering clean audio and ambient noise to the target clean audio. Specifically, after obtaining the target clean audio and interfering clean audio, the terminal can add noise to the target clean audio to obtain the first noisy audio, and simultaneously add interfering clean audio and noise to the target clean audio to obtain the second noisy audio.

[0081] Step S103: Input the first noisy audio into the audio denoising model to be trained, perform denoising processing on the first noisy audio through the audio denoising model to obtain the first denoised frequency, and use the degree of difference between the first denoised frequency and the target clean audio to train the audio denoising model to obtain the initial audio denoising model after training.

[0082] The audio noise reduction model to be trained refers to the neural network model that needs to be trained to remove audio noise. This model may include a noise reducer to remove noise signals. The initial audio noise reduction model is the audio noise reduction model trained using the first noisy audio. In this embodiment, the training of the audio noise reduction model can be roughly divided into two stages. The first stage mainly uses the first noisy audio to train the audio noise reduction model, which is used to remove the noise signals contained in the first noisy audio.

[0083] Specifically, the terminal can input the first noisy audio into the audio denoising model. The audio denoising model can then use the denoiser built into the model to denoise the first noisy audio, obtaining the denoised first noisy audio, i.e., the first denoised frequency. Then, the audio denoising model can be trained by using the degree of difference between the first denoised frequency and the target clean audio. For example, the target clean audio can be used as the target, and a loss function can be constructed based on the mean square error between the first denoised frequency and the target clean audio. The constructed loss function can then be used to train the audio denoising model to obtain the initial audio denoising model.

[0084] Step S104: Input the second noisy audio into the initial audio denoising model, perform denoising processing on the second noisy audio through the initial audio denoising model to obtain the second denoised frequency, and use the degree of difference between the second denoised frequency and the target clean audio to train the initial audio denoising model to obtain the trained audio denoising model.

[0085] Since the initial audio noise reduction model only has the ability to remove environmental noise, in singing scenarios, in addition to environmental noise and the user's singing voice, there is usually other human voice information. This human voice information is also part of the noise information in relation to the user's singing voice. The initial audio noise reduction model trained so far does not have the ability to remove other human voice information. Therefore, the audio noise reduction model needs to be further trained so that it can remove other human voice information, that is, remove the human voice information of the second sample user.

[0086] Specifically, the terminal can input the second noisy audio again into the initial audio denoising model. Then, the denoiser in the initial audio denoising model can be used to denoise the second noisy audio to obtain the denoised second noisy audio, i.e., the second denoised frequency. Then, the difference between the second denoised frequency and the target clean audio can be used to train the initial audio denoising model. For example, the target clean audio can be used as the target, and a loss function can be constructed based on the mean square error between the second denoised frequency and the target clean audio. The constructed loss function can then be used to train the audio denoising model to obtain the trained audio denoising model.

[0087] In the above-mentioned audio denoising model training method, the following steps are taken: First, target clean audio from a first sample user and interference clean audio from a second sample user are acquired. Then, a first noisy audio is constructed using the target clean audio, and a second noisy audio is constructed using the target clean audio and interference clean audio. The first noisy audio is input into the audio denoising model to be trained, and the model denoises the first noisy audio to obtain a first denoised frequency. The difference between the first denoised frequency and the target clean audio is used to train the audio denoising model, resulting in a trained initial audio denoising model. Finally, the second noisy audio is input into the initial audio denoising model, and the model denoises the second noisy audio to obtain a second denoised frequency. The difference between the second denoised frequency and the target clean audio is used to train the initial audio denoising model, resulting in a trained audio denoising model. This application constructs a first noisy audio using clean audio from a first sample user, and constructs a second noisy audio using clean audio from a second sample user and the clean audio from the first sample user. The initial audio denoising model is first trained using the first noisy audio, and then further trained using the second noisy audio. This results in an audio denoising model that not only removes environmental noise and musical accompaniment, but also removes interference audio from other users affecting the target user, thereby improving the denoising accuracy of singing audio.

[0088] In one embodiment, such as Figure 2 As shown, the audio noise reduction model may include: a first sub-model and a second sub-model; step S103 may further include:

[0089] Step S201: Input the first noisy audio into the first sub-model of the audio denoising model to be trained, and perform the first denoising process on the first noisy audio through the first sub-model to obtain the first denoised audio.

[0090] In this embodiment, the audio denoising model can consist of two parts: a first sub-model and a second sub-model. The first sub-model is mainly used to remove environmental noise, while the second sub-model is mainly used to remove interference audio from other users affecting the target user. The first denoising process is implemented through the first sub-model. Specifically, the terminal can input the first noisy audio into the first sub-model of the audio denoising model to be trained, and the first sub-model will perform denoising processing on the first noisy audio to obtain the first denoised frequency.

[0091] Step S202: Obtain the degree of difference between the first noise-reduced frequency and the target clean audio, and train the first sub-model based on the degree of difference between the first noise-reduced frequency and the target clean audio to obtain the trained first sub-model;

[0092] Step S203: Based on the first and second sub-models that have been trained, obtain the initial audio noise reduction model.

[0093] The terminal can then calculate the degree of difference between the first noise-reduced frequency obtained from the first sub-model and the target clean audio, and use this degree of difference to train the first sub-model to obtain the trained first sub-model. The trained first sub-model and the untrained second sub-model can then be used as the initial audio noise reduction model.

[0094] In this embodiment, by dividing the audio denoising model into a first sub-model and a second sub-model, the first sub-model can be trained using the first noisy audio, and the trained first sub-model and the second sub-model can be combined to obtain the initial audio denoising model. This method can improve the ability of the initial audio denoising model to remove environmental noise, and further improve the audio denoising accuracy for environmental noise.

[0095] Furthermore, such as Figure 3 As shown, step S104 may further include:

[0096] Step S301: Input the second noisy audio into the first sub-model trained in the initial audio denoising model, and perform the first denoising process on the second noisy audio through the trained first sub-model to obtain the initial denoised audio.

[0097] The initial noise-reduced frequency refers to the noise-reduced frequency obtained by processing the second noisy audio through the first sub-model that has been trained. After the terminal completes the training of the initial audio noise reduction model, it can input the second noisy audio into the initial audio noise reduction model and use the first sub-model that has been pre-trained in the initial audio noise reduction model to process the second noisy audio to obtain the initial noise-reduced frequency.

[0098] Step S302: Input the initial noise-reduced frequency into the second sub-model in the initial audio noise reduction model, and perform second noise reduction processing on the initial noise-reduced frequency through the second sub-model to obtain the second noise-reduced frequency;

[0099] Step S303: Obtain the degree of difference between the second noise-reduced frequency and the target clean audio, and train the second sub-model based on the degree of difference between the second noise-reduced frequency and the target clean audio to obtain the trained second sub-model;

[0100] Step S304: Based on the first sub-model and the second sub-model that have been trained, obtain the trained audio noise reduction model.

[0101] After obtaining the initial noise-reduced frequency, the terminal can also input the initial noise-reduced frequency into the second sub-model in the initial audio noise reduction model to train the second sub-model. The second sub-model then performs noise reduction processing on the initial noise-reduced frequency again to remove the interference clean audio of the second sample user in the second noisy audio. Thus, the second noise-reduced frequency can be obtained through the second sub-model. Afterward, the terminal can train the second sub-model based on the degree of difference between the second noise-reduced frequency and the target clean audio of the first sample user to obtain the trained second sub-model. This allows the terminal to combine the pre-trained first sub-model in the initial audio noise reduction model and the trained second sub-model in step S303 to obtain the trained audio noise reduction model.

[0102] In this embodiment, after the training of the first sub-model is completed, the initial noise reduction frequency of the second noisy audio can be obtained using the first sub-model, and the initial noise reduction frequency can be input into the second sub-model to obtain the second noise reduction frequency. Thus, the second sub-model can be trained using the degree of difference between the second noise reduction frequency and the target clean audio, which enables the second sub-model to have the ability to remove the interference clean audio of the second sample user, further improving the removal accuracy of interference noise for other users.

[0103] Furthermore, the second sub-model includes a timbre feature extraction module; step S302 may further include: inputting the initial noise-reduced frequency into the timbre feature extraction module to obtain the timbre features corresponding to the initial noise-reduced frequency; and performing a second noise reduction process on the initial noise-reduced frequency based on the timbre features to obtain the second noise-reduced frequency.

[0104] The timbre feature extraction module is used to extract user timbre features from audio. This module can be implemented using models such as ECAPA-TDNN or d-vector and can be used to identify the speaking user. Specifically, after obtaining the initial noise-reduced frequency, the terminal can first input the initial noise-reduced frequency into the timbre feature extraction module in the second sub-model. The timbre feature extraction module extracts the timbre features corresponding to the initial noise-reduced frequency. Then, the terminal can also use the timbre features to perform a second noise reduction process on the initial noise-reduced frequency. By combining the timbre features and the initial noise-reduced frequency, and inputting them into the convolutional noise reduction module in the second sub-model, the convolutional noise reduction module performs a second noise reduction process on the combined timbre features and the initial noise-reduced frequency to obtain the second noise-reduced frequency.

[0105] In this embodiment, timbre features can also be extracted through the timbre feature extraction module, thereby performing noise reduction processing on the initial noise reduction frequency based on the timbre features. The timbre features can be used to identify the user corresponding to the clean audio sample, thereby more accurately removing the interference noise from other users.

[0106] In one embodiment, such as Figure 4As shown, step S103 may further include:

[0107] Step S401: Input the first noisy audio into the audio denoising model to be trained, and perform sub-band decomposition processing on the first noisy audio through the audio denoising model to obtain multiple noise sub-bands of the first noisy audio.

[0108] Subband decomposition can be achieved using PQMF (Pseudo-Qarbitrary Filtering), a technique that uses a pseudo-orthogonal mirror filter bank to divide the time-domain waveform signal into multiple subbands, thus reducing computational complexity. The noise subbands are the multiple subbands obtained after subband decomposition of the first noise audio signal. The terminal can utilize the subband module included in the audio denoising model to perform subband decomposition on the first noise audio signal input to the model, thereby obtaining multiple subbands corresponding to the first noise audio signal.

[0109] Step S402: Obtain the spectral characteristics of each noise sub-band, and use the denoising module of the audio denoising model to denoise the spectral characteristics to obtain the denoised spectral characteristics of each noise sub-band.

[0110] Spectral features can be obtained by performing a Fourier transform on the noise sub-bands. This involves dividing each noise sub-band into frames, applying a windowing operation, and then performing a Fourier transform. This yields the spectral amplitude and phase features of each noise sub-band, which are then used as its spectral features. The terminal can then use the denoising module of an audio denoising model, such as a denoiser, to denoise the obtained spectral features, thus obtaining the denoised spectral features of each noise sub-band.

[0111] Step S403: Based on the spectral characteristics of each noise sub-band after noise reduction, obtain the corresponding noise reduction sub-band, and use each noise reduction sub-band to obtain the first noise reduction frequency.

[0112] The noise reduction sub-band is the sub-band signal obtained after noise reduction processing of each noise sub-band. The noise reduction sub-band can be achieved by performing an inverse Fourier transform on the noise reduction spectral features. By performing an inverse Fourier transform on each noise reduction spectral feature, the noise reduction sub-band corresponding to each noise reduction spectral feature can be obtained. Furthermore, each noise reduction sub-band can be processed by sub-band synthesis to generate the first noise reduction frequency.

[0113] Similarly, the method of obtaining the second noise frequency using the second noise audio can be similar to the method of obtaining the first noise frequency using the first noise audio. By inputting the second noise audio into the initial audio denoising model, the initial audio denoising model decomposes the second noise audio into sub-bands to obtain multiple noise sub-bands. Then, a Fourier transform is performed on each noise sub-band to obtain the corresponding spectral features. These features are then input into the denoising module, which performs denoising processing on the spectral features. An inverse Fourier transform is then performed on each denoised spectral feature to obtain the denoised sub-bands corresponding to each denoised spectral feature. Finally, the denoised sub-bands can be synthesized to obtain the second noise frequency.

[0114] In this embodiment, the noisy audio is decomposed into subbands, and the spectral features of each subband are extracted. Then, the denoising module is used to denoise the spectral features of each subband, and the denoised subbands are obtained. These subbands are then combined to form the corresponding denoised audio. This method can reduce computational complexity and improve the efficiency of obtaining denoised audio.

[0115] In one embodiment, such as Figure 5 As shown, step S102 may further include:

[0116] Step S501: Obtain sample noise data; the sample noise data includes: sample environmental noise and / or sample accompaniment audio.

[0117] Sample noise data refers to pre-collected data used to add noise to clean audio samples of users. This noise data can consist of two parts, or one of the following: ambient noise and accompaniment audio. Sample ambient noise refers to the ambient noise portion of the sample noise data, while sample accompaniment audio refers to the accompaniment audio portion of the sample noise data. Specifically, the terminal can pre-collect ambient noise data and accompaniment audio data for training the audio denoising model, serving as sample ambient noise and sample accompaniment audio.

[0118] Step S502: Using the target clean audio and sample noise data, construct the first noisy audio according to the preset first signal-to-noise ratio.

[0119] The first signal-to-noise ratio refers to the ratio between the target clean audio signal and the sample noise data during the construction of the first noisy audio. The terminal can use the target clean audio and the sample noise data to combine them according to the first signal-to-noise ratio to obtain the corresponding first noisy audio. Since the sample noise data can be composed of either sample ambient noise and sample accompaniment audio, or sample ambient noise and sample accompaniment audio, the constructed first noisy audio also includes several types, namely the first noisy audio composed of the target clean audio and sample ambient noise, the first noisy audio composed of the target clean audio and sample accompaniment audio, and the first noisy audio composed of the target clean audio, sample ambient noise, and sample accompaniment audio.

[0120] Step S503: Using the target clean audio, the interference clean audio, and the sample noise data, construct the second noise audio according to the preset second signal-to-noise ratio.

[0121] The second signal-to-noise ratio (SNR) refers to the ratio between the target clean audio signal, the interference clean audio signal, and the sample noise data during the construction of the second noisy audio. The terminal can combine the target clean audio, the interference clean audio, and the sample noise data according to the second SNR to obtain the corresponding second noisy audio. Similar to the first noisy audio, since the sample noise data can be composed of either sample ambient noise and sample accompaniment audio, or sample ambient noise and sample accompaniment audio, the constructed second noisy audio also includes several types: second noisy audio composed of target clean audio, interference clean audio, and sample ambient noise; second noisy audio composed of target clean audio, interference clean audio, and sample accompaniment audio; and second noisy audio composed of target clean audio, interference clean audio, sample ambient noise, and sample accompaniment audio.

[0122] In this embodiment, the sample noise data can be composed of sample environmental noise and sample accompaniment audio. Therefore, the trained audio denoising model can be more effective in removing environmental noise and accompaniment audio, which are common noise signals in singing works. This can further improve the ability of the audio denoising model to remove environmental noise and musical accompaniment.

[0123] In one embodiment, such as Figure 6 As shown, a method for processing singing works is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and furthermore, to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0124] Step S601: Obtain the accompaniment audio and vocal audio used to generate the singing piece.

[0125] Accompaniment audio can refer to the mid-range accompaniment of a singing work, while vocal audio is a user's recorded singing voice. The accompaniment audio can be downloaded to the terminal in advance via the network, while the vocal audio can be obtained by the terminal's audio signal acquisition device, such as a microphone connected to the terminal, by acquiring the user's singing audio.

[0126] Step S602: Input the singing audio into the trained audio denoising model, and perform denoising processing on the singing audio through the audio denoising model to obtain the denoised singing audio; the audio denoising model is trained by the audio denoising model training method described in any of the above embodiments.

[0127] After obtaining the singing audio, it can be input into the audio denoising model trained in the above embodiment. Since the audio denoising model has the ability to remove environmental noise, accompaniment audio, and interference audio from other users, the singing audio can be denoised by the audio denoising model, so only the clean singing audio of the singing user can be retained, while noise such as environmental noise, accompaniment audio, and interference audio from other users can be removed.

[0128] Step S603: Using the accompaniment audio and the noise-reduced vocal audio, generate a singing piece.

[0129] After obtaining the accompaniment audio and the noise-reduced vocal audio, the terminal can combine them to generate the final singing performance. For example, the accompaniment can be pitch-adjusted and its loudness equalized, while the noise-reduced vocal audio can be equalized, dynamically compressed, enhanced with vocals, and reverb-equipped before its loudness equalization. The loudness-equalized accompaniment audio and the loudness-equalized vocal audio are then combined to obtain the final singing performance.

[0130] In the above-described method for processing singing works, the accompaniment audio and vocal audio used to generate the singing work are acquired; the vocal audio is input into a trained audio denoising model, and the audio denoising model is used to denoise the vocal audio to obtain denoised vocal audio; the audio denoising model is trained using the audio denoising model training method of any of the above embodiments; and the singing work is generated using the accompaniment audio and the denoised vocal audio. This application denoises vocal audio by training an audio denoising model. Since this audio denoising model not only has the ability to remove environmental noise and musical accompaniment, but also the ability to remove interference audio from other users affecting the target user, it can accurately remove environmental noise, musical accompaniment, and interference audio from other users affecting the target user, thus improving the denoising accuracy of the singing audio and further improving the quality of the generated singing work.

[0131] In one embodiment, before step S602, the method may further include: obtaining the clean audio of the target user corresponding to the singing audio, and pre-storing the clean audio into the timbre feature extraction module of the trained audio denoising model; the timbre feature extraction module is used to extract the timbre features of the clean audio as the timbre features of the target user; step S602 may further include: inputting the singing audio into the trained audio denoising model, and performing denoising processing on the singing audio based on the timbre features of the target user to obtain the denoised singing audio.

[0132] In this embodiment, the target user refers to the user singing the song in the audio. Before inputting the target user's singing audio into the trained audio denoising model, the terminal needs to obtain the target user's clean audio. This clean audio can be obtained by the target user recording their singing / voice in a quiet environment. If the target user has previously recorded a song, relatively clean audio can also be extracted from the previously recorded song. The clean audio is then stored in the timbre feature extraction module of the trained audio denoising model. The timbre feature extraction module extracts the timbre features of the clean audio as the timbre features of the target user.

[0133] Afterwards, once the terminal obtains the singing audio recorded by the target user and inputs it into the trained audio denoising model, the audio denoising model can then perform denoising processing on the singing audio based on the timbre features of the target user pre-stored in the timbre feature extraction module, so as to obtain the final denoised singing audio.

[0134] In this embodiment, the terminal can pre-acquire the clean audio of the target user corresponding to the singing audio, and pre-input the clean audio into the timbre feature extraction module of the audio noise reduction model. The timbre feature extraction module extracts the timbre features of the target user, so that when the singing audio is input into the trained audio noise reduction model, the timbre features of the target user can be used for noise reduction, which can further improve the noise reduction accuracy of the singing audio.

[0135] In one embodiment, a method for enhancing singing audio is also provided, which can be applied to generating singing works, such as... Figure 7As shown, the current generation of vocal works typically involves processing the user's vocals with various equalization and reverb effects, followed by loudness equalization with the accompaniment, mixing, and finally creating a complete musical piece. However, when capturing vocals through a microphone, in addition to the user's singing, the accompaniment and ambient noise are also captured simultaneously. During equalization and reverb processing, the influence of the accompaniment and noise makes it difficult to achieve the desired effect, especially in accompaniment scenarios where the captured accompaniment is played through speakers. Due to the different paths taken by different speakers and microphones, it differs from the original well-produced accompaniment. Furthermore, the final work needs to be mixed with the accompaniment, resulting in two layers of accompaniment. Because the accompaniment is present within the vocals, the exact proportion cannot be accurately calculated, making loudness equalization of the vocals and accompaniment difficult and hard to guarantee the desired effect. Therefore, removing noise and accompaniment from the vocals makes the various processes more controllable and improves the quality.

[0136] However, in the surrounding environment while singing, besides the user's own voice, there may be other human voices, such as the voices of passersby on the street; or the sounds of family and friends talking while singing at home. Even the accompaniment in a singing scenario may contain harmonies and other human voices. Therefore, the noise reduction methods described above can only eliminate noise and even accompaniment, but still retain human voices. Therefore, the method for generating singing works provided in this embodiment can be as follows... Figure 8 As shown, by using the user's singing voice as the target vocals, other voices that are not the user's voice are eliminated. Therefore, in addition to eliminating noise and accompaniment, it can also eliminate other voices besides the user, preserving only the user's voice as much as possible, thereby better enhancing the user's singing voice.

[0137] Specifically, the audio noise reduction system structure provided in this embodiment can be as follows: Figure 9 As shown, this can be achieved through the following steps:

[0138] 1. Training process:

[0139] (1) Collect a large amount of clean audio and various types of noise audio, such as noise from squares, roads, conference rooms, restaurants, coffee shops, keyboard typing, etc.

[0140] (2) Collect a large amount of accompaniment audio, such as accompaniment for instruments like piano, guitar, and drums, and original song accompaniment. Here, accompaniment can be broadly considered as noise, because it needs to be eliminated.

[0141] (3) The clean audio, noise, and accompaniment are mixed to obtain a noisy audio signal. This signal-to-noise ratio (SNR) can be adjusted to suit different noise levels. For example, the noisy audio signal can be: a) clean audio + noisy audio; b) clean audio + accompaniment audio; c) clean audio + noisy audio + accompaniment audio. The SNR required for the noise and accompaniment can be adjusted based on the application scenario. For example, the SNR can be selected from -15 to 20 dB. This solution does not impose further limitations on this; the appropriate SNR can be chosen based on the actual needs of the application scenario.

[0142] (4) Extract features from the noisy frequency. This can be done by Fourier transform, which converts the time-domain waveform to the frequency domain to obtain the amplitude and phase features of the noisy frequency.

[0143] (5) Train the noise reduction model obtained after feature extraction, i.e., the neural network. Here, the neural network is generally a neural network with structures such as CNN, RNN, and DNN.

[0144] (6) Training is performed by training on noisy frequency signals, with the corresponding clean audio signals as the target and the loss function as the constraint.

[0145] For ease of description, let the noisy frequency signal be denoted as x, the clean audio signal as s, and the predicted audio signal of the denoising model as z. Fourier transforms are performed on each signal, and the resulting spectral distributions are denoted as Xf, Sf, and Zf. The errors in Sf and Zf are calculated, and then the hyperparameters in the model are adjusted based on these errors to ensure the weights in the model are sufficiently trained and learned. Through continuous iteration, training is completed when the aforementioned errors are less than a preset value, resulting in the trained preset neural network model.

[0146] 2. Prediction process:

[0147] By extracting features from noisy frequencies, and then using a pre-trained denoising model to make predictions, the predicted audio spectrum features are output. Then, through parallel reconstruction, i.e. inverse Fourier transform, the denoised audio signal can be obtained.

[0148] Among them, such as Figure 10 As shown, the training process can specifically include the following steps:

[0149] 1. Sub-band module:

[0150] Subband decomposition is performed here, using PQMF, although other similar techniques can also be used. Several objectives are involved:

[0151] 1) By using a pseudo-orthogonal mirror filter bank, the time-domain waveform signal is divided into C sub-band signals, thus achieving signal frequency division. This can reduce computational complexity.

[0152] 2) Because more energy in the human voice is concentrated in the low frequency range. Moreover, in accompaniment, different instruments are distributed in different frequency ranges. By dividing the signal into different sub-bands, the accompaniment and noise cancellation can be better performed, thus improving the performance.

[0153] Finally, because the noise reduction unit processes different subbands, subband synthesis is required at the end.

[0154] 2. Time-frequency transformation (i.e., FFT / iFFT) module:

[0155] First, the input waveform signal x is divided into frames, where the frame length is L (generally a power of 2, such as 1024), and the frame shift is P (such as 0.5L, e.g., ...). Figure 11 (As shown), then a windowing operation is performed, followed by a Fourier transform. If the frame length is L, then there are a total of L frequency points after the FFT transform. Due to the symmetry and conjugation of frequency points, it is generally sufficient to take L / 2+1 frequency points.

[0156] Assume that the Fourier transform of the audio result is in the complex field: X = X r +iX i The corresponding amplitude is: Its corresponding phase is: α = arctan(X) i / X r Here, arctan is the arctangent function.

[0157] Here, a short-time Fourier transform is performed on the time-domain waveform of the noise-reducing frequency signal to obtain the transformed initial spectrum; where |X| is the amplitude feature of the extracted initial spectrum, and α is the phase feature of its initial spectrum.

[0158] 3. Noise Reduction Module:

[0159] Noise reduction network structure such as Figure 12 As shown, it is divided into two stages:

[0160] 1. First stage:

[0161] Data construction: Clean audio is mixed with noise and accompaniment to obtain noisy audio. It can be synthesized according to different signal-to-noise ratios to meet different noise levels in different scenarios. For example, noisy audio A can be: a) clean audio + noisy audio; b) clean audio + accompaniment audio; c) clean audio + noisy audio + accompaniment audio.

[0162] The clean audio here contains only one human voice and can be called the target clean audio.

[0163] Loss function: The training process mainly involves using paired clean signals and noisy signals A, with the clean signal as the target, to predict the mean square error between the waveform signal and the actual clean signal, which serves as the loss function.

[0164] It learns each subband independently, with input X and output Y.

[0165] It first processes the information on the frequency domain axis, which mainly learns the information within the frame; then it processes the information on the time axis, because audio is a temporal sequence, and on the time axis, it mainly learns the information between frames.

[0166] Meanwhile, audio contains various harmonic components and overtones, so a larger field of view in the frequency domain is more helpful in capturing correlations. A dilated convolutional network (DCNN) was used, specifically a 6-layer DCNN with dilation values ​​of 1, 2, 4, 8, 16, and 32 (powers of 2). BatchNorm layers were used for normalization, and PReLU was used as the activation layer. In particular, the frequency domain modeling showed good fidelity in capturing the numerous overtones and harmonics in the vocals, indicating better harmonic modeling. It also demonstrated strong suppression of transient noise (such as door opening sounds, keyboard sounds, and saliva sounds). Furthermore, the use of a ResNet structure resulted in faster convergence.

[0167] Audio often features strong contextual relationships; for example, singing elongates vowels to create a more rhythmic sound. Therefore, a broader perspective is established on the time axis to capture temporal relevance. Similarly, the same dilated convolutional network with consistent parameters is used as in the frequency domain. In singing, elongating a word and maintaining it for a longer duration helps preserve its integrity and avoids swallowing sounds.

[0168] Meanwhile, the entire network adopts a CNN structure, which can be parallelized or implemented using separable convolutions, making it a lightweight network capable of real-time processing.

[0169] Parameter configuration:

[0170] It was found that dividing the band into 4 subbands yielded better results; here, we take C=4 as an example:

[0171] The first Conv is a 2D convolution with 4 input channels and 8 output channels, kernel_size:(1,1), stride(1,1). It is a dilated convolution with 8 input channels and 8 output channels, kernel_size:(3,3), stride(1,1). The second Conv is a 2D convolution with 8 input channels and 4 output channels, kernel_size:(1,1), stride(1,1).

[0172] 2. Second stage:

[0173] Data construction: Clean audio, clean vocal interference, noise, and accompaniment are mixed to obtain a noisy frequency. It can be synthesized according to different signal-to-noise ratios to meet different noise levels in different scenarios. For example, the noisy frequency B can be: a) clean audio + clean vocal interference + noisy audio; b) clean audio + clean vocal interference + accompaniment audio; c) clean audio + clean vocal interference + noisy audio + accompaniment audio.

[0174] The difference between this stage and the first stage is that the first stage only had one human voice, while in the second stage, I added interfering human voices, mainly used to identify and extract the target human voice, thereby suppressing the interfering human voices and retaining only the target human voice.

[0175] Loss function: The training process mainly involves using a pair of clean signals and noisy signals B, with the clean signal as the target, to predict the mean square error between the waveform signal and the actual clean signal, which serves as the loss function.

[0176] It learns each subband independently, with input X and output Z.

[0177] During the training of the second stage, since the first stage has already been trained, the parameters of the first stage can be frozen, and only the model of the second stage is updated.

[0178] The Speaker Embedding module, which extracts timbre features, is a speaker recognition and extraction embedding. Industry-standard models such as ECAPA-TDNN or d-vector can be used. Since the scale of the output A after Speaker Embedding may differ from the output Y of the first stage, a Conv convolutional layer is added to output B, primarily to transform the size of A to match the scale of Y. Next, Y and B are multiplied by a dot product and then input into the dilated convolutional module. This is the same as the first stage, except the number of layers is increased to four, and it is not expanded further.

[0179] like Figure 13 As shown, the prediction process can specifically include the following steps:

[0180] Since the noise reducer contains a Speaker Embedding module, which is a timbre feature extraction module, the target timbre's Embedding needs to be extracted before noise reduction and then embedded into the noise reducer's Speaker Embedding.

[0181] Therefore, users need to first record a segment of their singing / voice in a quiet environment, and then input it into the Speaker Embedding module for extraction. Of course, if the user has already sung a song, we can also extract a relatively clean user voice from the user's previous dry vocals / works and perform the above operations, without requiring the user to re-record a new singing / voice segment.

[0182] Noise reduction prediction process as follows Figure 13 As shown, the denoiser is a pre-trained model module, and its calculation steps are the same as those used during training, so they will not be described in detail here.

[0183] This embodiment allows for appropriate noise reduction of noisy frequency signals to obtain cleaner audio signals, thereby improving the quality of the generated singing works.

[0184] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0185] Based on the same inventive concept, this application also provides an audio noise reduction model training device for implementing the audio noise reduction model training method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more audio noise reduction model training device embodiments provided below can be found in the limitations of the audio noise reduction model training method described above, and will not be repeated here.

[0186] In one embodiment, such as Figure 14 As shown, an audio denoising model training device is provided, including: a clean audio sample acquisition module 1401, a sample environmental noise acquisition module 1402, an initial denoising model training module 1403, and an initial denoising model training module 1404, wherein:

[0187] The sample clean audio acquisition module 1401 is used to acquire the target clean audio of the first sample user and the interference clean audio of the second sample user.

[0188] The sample environment noise acquisition module 1402 is used to construct a first noise audio using the target clean audio, and to construct a second noise audio using the target clean audio and the interference clean audio.

[0189] The initial noise reduction model training module 1403 is used to input the first noisy audio into the audio noise reduction model to be trained, perform noise reduction processing on the first noisy audio through the audio noise reduction model to obtain the first noise reduction frequency, and use the degree of difference between the first noise reduction frequency and the target clean audio to train the audio noise reduction model to obtain the trained initial audio noise reduction model.

[0190] The audio denoising model training module 1404 is used to input the second noisy audio into the initial audio denoising model, perform denoising processing on the second noisy audio through the initial audio denoising model to obtain the second denoised frequency, and use the degree of difference between the second denoised frequency and the target clean audio to train the initial audio denoising model to obtain the trained audio denoising model.

[0191] Based on the same inventive concept, this application also provides a singing work processing apparatus for implementing the singing work processing method described above. The solution provided by this apparatus is similar to the solution described in the above method; therefore, the specific limitations in one or more singing work processing apparatus embodiments provided below can be found in the limitations of the singing work processing method described above, and will not be repeated here.

[0192] In one embodiment, such as Figure 15 As shown, a singing work processing device is provided, including: an accompaniment vocal acquisition module 1501, a noise-reduced vocal acquisition module 1502, and a singing work generation module 1503, wherein:

[0193] The accompaniment and vocal acquisition module 1501 is used to acquire the accompaniment audio and vocal audio used to generate the singing work;

[0194] The noise-reduced vocal acquisition module 1502 is used to input the vocal audio into a trained audio noise reduction model, and to perform noise reduction processing on the vocal audio through the audio noise reduction model to obtain the noise-reduced vocal audio; the audio noise reduction model is trained by the audio noise reduction model training method as described in any of the above embodiments.

[0195] The singing composition generation module 1503 is used to generate a singing composition using the accompaniment audio and the noise-reduced vocal audio.

[0196] The modules in the aforementioned audio noise reduction model training device and singing work processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0197] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 16 As shown, the computer device includes a processor, memory, and a communication interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an audio noise reduction model training method or a singing performance processing method.

[0198] Those skilled in the art will understand that Figure 16 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0199] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0200] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0201] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0202] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0203] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0204] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0205] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for training an audio noise reduction model, characterized in that, The method includes: Obtain the target clean audio of the first sample user and the interference clean audio of the second sample user; A first noisy audio is constructed using the target clean audio, and a second noisy audio is constructed using the target clean audio and the interfering clean audio; The first noisy audio is input into the audio denoising model to be trained. The audio denoising model is used to denoise the first noisy audio to obtain the first denoised frequency. The audio denoising model is trained by using the degree of difference between the first denoised frequency and the target clean audio to obtain the initial audio denoising model after training. The second noisy audio is input into the initial audio denoising model, and the second noisy audio is denoised by the initial audio denoising model to obtain the second denoised frequency. The initial audio denoising model is then trained using the degree of difference between the second denoised frequency and the target clean audio to obtain the trained audio denoising model.

2. The method according to claim 1, characterized in that, The audio noise reduction model includes: a first sub-model and a second sub-model; The process of inputting the first noisy audio into the audio denoising model to be trained, performing denoising processing on the first noisy audio through the audio denoising model to obtain a first denoised frequency, and training the audio denoising model using the degree of difference between the first denoised frequency and the target clean audio to obtain a trained initial audio denoising model includes: The first noisy audio is input into the first sub-model of the audio denoising model to be trained, and the first noisy audio is subjected to first denoising processing through the first sub-model to obtain the first denoised audio. Obtain the degree of difference between the first noise-reduced frequency and the target clean audio, and train the first sub-model based on the degree of difference between the first noise-reduced frequency and the target clean audio to obtain the trained first sub-model; The initial audio noise reduction model is obtained based on the first sub-model and the second sub-model that have been trained.

3. The method according to claim 2, characterized in that, The process of inputting the second noisy audio into the initial audio denoising model, performing denoising processing on the second noisy audio using the initial audio denoising model to obtain a second denoised frequency, and training the initial audio denoising model using the difference between the second denoised frequency and the target clean audio to obtain a trained audio denoising model includes: The second noisy audio is input into the first sub-model trained in the initial audio denoising model, and the second noisy audio is subjected to first denoising processing through the first sub-model trained to obtain the initial denoised audio. The initial noise-reduced frequency is input into the second sub-model in the initial audio noise reduction model, and the second noise reduction process is performed on the initial noise-reduced frequency through the second sub-model to obtain the second noise-reduced frequency. Obtain the degree of difference between the second noise-reduced frequency and the target clean audio, and train the second sub-model based on the degree of difference between the second noise-reduced frequency and the target clean audio to obtain the trained second sub-model; The trained audio noise reduction model is obtained based on the first sub-model and the second sub-model that have been trained.

4. The method according to claim 3, characterized in that, The second sub-model includes a timbre feature extraction module; The step of inputting the initial noise-reduced frequency into the second sub-model of the initial audio noise reduction model, and performing a second noise reduction process on the initial noise-reduced frequency through the second sub-model to obtain the second noise-reduced frequency includes: The initial noise-reduced frequency is input into the timbre feature extraction module to obtain the timbre features corresponding to the initial noise-reduced frequency; Based on the timbre characteristics, the initial noise-reduced frequency is subjected to a second noise reduction process to obtain the second noise-reduced frequency.

5. The method according to claim 1, characterized in that, The step of inputting the first noisy audio into the audio denoising model to be trained, and performing denoising processing on the first noisy audio through the audio denoising model to obtain the first denoised audio, includes: The first noisy audio is input into the audio denoising model to be trained, and the first noisy audio is decomposed into multiple noise sub-bands by the audio denoising model. The spectral characteristics of each noise sub-band are obtained, and the denoising module of the audio denoising model is used to denoise the spectral characteristics to obtain the denoised spectral characteristics of each noise sub-band. Based on the spectral characteristics of each noise sub-band after denoising, the corresponding denoised sub-band is obtained, and the first denoised frequency is obtained using each denoised sub-band.

6. The method according to any one of claims 1 to 5, characterized in that, The step of constructing a first noisy audio using the target clean audio, and constructing a second noisy audio using the target clean audio and the interfering clean audio, includes: Acquire sample noise data; the sample noise data includes: sample ambient noise and / or sample accompaniment audio; Using the target clean audio and the sample noise data, the first noisy audio is constructed according to a preset first signal-to-noise ratio; Using the target clean audio, the interference clean audio, and the sample noise data, the second noisy audio is constructed according to a preset second signal-to-noise ratio.

7. A method for processing singing works, characterized in that, The method includes: Obtain the accompaniment audio and vocal audio used to generate the singing piece; The singing audio is input into the trained audio denoising model, and the audio denoising audio is processed by the audio denoising model to obtain the denoised singing audio; the audio denoising model is trained by the audio denoising model training method as described in any one of claims 1 to 6. A singing piece is generated using the accompanying audio and the noise-reduced vocal audio.

8. The method according to claim 7, characterized in that, Before inputting the singing audio into the trained audio noise reduction model, the method further includes: Obtain the clean audio of the target user corresponding to the singing audio, and pre-store the clean audio into the timbre feature extraction module of the trained audio noise reduction model; the timbre feature extraction module is used to extract the timbre features of the clean audio as the timbre features of the target user; The process of inputting the singing audio into a trained audio denoising model and performing denoising processing on the singing audio using the audio denoising model to obtain denoised singing audio includes: The singing audio is input into the trained audio noise reduction model, and noise reduction processing is performed on the singing audio based on the timbre characteristics of the target user to obtain the noise-reduced singing audio.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images