Method for adjusting human voice in song

Abstract

A method and apparatus for adjusting human voice in a song, and a terminal device, a medium and a product. The method comprises: converting a first time-domain signal in a song into a first time-frequency-domain signal (102); performing feature extraction on the first time-frequency-domain signal, so as to obtain a time-frequency-domain audio feature (104); inputting the time-frequency-domain audio feature into a human voice extraction model, so as to obtain a time-frequency mask outputted by the human voice extraction model, wherein the time-frequency mask is used for representing the strength of a human voice signal at corresponding time and frequency points in the first time-frequency-domain signal (106); on the basis of the time-frequency mask, adjusting the first time-frequency-domain signal, so as to obtain a second time-frequency-domain signal (108); and converting the second time-frequency-domain signal into a second time-domain signal, so as to obtain an adjusted song (110).

Description

How to adjust vocals in a song Cross-reference to related applications

[0001] This application claims priority to Chinese patent application No. 202411396078.X, filed with the Chinese Patent Office on October 8, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of this application relate to, but are not limited to, the field of audio technology, and particularly to, but are not limited to, methods for adjusting vocals in songs. Background Technology

[0003] In the song production and distribution process, the audio source of a song usually needs to be recorded in a professional recording studio. Each song will be recorded separately as a full version including vocals and accompaniment, as well as a version with only accompaniment. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims. According to a first aspect of any embodiment of this application, a method for adjusting vocals in a song is provided, the method being applied to a terminal device, the method comprising: converting a first time-domain signal in a song into a first time-frequency domain signal; extracting features from the first time-frequency domain signal to obtain time-frequency domain audio features; inputting the time-frequency domain audio features into a vocal extraction model to obtain a time-frequency mask output by the vocal extraction model; the time-frequency mask representing the intensity of the vocal signal at corresponding time and frequency points in the first time-frequency domain signal; adjusting the first time-frequency domain signal based on the time-frequency mask to obtain a second time-frequency domain signal; and converting the second time-frequency domain signal into a second time-domain signal to obtain the adjusted song.

[0005] According to a second aspect of any embodiment of this application, a vocal adjustment device for a song is provided. The device is applied to a terminal device and includes: a time-frequency domain conversion module configured to convert a first time-domain signal in a song into a first time-frequency domain signal; a feature extraction module configured to extract features from the first time-frequency domain signal to obtain time-frequency domain audio features; a vocal extraction module configured to input the time-frequency domain audio features into a vocal extraction model to obtain a time-frequency mask output by the vocal extraction model; the time-frequency mask is used to represent the intensity of the vocal signal at corresponding time and frequency points in the first time-frequency domain signal; a vocal adjustment module configured to adjust the first time-frequency domain signal based on the time-frequency mask to obtain a second time-frequency domain signal; and a time-domain conversion module configured to convert the second time-frequency domain signal into a second time-domain signal to obtain the adjusted song.

[0006] According to a third aspect of any embodiment of this application, a terminal device is provided, comprising: a processor; and a memory configured to store processor-executable instructions; wherein the processor implements the method described in any embodiment of this application by executing the executable instructions.

[0007] According to a fourth aspect of any embodiment of the present application, a computer-readable storage medium is provided having computer instructions stored thereon that, when executed by a processor, implement the method described in any of the embodiments of the present application described above.

[0008] According to a fifth aspect of any embodiment of this application, a computer program product is provided, having a computer program / instructions stored thereon, which, when executed by a processor, implement the method described in any of the embodiments of this application described above.

[0009] The technical solution provided by this application can include the following beneficial effects: As can be seen from the above embodiments, by converting the first time-domain signal in the song into a first time-frequency domain signal, extracting features from the first time-frequency domain signal to obtain time-frequency domain audio features, inputting the time-frequency domain audio features into the human voice extraction model to obtain the time-frequency mask output by the human voice extraction model, adjusting the first time-frequency domain signal based on the time-frequency mask to obtain a second time-frequency domain signal, converting the second time-frequency domain signal into a second time-domain signal to obtain the adjusted song, the human voice signal in the song can be separated and adjusted, and the human voice volume in the song can be adjusted in real time on the terminal device.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Other aspects will become clear after reading and understanding the accompanying drawings and detailed description. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this application, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0012] Figure 1 is a flowchart illustrating a method for adjusting vocals in a song according to an exemplary embodiment of this application.

[0013] Figure 2 is a schematic diagram of the application layer and system layer of a terminal device according to an exemplary embodiment of this application.

[0014] Figure 3 is a schematic diagram of the structure of an adjustment module for a first time-domain signal according to an exemplary embodiment of this application.

[0015] Figure 4 is a schematic diagram illustrating an FBank feature according to an exemplary embodiment of this application.

[0016] Figure 5 is a schematic diagram of the structure of a human voice extraction model according to an exemplary embodiment of this application.

[0017] Figure 6 is a schematic diagram of an analog audio signal according to an exemplary embodiment of this application.

[0018] Figure 7 is a flowchart illustrating another method for adjusting vocals in a song according to an exemplary embodiment of this application.

[0019] Figure 8 is a schematic diagram of the structure of a terminal device according to an exemplary embodiment of this application.

[0020] Figure 9 is a block diagram of a song vocal adjustment device according to an exemplary embodiment of this application. Detailed Implementation

[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0022] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0023] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0024] Currently, songs are usually complete versions with vocals and accompaniment. Therefore, in scenarios such as karaoke, audio editing, and music production, it is not possible to directly adjust the vocals in a song.

[0025] This application proposes a method for adjusting vocals in a song. To further illustrate this application, the following embodiments are provided.

[0026] Please refer to Figure 1, which is a flowchart illustrating a method for adjusting vocals in a song according to an exemplary embodiment of this application. This method can be applied to terminal devices such as in-vehicle systems, smart speakers, mobile phones, computers, messaging devices, tablets, and personal digital assistants, and can be used in scenarios such as karaoke, audio editing, and music production.

[0027] As shown in Figure 1, the method for adjusting the vocals of this song may include steps 102 to 110.

[0028] Step 102: Convert the first time-domain signal in the song into the first time-frequency domain signal.

[0029] In this step, the terminal device can acquire the first time-domain signal of a frame from the song. The first time-domain signal is the sound wave signal of the song in the time domain. It is used to describe the changes of the song on the time axis, characterize the value or amplitude of the song at different points in time, and reveal information such as the waveform, amplitude, and periodicity of the song.

[0030] The first time-domain signal can be a single frame of a song. A single frame is a very small, continuous time segment in the audio stream of a song, for example, 10ms. A single frame contains several digital samples, which are arranged in chronological order to form the time-domain representation of the first time-domain signal.

[0031] Terminal devices can use methods such as Short Time Fourier Transform (STFT), Mel-Frequency Cepstral Coefficients (MFCC), and Constant-Q Transform (CQT) to convert the first time-domain signal to the time-frequency domain, thereby obtaining the first time-frequency domain signal corresponding to the first time-domain signal.

[0032] Among them, the first time-frequency domain signal is used to describe the characteristics of the song when considering both time and frequency information. In the time-frequency domain, we can focus on the changes of the song at different time points and different frequencies, which can provide a more comprehensive description of the signal characteristics.

[0033] For example, the frequency of a song may change over time. By using the first time-frequency domain signal, both aspects of information can be considered simultaneously, which helps to understand and analyze the song more accurately.

[0034] Step 104: Extract features from the first time-frequency domain signal to obtain time-frequency domain audio features.

[0035] In this step, the terminal device can extract the time-frequency domain audio features from the first time-frequency domain signal by analyzing spectrograms, deep learning models, and other methods.

[0036] Among them, time-frequency domain audio features are parameters or indicators that describe the characteristics of a song and are used for in-depth analysis and processing of the song. These features can include the spectrum, power spectrum, filter bank (FBank) features, etc.

[0037] Time-frequency domain audio feature extraction is essentially a process of refining and simplifying the first time-frequency domain signal based on objective physical or statistical laws. Its purpose is to highlight information useful for subsequent machine learning algorithms in voice extraction models, while compressing and removing useless or redundant information from the song, so that the subsequent model can better complete the machine learning task.

[0038] Step 106: Input the time-frequency domain audio features into the human voice extraction model to obtain the time-frequency mask output by the human voice extraction model; the time-frequency mask is used to represent the intensity of the human voice signal at the corresponding time and frequency points in the first time-frequency domain signal.

[0039] In this step, the terminal device inputs the time-frequency domain audio features into the voice extraction model. The voice extraction model is a trained deep convolutional neural network (DCNN) model that can identify the human voice components in a song and extract the time-frequency mask from the first time-frequency domain signal.

[0040] The terminal device obtains the time-frequency mask in the first time-frequency domain signal output by the voice extraction model. The time-frequency mask is used to represent the intensity of the voice signal at the corresponding time and frequency point in the first time-frequency domain signal. The voice signal is the voice component of the song in the time-frequency domain, including information such as the singer's voice characteristics, pitch, timbre, and volume.

[0041] The time-frequency masking is a two-dimensional matrix with the same size as the input time-frequency domain signal. In the time-frequency masking, the value of each element represents the intensity or presence of the human voice signal at the corresponding time and frequency point.

[0042] For example, the value of each element in the time-frequency mask can be normalized to between 0 and 1, where 1 represents that it belongs entirely to the human voice signal and 0 represents that it does not belong to the human voice signal at all.

[0043] Step 108: Adjust the first time-frequency domain signal based on time-frequency masking to obtain the second time-frequency domain signal.

[0044] In this step, the terminal device adjusts the first time-frequency domain signal based on time-frequency masking, such as suppressing the volume of human voice and retaining only human voice, to obtain the second time-frequency domain signal. The second time-frequency domain signal is the time-frequency domain signal after adjusting the first time-frequency domain signal for human voice.

[0045] For example, the terminal device can apply the time-frequency masking output by the voice extraction model to the first time-frequency domain signal, thereby eliminating the accompaniment in the first time-frequency domain signal and retaining only the voice signal in the first time-frequency domain signal. The second time-frequency domain signal retaining only the voice is obtained according to the following formula (1): y(k,τ)=g(k,τ)x(k,τ) (1)

[0046] Where y(k,τ) represents the second time-frequency domain signal that retains only the human voice, g(k,τ) represents the time-frequency masking output by the human voice extraction model, x(k,τ) represents the first time-frequency domain signal, k represents the frequency band index, and τ represents the data block index.

[0047] The terminal device can also reduce the time-frequency masking of the voice extraction model output, and apply the reduced time-frequency masking to the first time-frequency domain signal corresponding to the song, adjusting the voice signal in the first time-frequency domain signal, thereby suppressing the volume of the voice in the song. The second time-frequency domain signal after voice volume suppression is obtained according to the following formula (2): y = dg s x+(1-g s x = [1 - (1 - d)g] s ]x (2)

[0048] Where y represents the second time-frequency domain signal after human voice volume suppression, g s denoted by x, the time-frequency masking output by the voice extraction model, and d, the suppression magnitude of the time-frequency masking, which ranges from [0,1].

[0049] Taking the suppression range of vocal volume in a song as [-40, 0] dB as an example, when d is 0, the suppression amplitude of time-frequency masking reaches its maximum, the suppression of vocal volume is -40 dB, and only the accompaniment is retained in the second time-frequency domain signal; when d is 1, the suppression amplitude of time-frequency masking reaches its minimum, the suppression of vocal volume is 0 dB, and the vocal volume in the second time-frequency domain signal is not suppressed.

[0050] It is understood that the terminal device can also perform operations such as boosting the time-frequency masking according to the actual business scenario to obtain the second time-frequency domain signal after the human voice signal is boosted. This application embodiment does not limit this.

[0051] In one embodiment, the terminal device can receive a user-defined voice adjustment ratio through its interface. Based on the user-defined voice adjustment ratio, the suppression amount of time-frequency masking is determined.

[0052] Among them, the vocal adjustment ratio is the proportion of vocal volume adjusted in a song, used to indicate the amount of suppression in time-frequency masking.

[0053] Taking formula (2) above as an example, the terminal device sets the suppression amount of time-frequency masking according to the human voice adjustment ratio set by the user. Based on the suppression amount of time-frequency masking, the human voice signal in the first time-frequency domain signal is suppressed.

[0054] For example, if the user-defined voice adjustment ratio is 30% lower than the voice volume, the suppression amplitude d for the time-frequency masking of the voice can be set to 0.7, thereby obtaining a second time-frequency domain signal with 30% suppression of the voice volume.

[0055] As described above, by determining the suppression amount of time-frequency masking based on the received user-set vocal adjustment ratio, and adjusting the vocal signal in the first time-frequency domain signal based on the time-frequency masking and the suppression amount of time-frequency masking, the volume of vocals in the song can be adjusted according to the user's needs, providing the user with a more personalized music experience.

[0056] Step 110: Convert the second time-frequency domain signal into a second time-domain signal to obtain the adjusted song.

[0057] In this step, the terminal device uses methods such as Inverse Short-Time Fourier Transform (ISTFT), Overlap-Add Method, and Overlap-Save Method to convert the second time-frequency domain signal into a second time-domain signal, thus obtaining the adjusted vocal frame of the song. The second time-domain signal is the time-domain signal obtained by converting the second time-frequency domain signal back to the time domain.

[0058] The vocal adjustment method in this embodiment converts a first time-domain signal in the song into a first time-frequency domain signal, extracts features from the first time-frequency domain signal to obtain time-frequency domain audio features, inputs the time-frequency domain audio features into a vocal extraction model to obtain a time-frequency mask output by the vocal extraction model, adjusts the first time-frequency domain signal based on the time-frequency mask to obtain a second time-frequency domain signal, converts the second time-frequency domain signal into a second time-domain signal to obtain the adjusted song. This method can separate and adjust the vocal signals in the song, enabling real-time adjustment of the vocal volume in the song on a terminal device.

[0059] Furthermore, the streaming acquisition of the first time-domain signal of a frame and the adjustment of the vocal signal of that frame have low output latency, thereby enabling real-time adjustment of the vocals in the song. At the same time, it consumes low computing power and storage resources, requiring only limited hardware resources in the terminal device to adjust the vocal components in the song.

[0060] In the foregoing embodiments, a time-frequency mask was extracted from the first time-frequency domain signal using a voice extraction model, and then applied to the first time-frequency domain signal to adjust the voice signal in the song. The following embodiments will provide a more detailed explanation of the application scenarios for adjusting the voice in songs, and these embodiments are applicable to any of the above embodiments.

[0061] In one embodiment, the terminal device may include a system layer and an application layer. The system layer deploys a voice extraction model, and the application layer deploys a music application. When playing a song, the music application in the application layer calls the interface of the system layer for playback. The terminal device can receive the first temporal signal of each frame sent by the music application from the interface of the system layer.

[0062] The system layer can convert the received first time-domain signal into a first time-frequency domain signal, extract features from the first time-frequency domain signal, and obtain time-frequency domain audio features. The system layer uses a human voice extraction model to obtain the time-frequency mask corresponding to the first time-frequency domain signal.

[0063] The system layer adjusts the vocal signal in the first time-frequency domain signal based on time-frequency masking to obtain the second time-frequency domain signal. The second time-frequency domain signal is then converted into a second time-domain signal to obtain the adjusted song.

[0064] Please refer to Figure 2, which shows a schematic diagram of the application layer and system layer of a terminal device. The terminal device may include an application layer 20, a system layer 21, a microphone 22, and a speaker 23.

[0065] The application layer 20 may include multiple music applications such as the first music application 201, the second music application 202, and the third music application 203.

[0066] The system layer 21 may include: adjustment module 211, environmental voice extraction module 212, mixing module 213, and frequency division module 214.

[0067] When a user plays a song using the third music application 203, the third music application 203 calls the system layer 21, and the adjustment module 211 receives a first time domain signal with a 48kHz sampling rate from a frame of the song sent by the third music application 203.

[0068] The adjustment module 211 converts the first time-domain signal into a first time-frequency domain signal. The time-frequency mask in the first time-frequency domain signal is extracted by the human voice extraction model set in the adjustment module 211. The human voice signal in the first time-frequency domain signal is adjusted based on the time-frequency mask. Then, the adjusted first time-frequency domain signal is converted into a time-domain signal, that is, a second time-domain signal with a sampling rate of 48kHz is obtained.

[0069] As described above, by receiving the first time-domain signal sent by the music application in the application layer, the terminal device does not need to obtain song resources in advance, and can adjust the vocals of the song simply by using the song played by the music application in the application layer.

[0070] Furthermore, by receiving the first time-domain signal from the song sent by the music application, adjusting the vocals, and then continuing to play the second time-domain signal (the adjusted first time-domain signal) through the audio path, the process is seamless for the music application at the application layer, and the music application does not need to make any changes.

[0071] In one embodiment, the terminal device can acquire ambient audio signals via a microphone. The acquired ambient audio signals are analyzed, and audio processing such as echo cancellation, feedback suppression, noise reduction, reverb addition, sound effects, and upsampling are performed to obtain a clean sing-along signal from the ambient audio signal. The second time-domain signal and the sing-along signal are then mixed to obtain the song sung by the user.

[0072] Among them, music applications are applications on terminal devices that have song playback functions.

[0073] Ambient audio signals are sound wave signals captured by a microphone from the environment surrounding the terminal device, containing key information such as the frequency, amplitude, and phase of the ambient sounds. Sing-along signals are the audio signals formed by the user singing along to a song within the ambient audio signal; they represent the vocal portion of the ambient audio signal.

[0074] Please refer to Figure 2. The terminal device collects ambient audio signals around the terminal device through microphone 22. The ambient audio signals include a 16kHz sampling rate vocal track and echo.

[0075] The environmental voice extraction module 212 performs audio processing on the environmental audio signal, including echo cancellation, howl suppression, noise reduction, reverb addition, sound effects, and upsampling, to obtain a clean sing-along signal with a sampling rate of 48kHz in the environmental audio signal.

[0076] The mixing module 213 mixes the second time domain signal and the backing vocal signal in the ambient audio signal to obtain a song with human backing vocals. The song with human backing vocals includes a backing vocal signal with a sampling rate of 48kHz, the song accompaniment, and the adjusted vocals.

[0077] The crossover module 214 divides the song sung by vocals into multiple frequency bands, such as low, mid, and high frequencies, which are then played through different speakers 23 to provide a richer and more realistic listening experience.

[0078] As described above, by analyzing the ambient audio signal collected from the terminal device, the singing signal in the ambient audio signal is obtained. The second time domain signal and the singing signal are mixed to obtain the time domain signal of the user singing along. This method has real-time performance and low resource consumption, and realizes the karaoke function of the original singer singing along softly.

[0079] In the foregoing embodiments, a karaoke function with a soft lead vocal was achieved by adjusting the first time-domain signal of a song in a music application. In the following embodiments, the various modules in the terminal device that implement vocal adjustment will be described in detail, and these embodiments are applicable to any of the above embodiments.

[0080] Please refer to Figure 3, which shows a schematic diagram of the structure of an adjustment module for a first time-domain signal. The adjustment module 211 may include: an STFT module 2110, an audio feature extraction module 2111, a human voice extraction model 2112, a human voice ratio adjustment module 2113, and an ISTFT module 2114.

[0081] The STFT module 2110 can perform a short-time Fourier transform on the first time-domain signal in the song to obtain a first time-frequency domain signal. The first time-frequency domain signal is then input to the audio feature extraction module 2111 and the vocal proportion adjustment module 2113. The audio feature extraction module 2111 extracts the time-frequency domain audio features from the first time-frequency domain signal.

[0082] The time-frequency domain audio features are input into the human voice extraction model 2112, and the human voice extraction model 2112 sends the extracted time-frequency mask to the human voice ratio adjustment module 2113.

[0083] The human voice ratio adjustment module 2113 determines the suppression amount of time-frequency masking based on the human voice adjustment ratio set or preset by the user, and applies the time-frequency masking and the suppression amount of time-frequency masking to the first time-frequency domain signal to adjust the human voice signal in the first time-frequency domain signal and obtain the second time-frequency domain signal.

[0084] The ISTFT module 2114 performs a short-time inverse Fourier transform on the second time-frequency domain signal, converting it into a second time-domain signal to obtain the song with adjusted vocals.

[0085] As described above, by performing a short-time Fourier transform on the first time-domain signal and then an inverse short-time Fourier transform on the second time-frequency domain signal, the second time-domain signal is obtained, thereby capturing the characteristics of the song in both time and frequency dimensions. Furthermore, the short-time Fourier transform and the inverse short-time Fourier transform can improve the computational efficiency of converting from the time domain to the time-frequency domain and then back to the time domain, further ensuring the real-time performance of the song's vocal adjustments.

[0086] In one embodiment, the time-frequency domain audio features may include FBank features. The audio feature extraction module 2111 first determines the number of Mel filters in the Mel filter bank, preferably between twenty and forty Mel filters. Based on the sampling rate of the first time-frequency domain signal, the frequency range to be analyzed is determined, and the Mel filter bank is set.

[0087] The linear frequency range is converted to the Mel frequency range, and intervals are uniformly divided on the Mel scale according to the selected number of filters. Each interval corresponds to a Mel filter. Then, each Mel interval is converted to the corresponding linear frequency range, and each Mel filter corresponds to a linear frequency band, resulting in a Mel filter bank.

[0088] Using a predefined Mel filter bank, the linear spectrum of the first time-frequency domain signal is filtered. The energy of the output of each Mel filter is calculated to obtain the Mel spectral energy of the first time-frequency domain signal. The natural logarithm of the Mel spectral energy is then taken to obtain the filter bank characteristics.

[0089] The Mel filter bank includes a series of triangular Mel filters distributed along the Mel scale, used to map linear spectra to nonlinear Mel spectra based on auditory perception.

[0090] The Mel scale can simulate the nonlinear characteristics of human hearing. The human ear has a high resolution for low-frequency components, and the perception of resolution gradually decreases as the frequency increases.

[0091] Mel spectral energy is obtained by calculating the output energy of each Mel filter and is used to simulate the nonlinear perception of frequency by the human auditory system.

[0092] For example, the audio feature extraction module 2111 can convert the frequency between the linear scale h (Hz) and the Mel scale m (Mel) according to the following formulas (3) and (4): h = 700 × (10 m / 2595 -1) (4)

[0093] Where m represents the Mel scale and h represents the linear scale.

[0094] Please refer to Figure 4, which shows a schematic diagram of an FBank feature. Taking a sampling rate of 48kHz for the first time-frequency domain signal as an example, a Mel filter bank covering the entire 0 to 24kHz frequency band can be set according to formulas (3) and (4).

[0095] The start and end frequencies of the 120-dimensional FBank feature in Figure 4 are set to h1 = 0 and h2 = 24000, respectively. According to formula (3), the start and end points on the Mel scale are m1 = 0 and m2 = 4016.

[0096] Since the 120-dimensional FBank feature corresponds to 120 peaks in a triangular window, and there is half overlap between adjacent triangular windows, the interval [m1, m2] needs to be evenly divided into 121 parts, requiring a total of 122 dividing points. Each part corresponds to half a triangular window, and its coverage area is...

[0097] Starting from the first boundary point in the interval [m1,m2], the three adjacent boundary points correspond to the start, center, and cutoff frequencies of a triangular window, respectively. Then, according to formula (4), the boundary points are converted back to the linear scale to obtain the non-uniform interval. After that, triangular windows are set up in groups of three boundary points, and each triangular window is a Mel filter. A series of Mel filters form a Mel filter bank.

[0098] By applying each Mel filter in the Mel filter bank to the linear spectrum and calculating the output energy of each filter, the Mel spectral energy of the first time-frequency domain signal is obtained. Taking the natural logarithm of the Mel spectral energy yields the FBank feature of the window structure shown in Figure 4.

[0099] As described above, by setting a Mel filter bank based on the sampling rate of the first time-frequency domain signal, the linear spectrum of the first time-frequency domain signal is filtered using the Mel filter bank to obtain the Mel spectral energy of the first time-frequency domain signal. The natural logarithm of the Mel spectral energy is then calculated to obtain the filter bank features. Based on the nonlinear perception characteristics of frequency by the human auditory system, the extracted features are more consistent with the actual situation of human hearing, thus improving the accuracy of vocal adjustment in songs.

[0100] In one embodiment, the human voice extraction model may include: a nonlinear transformation unit, a multi-layer convolution unit, and a masking processing unit.

[0101] Terminal devices can splice the time-frequency domain audio features of multiple channels, such as stereo channels, through methods such as serialization and fusion to obtain spliced ​​features. Among them, the time-frequency domain audio features are the feature data of the first time-frequency domain signal on multiple channels.

[0102] The terminal device inputs the spliced ​​features into a nonlinear transformation unit, which performs a nonlinear transformation on the spliced ​​features to obtain nonlinear transformed data. This nonlinear transformed data is used to enhance the model's nonlinear expressive power, helping the model learn more complex audio feature representations.

[0103] The nonlinear transformation unit inputs the nonlinear transformation data into a multi-layer convolutional unit. The multi-layer convolutional unit performs convolution processing on the nonlinear transformation data. By stacking multiple convolutional layers, a deeper network structure is built to extract local features and reduce the spatial dimension of the features, thus obtaining filtered data.

[0104] Multi-layer convolutional units input filtered data into the masking unit, which calculates and outputs a time-frequency mask based on the filtered data. In the fully connected layers, each input node in the masking unit is connected to an output node via weights, and a bias term is added.

[0105] Through forward propagation, the values ​​of each output node are calculated, corresponding to the real and imaginary parts of the time-frequency mask. An activation function is then applied to the output of the fully connected layer to obtain the final time-frequency mask composed of the real and imaginary parts.

[0106] The time-frequency masking is in complex form, consisting of a real part and an imaginary part. The real and imaginary parts in the time-frequency masking together describe the characteristics of the first time-frequency domain signal in the time-frequency domain.

[0107] Please refer to Figure 5, which shows a schematic diagram of a human voice extraction model. The human voice extraction model may include: a nonlinear transformation unit 50, a multi-layer convolutional unit 51, and a masking processing unit 52.

[0108] The nonlinear transformation unit 50 may include a fully connected layer (Affine) and a Rectified Linear Unit (ReLU) activation function. The masking processing unit 52 may include a fully connected layer (Affine) and a tanh activation function.

[0109] The multi-layer convolutional unit 51 may include a multi-layer feedforward sequential memory network (FSMN) unit, and each FSMN unit may include: a linear function 510, an FSMN module 511, and a ReLU (Affine) 512.

[0110] Linear function 510 is used to receive input data and generate output through linear transformation. ReLU (Affine) 512 is used to perform nonlinear transformation on the output of FSMN module 511.

[0111] The FSMN module 511 consists of convolutional structures, which is equivalent to the frequency-band filtering operation in frequency domain signal processing algorithms. It performs one-dimensional convolution on each feature dimension of the concatenated features.

[0112] In the fully connected layer, the nonlinear transformation unit 50 uses the ReLU activation function to perform a nonlinear transformation on the concatenated features to obtain nonlinear transformed data. The multi-layer convolutional unit 51 uses multi-layer FSMN units to perform convolution processing on the nonlinear transformed data to obtain filtered data.

[0113] The FSMN module 511 can obtain the convolutional data according to the following formula (5):

[0114] Where x represents the input data of FSMN module 511, y represents the output data of FSMN module 511, w and v represent the filter coefficients learned by the neural network, and i represents the data dimension. represents the frame index, T represents the convolution length, and t represents the time index.

[0115] In the fully connected layer, the masking processing unit 52 calculates the value of each output node based on the filtered data through forward propagation, and uses the tanh activation function to output a time-frequency mask composed of real and imaginary parts.

[0116] It is understood that the structure of the voice extraction model shown in Figure 5 is only an example, and other neural network structures can also be used, as long as the voice extraction model can achieve the song voice adjustment method described in the embodiments of this application. The embodiments of this application do not limit this.

[0117] As described above, by splicing the time-frequency domain audio features, spliced ​​features are obtained. The nonlinear transformation unit 50 performs nonlinear transformation on the spliced ​​features to obtain nonlinear transformation data. The multi-layer convolution unit 51 performs convolution processing on the nonlinear transformation data to obtain filtered data, which can further accelerate the model calculation speed and reduce the computational complexity. Based on the filtered data, the masking processing unit 52 calculates and outputs a time-frequency mask composed of real and imaginary parts. The time-frequency mask composed of real and imaginary parts can more accurately distinguish the time-frequency domain audio features of different sound sources, thereby more effectively reducing the influence of background noise, accompaniment music and other interference components on the time-frequency domain audio features and improving the accuracy of human voice extraction.

[0118] In one embodiment, the terminal device can use pre-constructed training data to train an initial voice extraction model. The training data is input into the initial voice extraction model, which then performs calculations through its network structure and outputs a predicted time-frequency mask. This predicted time-frequency mask is used to obtain the voice signal predicted by the initial voice extraction model from the training data.

[0119] The loss value of the loss function is calculated based on the temporal and frequency domain prediction losses of the initial human voice extraction model for the training data.

[0120] By using optimization algorithms such as Stochastic Gradient Descent (SGD) and Adam, the parameters of the initial human voice extraction model are adjusted to reduce the loss value of the loss function. The above steps of time-frequency masking prediction, loss value calculation and parameter update are repeated until the loss value is minimized, and the human voice extraction model is obtained.

[0121] The training data includes multiple audio samples, each containing mixed audio (e.g., a mixture of accompaniment and vocals) and the corresponding vocal signal, used to train the model to learn how to extract the vocal signal from the mixed audio.

[0122] The temporal prediction loss is the difference in the time domain between the human voice signal in the training data and the predicted human voice signal obtained by the prediction time-frequency mask based on the output of the initial human voice extraction model. The frequency domain prediction loss is the difference in the frequency domain between the human voice signal in the training data and the predicted human voice signal obtained by the prediction time-frequency mask.

[0123] For example, the loss value of the loss function can be calculated according to the following formula (6):

[0124] Where Loss represents the loss value of the loss function, y represents the prediction time-frequency masking, and s represents the real human voice signal in the training data. This represents the average value of the time-domain prediction loss. This represents the average value of the frequency domain prediction loss. k represents the frequency band index, τ represents the data block index, and t represents the time index.

[0125] The loss function mentioned above also implicitly includes the loss of the phase information of the predicted human voice signal. The phase information of the predicted human voice signal can also be continuously optimized as the model is trained. When the loss value is minimized, the phase information of the predicted human voice signal will match the phase information of the human voice signal in the training data more closely.

[0126] As described above, the initial voice extraction model is trained using the constructed training data. The loss value of the loss function is calculated based on the time-domain prediction loss and the frequency-domain prediction loss. By adjusting the parameters of the initial voice extraction model, the loss value is minimized to obtain the voice extraction model. Since the loss function considers the prediction loss in both the time and frequency domains, and the phase information of the predicted voice signal output when the loss value is minimized is more accurate, the trained voice extraction model can extract more accurate time-frequency masks.

[0127] In one embodiment, in order to enhance the stability and generalization of the voice extraction model to cover more and more comprehensive application scenarios, it is necessary to use a variety of massive amounts of data to train the initial voice extraction model.

[0128] The training data may include at least one of the following: analog audio signals, audio signals synchronized with audio tracks.

[0129] By splicing together various previously stored human voice and accompaniment sound sources, and then adding reverb, noise, nonlinear processing, and adjusting the signal-to-noise ratio, analog audio signals under various conditions collected in different environments are simulated.

[0130] Please refer to Figure 6, which shows a schematic diagram of an analog audio signal. Randomly selected human voice sources and accompaniment sources are subjected to audio processing such as voice changing, speech rate changing, and short sentence splicing to generate long audio recordings of both the human voice and accompaniment sources.

[0131] Using the transfer functions in the transfer function library or the transfer functions simulated by the transfer function, convolve the long audio of the human voice source and the long audio of the accompaniment source respectively to obtain the human voice signal and the accompaniment signal.

[0132] Among them, the transfer function library refers to a library containing a variety of transfer functions, which can be directly applied to modify the long audio of the human voice source and the long audio of the accompaniment source.

[0133] Transfer function simulation refers to using mathematical models or simulation software to simulate a transfer function, enabling the simulation of audio effects without actually building a physical system. By adjusting the parameters in the simulation, it ensures that long audio recordings of vocal and accompaniment sources achieve the desired audio effects in practical applications.

[0134] The vocal and accompaniment signals are randomly adjusted for signal-to-noise ratio and volume to obtain a simulated audio signal from a multi-channel microphone. This simulated audio signal can then be used to train the initial vocal extraction model using model training tools such as TensorFlow and PyTorch.

[0135] Terminal devices can also utilize the Teacher-Student mode to select a Teacher model (such as a high-quality pre-trained model like Demucs) to process large amounts of audio resources and separate various instrument components. The audio resources can be unseparated audio files.

[0136] The separated instrument components are then synchronized. This can be achieved by comparing and analyzing the timecode of the audio signal or by using the phase information of the audio signal. Furthermore, rhythm analysis can be used to ensure that the rhythms of each track match each other, generating a synchronized audio signal. In this synchronized audio signal, each track maintains precise synchronization in time and rhythm.

[0137] The initial voice extraction model was used as the Student model, and the initial voice extraction model was trained using audio signals synchronized with the audio track.

[0138] The training data can also include open-source datasets such as MUSDB18-HQ that contain synchronized song data.

[0139] As mentioned above, training the initial voice extraction model with simulated audio signals obtained by audio processing of the human voice source and the accompaniment source can obtain diverse training data more economically and efficiently. Training the initial voice extraction model with audio signals generated by separating and synchronizing the instrument components of the audio resources can help the voice extraction model extract human voices more accurately in complex musical environments.

[0140] To further illustrate the vocal adjustment process, Figure 7 shows a flowchart of another vocal adjustment method. This vocal adjustment method may include steps 702 to 724.

[0141] Step 702: Train the initial human voice extraction model using the training data.

[0142] In this step, the terminal device uses pre-constructed training data, such as analog audio signals and audio signals synchronized with the audio track, to train the initial human voice extraction model.

[0143] Step 704: Calculate the loss value of the loss function based on the time-domain prediction loss and the frequency-domain prediction loss.

[0144] In this step, the terminal device calculates the loss value of the loss function based on the temporal prediction loss and frequency domain prediction loss between the human voice signal in the training data and the predicted human voice signal (the predicted human voice signal is obtained based on the predicted time-frequency masking output by the initial human voice extraction model).

[0145] Step 706: The terminal device minimizes the loss value by adjusting the parameters of the initial human voice extraction model to obtain the human voice extraction model.

[0146] Step 708: Receive the first time-domain signal sent by the music application.

[0147] In this step, the system layer in the terminal device receives the first time-domain signal from the song sent by the music application in the application layer.

[0148] Step 710: Perform a short-time Fourier transform on the first time-domain signal to obtain the first time-frequency domain signal.

[0149] In this step, the terminal device performs a short-time Fourier transform on the first time-domain signal to obtain the first time-frequency domain signal of the first time-domain signal in the time-frequency domain.

[0150] Step 712: The terminal device extracts features from the first time-frequency domain signal to obtain FBank features.

[0151] In this step, the terminal device sets up a Mel filter bank based on the sampling rate of the first time-frequency domain signal, uses the Mel filter bank to filter the linear spectrum of the first time-frequency domain signal to obtain the Mel spectrum energy of the first time-frequency domain signal, and calculates the natural logarithm of the Mel spectrum energy to obtain the FBank feature.

[0152] Step 714: The terminal device splices the FBank features of the multi-channel array to obtain the spliced ​​features.

[0153] Step 716: Input the spliced ​​features into the voice extraction model to obtain the time-frequency mask output by the voice extraction model.

[0154] In this step, the nonlinear transformation unit in the human voice extraction model performs nonlinear transformation on the splicing features to obtain nonlinear transformation data. The multilayer convolution unit performs convolution processing on the nonlinear transformation data to obtain filtered data. Based on the filtered data, the masking processing unit calculates and outputs a time-frequency mask composed of real and imaginary parts.

[0155] Step 718: Adjust the first time-frequency domain signal based on time-frequency masking according to the set human voice adjustment ratio.

[0156] In this step, the terminal device determines the suppression amount of time-frequency masking corresponding to the first time-frequency domain signal based on the received set human voice adjustment ratio. The time-frequency masking and the suppression amount of time-frequency masking are applied to the first time-frequency domain signal to adjust the human voice signal in the first time-frequency domain signal, thereby generating the second time-frequency domain signal.

[0157] Step 720: The terminal device performs a short-time inverse Fourier transform on the second time-frequency domain signal to obtain the second time-domain signal.

[0158] Step 722: Analyze the ambient audio signal collected from the terminal device to obtain the singing signal in the ambient audio signal.

[0159] In this step, the terminal device performs audio processing such as echo cancellation, noise reduction, and reverb on the environmental audio signal collected from the terminal device to obtain the sing-along signal in the environmental audio signal.

[0160] Step 724: Mix the second time domain signal and the sing-along signal to obtain the song sung by the user.

[0161] In this step, the terminal device mixes the second time domain signal and the sing-along signal to obtain the song sung by the user, thereby enabling real-time adjustment of the vocal volume of the song played in the music application.

[0162] Figure 8 is a schematic diagram of a terminal device according to an exemplary embodiment of this application. This terminal device may be, for example, a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, personal digital assistant, server, smart home appliance, in-vehicle system, etc. Referring to Figure 8, at the hardware level, the terminal device includes a processor 802, an internal bus 804, a network interface 806, memory 808, and non-volatile memory 810, and may also include other hardware required for services. The processor 802 reads the corresponding computer program from the non-volatile memory 810 into the memory 808 and then runs it, forming a song vocal adjustment device at the logical level. Of course, besides software implementation, this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0163] Figure 9 is a block diagram of a song vocal adjustment device according to an exemplary embodiment of this application. Referring to Figure 9, the device may include: a time-frequency domain conversion module 902, a feature extraction module 904, a vocal extraction module 906, a vocal adjustment module 908, and a time-domain conversion module 910, wherein: the time-frequency domain conversion module 902 is configured to convert a first time-domain signal in the song into a first time-frequency domain signal; the feature extraction module 904 is configured to perform feature extraction on the first time-frequency domain signal to obtain time-frequency domain audio features; the vocal extraction module 906 is configured to input the time-frequency domain audio features into a vocal extraction model to obtain a time-frequency mask output by the vocal extraction model; the time-frequency mask is used to represent the intensity of the vocal signal at the corresponding time and frequency points in the first time-frequency domain signal; the vocal adjustment module 908 is configured to adjust the first time-frequency domain signal based on the time-frequency mask to obtain a second time-frequency domain signal; and the time-domain conversion module 910 is configured to convert the second time-frequency domain signal into a second time-domain signal to obtain the adjusted song.

[0164] In one example, the terminal device includes a system layer and an application layer, wherein the system layer deploys the human voice extraction model and the application layer deploys a music application; the time-frequency domain conversion module 902, before being configured to convert the first time-domain signal in the song into a first time-frequency domain signal, further includes: the system layer receiving the first time-domain signal sent by the music application.

[0165] In one example, when the time-domain conversion module 910 is configured to convert the second time-frequency domain signal into a second time-domain signal to obtain an adjusted song, it includes: analyzing the ambient audio signal collected from the terminal device to obtain the sing-along signal in the ambient audio signal; and mixing the second time-domain signal and the sing-along signal to obtain the song sung by the user.

[0166] In one example, before the voice extraction module 906 is configured to input the time-frequency domain audio features into the voice extraction model and obtain the time-frequency mask output by the voice extraction model, it further includes: training an initial voice extraction model using training data; calculating a loss value based on the time-domain prediction loss and the frequency-domain prediction loss; wherein the time-domain prediction loss is the difference in the time domain between the voice signal in the training data and the predicted voice signal obtained based on the predicted time-frequency mask output by the initial voice extraction model, and the frequency-domain prediction loss is the difference in the frequency domain between the voice signal in the training data and the predicted voice signal; minimizing the loss value by adjusting the parameters of the initial voice extraction model to obtain the voice extraction model.

[0167] In one example, the training data includes at least one of the following: a simulated audio signal obtained by splicing together a human voice source and an accompaniment source; and a track-synchronized audio signal generated by separating and synchronizing the instrument components of the audio resources.

[0168] In one example, the time-frequency domain audio features include: filter bank features; the feature extraction module 904, when configured to extract features from the first time-frequency domain signal to obtain time-frequency domain audio features, includes: setting a Mel filter bank based on the sampling rate of the first time-frequency domain signal; using the Mel filter bank to filter the linear spectrum of the first time-frequency domain signal to obtain the Mel spectral energy of the first time-frequency domain signal; and taking the natural logarithm of the Mel spectral energy to obtain the filter bank features.

[0169] In one example, when the time-frequency domain conversion module 902 is configured to convert a first time-domain signal in a song into a first time-frequency domain signal, it includes: performing a short-time Fourier transform on the first time-domain signal to obtain the first time-frequency domain signal; when the time-domain conversion module 910 is configured to convert a second time-frequency domain signal into a second time-domain signal to obtain an adjusted song, it includes: performing an inverse short-time Fourier transform on the second time-frequency domain signal to obtain the second time-domain signal.

[0170] In one example, the voice extraction model includes: a nonlinear transformation unit, a multi-layer convolution unit, and a masking processing unit; the voice extraction module 906, when configured to input the time-frequency domain audio features into the voice extraction model to obtain the time-frequency mask output by the voice extraction model, includes: concatenating the time-frequency domain audio features to obtain concatenated features; wherein, the time-frequency domain audio features are feature data of the first time-frequency domain signal in multiple channels; the nonlinear transformation unit performs a nonlinear transformation on the concatenated features to obtain nonlinear transformation data; the multi-layer convolution unit performs convolution processing on the nonlinear transformation data to obtain filtered data; the masking processing unit calculates and outputs the time-frequency mask based on the filtered data; the time-frequency mask consists of a real part and an imaginary part.

[0171] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0172] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0173] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions, such as a memory including instructions, is also provided, which can be executed by a processor of a song vocal adjustment device to implement the method as described in any of the above embodiments.

[0174] The non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc., and this application does not limit it.

[0175] In an exemplary embodiment, a computer program product including a computer program / instruction is also provided, which can be executed by a processor of a song vocal adjustment device to implement the method described in any of the above embodiments.

[0176] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0177] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention filed herein. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and alterations can be made without departing from its scope. The scope of this application is limited only by the appended claims.

[0178] The above descriptions are some embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for adjusting vocals in a song, applied to a terminal device, the method comprising: Convert the first time-domain signal in the song into the first time-frequency domain signal; Feature extraction is performed on the first time-frequency domain signal to obtain time-frequency domain audio features; The time-frequency domain audio features are input into the human voice extraction model to obtain the time-frequency mask output by the human voice extraction model; the time-frequency mask is set to represent the intensity of the human voice signal at the corresponding time and frequency point in the first time-frequency domain signal; The first time-frequency domain signal is adjusted based on the time-frequency masking to obtain the second time-frequency domain signal; The second time-frequency domain signal is converted into a second time-domain signal to obtain the adjusted song.

2. The method according to claim 1, wherein, The terminal device includes a system layer and an application layer. The system layer deploys the human voice extraction model, and the application layer deploys a music application. Before converting the first time-domain signal in the song into a first time-frequency domain signal, the method further includes: The system layer receives the first time-domain signal sent by the music application.

3. The method according to claim 1 or 2, wherein, The step of converting the second time-frequency domain signal into a second time-domain signal to obtain the adjusted song includes: The ambient audio signal collected from the terminal device is analyzed to obtain the sing-along signal in the ambient audio signal; The second time-domain signal and the sing-along signal are mixed to obtain the song that the user sings along to.

4. The method according to any one of claims 1 to 3, wherein, Before inputting the time-frequency domain audio features into the voice extraction model to obtain the time-frequency mask output by the voice extraction model, the method further includes: The initial voice extraction model is trained using the training data; The loss value is calculated based on the time-domain prediction loss and the frequency-domain prediction loss; wherein, the time-domain prediction loss is the difference in the time domain between the human voice signal in the training data and the predicted human voice signal obtained by the prediction time-frequency masking based on the output of the initial human voice extraction model, and the frequency-domain prediction loss is the difference in the frequency domain between the human voice signal in the training data and the predicted human voice signal. By adjusting the parameters of the initial voice extraction model, the loss value is minimized to obtain the voice extraction model.

5. The method according to claim 4, wherein, The training data includes at least one of the following: The simulated audio signal is obtained by splicing together a human voice source and an accompaniment sound source. The audio signal is synchronized with the audio track, which is generated by separating and synchronizing the instrument components of the audio resource.

6. The method according to any one of claims 1 to 5, wherein, The time-frequency domain audio features include: filter bank features; The step of extracting features from the first time-frequency domain signal to obtain time-frequency domain audio features includes: Based on the sampling rate of the first time-frequency domain signal, the Mel filter bank is set; Using the Mel filter bank, the linear spectrum of the first time-frequency domain signal is filtered to obtain the Mel spectrum energy of the first time-frequency domain signal; The natural logarithm of the Mel spectrum energy is used to obtain the characteristics of the filter bank.

7. The method according to any one of claims 1 to 6, wherein, The step of converting the first time-domain signal in the song into a first time-frequency domain signal includes: Perform a short-time Fourier transform on the first time-domain signal to obtain the first time-frequency domain signal; The step of converting the second time-frequency domain signal into a second time-domain signal to obtain the adjusted song includes: The second time-domain signal is obtained by performing a short-time inverse Fourier transform on the second time-frequency domain signal.

8. The method according to any one of claims 1 to 7, wherein, The human voice extraction model includes: a nonlinear transformation unit, a multi-layer convolutional unit, and a masking processing unit; The step of inputting the time-frequency domain audio features into the voice extraction model to obtain the time-frequency mask output by the voice extraction model includes: The time-frequency domain audio features are spliced ​​together to obtain spliced ​​features; wherein, the time-frequency domain audio features are the feature data of the first time-frequency domain signal in multiple channels; The nonlinear transformation unit performs a nonlinear transformation on the splicing features to obtain nonlinear transformation data; The multi-layer convolutional unit performs convolution processing on the nonlinear transformed data to obtain filtered data; The masking processing unit calculates and outputs the time-frequency mask based on the filtered data; the time-frequency mask consists of a real part and an imaginary part.

9. A vocal adjustment device for songs, applied to a terminal device, the device comprising: The time-frequency domain conversion module is configured to convert the first time-domain signal in the song into the first time-frequency domain signal. The feature extraction module is configured to extract features from the first time-frequency domain signal to obtain time-frequency domain audio features; The voice extraction module is configured to input the time-frequency domain audio features into the voice extraction model to obtain the time-frequency mask output by the voice extraction model; the time-frequency mask is configured to represent the voice signal in the first time-frequency domain signal. The voice adjustment module is configured to adjust the first time-frequency domain signal based on the time-frequency masking to obtain a second time-frequency domain signal; The time-domain conversion module is configured to convert the second time-frequency domain signal into a second time-domain signal to obtain the adjusted song.

10. A terminal device, comprising: One or more processors; The memory is configured to store one or more processor-executable instructions; The one or more processors implement the method as described in any one of claims 1-8 by executing the executable instructions.

11. A computer-readable storage medium having computer instructions stored thereon, wherein, When executed by one or more processors, this instruction implements the method as described in any one of claims 1-8.

12. A computer program product having a computer program / instruction stored thereon, wherein, When the computer program / instructions are executed by one or more processors, they implement the method as described in any one of claims 1-8.

Images