A method, device, terminal equipment, medium and product for adjusting song vocals

Abstract

The present application provides a method, device, terminal device, medium and product for adjusting the vocals of a song. According to an example of the present application, the method is applied to a terminal device, and the method may include: converting a first time domain signal in a song into a first time-frequency domain signal; extracting features from the first time-frequency domain signal to obtain time-frequency domain audio features; inputting the time-frequency domain audio features into a vocal extraction model to obtain a time-frequency mask output by the vocal extraction model, the time-frequency mask being used to represent the vocal signal in the first time-frequency domain signal; adjusting the first time-frequency domain signal based on the time-frequency mask to obtain a second time-frequency domain signal; converting the second time-frequency domain signal into a second time domain signal to obtain an adjusted song, and the vocal signal in the song can be separated and adjusted to achieve real-time adjustment of the vocal volume in the song on the terminal device.

Description

Technical Field

[0001] The present application relates to the field of audio technology, and in particular to a method, device, terminal equipment, medium and product for adjusting the vocals of a song. Background Art

[0002] In the process of song production and distribution, the sound source of the song usually needs to be recorded in a professional recording studio. Each song will be recorded as a complete version including vocals and accompaniment, as well as a separate accompaniment version.

[0003] Currently, most songs are complete versions of vocals and accompaniment. Therefore, in scenarios such as karaoke, audio editing, and music production, users are faced with the challenge of not being able to directly adjust the vocals in the songs. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present application provides a song vocal adjustment method, device, terminal equipment, medium and product.

[0005] According to a first aspect of any embodiment of the present application, a method for adjusting vocals in a song is provided, the method being applied to a terminal device, the method comprising:

[0006] Converting a first time domain signal in the song into a first time-frequency domain signal;

[0007] Performing feature extraction on the first time-frequency domain signal to obtain time-frequency domain audio features;

[0008] Inputting the time-frequency domain audio features into a human voice extraction model to obtain a time-frequency mask output by the human voice extraction model; the time-frequency mask is used to represent the human voice signal in the first time-frequency domain signal;

[0009] Adjusting the first time-frequency domain signal based on the time-frequency mask to obtain a second time-frequency domain signal;

[0010] The second time-frequency domain signal is converted into a second time domain signal to obtain an adjusted song.

[0011] According to a second aspect of any embodiment of the present application, a device for adjusting vocals of a song is provided, the device being applied to a terminal device, the device comprising:

[0012] A time-frequency domain conversion module, used for converting a first time-domain signal in a song into a first time-frequency domain signal;

[0013] A feature extraction module, used to extract features from the first time-frequency domain signal to obtain time-frequency domain audio features;

[0014] A human voice extraction module, used for inputting the time-frequency domain audio features into a human voice extraction model to obtain a time-frequency mask output by the human voice extraction model; the time-frequency mask is used to represent the human voice signal in the first time-frequency domain signal;

[0015] A human voice adjustment module, used for adjusting the first time-frequency domain signal based on the time-frequency masking to obtain a second time-frequency domain signal;

[0016] The time domain conversion module is used to convert the second time-frequency domain signal into a second time domain signal to obtain an adjusted song.

[0017] According to a third aspect of any embodiment of the present application, a terminal device is provided, including:

[0018] processor;

[0019] a memory for storing processor-executable instructions;

[0020] The processor implements the method described in any embodiment of the present application by running the executable instructions.

[0021] According to a fourth aspect of any embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the method described in any embodiment of the present application is implemented.

[0022] According to a fifth aspect of any embodiment of the present application, a computer program product is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the method described in any embodiment of the present application is implemented.

[0023] The technical solution provided by this application may have the following beneficial effects:

[0024] According to the above embodiments, by converting the first time domain signal in the song into a first time-frequency domain signal, performing feature extraction on the first time-frequency domain signal to obtain time-frequency domain audio features, inputting the time-frequency domain audio features into the human voice extraction model, obtaining the time-frequency masking output by the human voice extraction model, adjusting the first time-frequency domain signal based on the time-frequency masking to obtain the second time-frequency domain signal, converting the second time-frequency domain signal into the second time domain signal to obtain the adjusted song, the human voice signal in the song can be separated and adjusted, and real-time adjustment of the human voice volume in the song can be achieved on the terminal device.

[0025] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in the specification and constitute a part of this application, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0027] Figure 1 is a flow chart of a method for adjusting vocals in a song according to an exemplary embodiment of the present application;

[0028] Figure 2 is a schematic diagram of the structure of the application layer and system layer of a terminal device according to an exemplary embodiment of the present application;

[0029] Figure 3 is a structural schematic diagram of an adjustment module of a first time domain signal according to an exemplary embodiment of the present application;

[0030] Figure 4 is a schematic diagram of a FBank feature according to an exemplary embodiment of the present application;

[0031] Figure 5 is a structural schematic diagram of a human voice extraction model according to an exemplary embodiment of the present application;

[0032] Figure 6 is a schematic diagram of a simulated audio signal according to an exemplary embodiment of the present application;

[0033] Figure 7 is a flow chart of another method for adjusting the vocals of a song according to an exemplary embodiment of the present application;

[0034] Figure 8 is a schematic diagram of the structure of a terminal device according to an exemplary embodiment of the present application;

[0035] Fig. 9 It is a block diagram of a device for adjusting vocals in a song according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0036] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0037] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0038] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0039] Current songs are usually complete versions of vocals and accompaniment, and it is impossible to directly adjust the vocals in the song.

[0040] In order to solve the above problems, the present application proposes a method for adjusting the vocals of a song. To further illustrate the present application, the following embodiments are provided:

[0041] See also Figure 1 , Figure 1 This is a flowchart of a method for adjusting the vocals of a song according to an exemplary embodiment of the present application. The method for adjusting the vocals of a song can be applied to terminal devices such as car computers, smart speakers, mobile phones, computers, messaging devices, tablet devices, personal digital assistants, and can be applied to scenes such as karaoke, audio editing, and music production.

[0042] like Figure 1 As shown, the song vocal adjustment method may include the following steps:

[0043] Step 102: Convert a first time domain signal in the song into a first time-frequency domain signal.

[0044] In this step, the terminal device can obtain a first time domain signal of a frame in the song. The first time domain signal is a sound wave signal of the song in the time domain. The first time domain signal is used to describe the changes of the song on the time axis, characterize the value or amplitude of the song at different time points, and reveal information such as the waveform, amplitude, and periodicity of the song.

[0045] The first time domain signal may be a time domain signal of a frame in a song. A time domain signal of a frame is a very small, continuous time segment in the audio stream of the song, for example, 10ms. A time domain signal of a frame contains a number of digital samples, which are arranged in time order to form a time domain representation of the first time domain signal.

[0046] The terminal device can use short time Fourier transform (STFT), Mel-Frequency Cepstral Coefficients (MFCC), constant Q transform (CQT), etc. to convert the first time domain signal into the time-frequency domain to obtain a first time-frequency domain signal corresponding to the first time domain signal.

[0047] Among them, the first time-frequency domain signal is used to describe the characteristics of the song while considering both time and frequency information. In the time-frequency domain, we can pay attention to the changes of the song at different time points and different frequencies, providing a more comprehensive description of signal characteristics.

[0048] For example, the frequency of a song may change over time, and the first time-frequency domain signal can simultaneously consider both aspects of information, which helps to understand and analyze the song more accurately.

[0049] Step 104: extract features from the first time-frequency domain signal to obtain time-frequency domain audio features.

[0050] In this step, the terminal device can extract the time-frequency domain audio features in the first time-frequency domain signal by analyzing the spectrum diagram, deep learning model and other methods.

[0051] Among them, the time-frequency domain audio features are parameters or indicators that describe the characteristics of songs, which are used for in-depth analysis and processing of songs. They can be spectrum, power spectrum, filter bank (FilterBank, FBank) features, etc.

[0052] The extraction of audio features in the time-frequency domain is equivalent to a process of refining and simplifying the first time-frequency domain signal according to objective physical laws or statistical laws. Its purpose is to highlight the information that is useful to the machine learning algorithm of the subsequent vocal extraction model, and to compress and delete useless or redundant information in the song, so that the subsequent model can better complete the machine learning task.

[0053] Step 106: input the time-frequency domain audio features into the human voice extraction model to obtain the time-frequency mask output by the human voice extraction model; the time-frequency mask is used to represent the human voice signal in the first time-frequency domain signal.

[0054] In this step, the terminal device inputs the time-frequency domain audio features into the vocal extraction model. The vocal extraction model is a trained deep convolutional neural network (DCNN) model that can identify the vocal component in the song and extract the time-frequency mask in the first time-frequency domain signal.

[0055] The terminal device obtains the time-frequency mask in the first time-frequency domain signal output by the human voice extraction model. The time-frequency mask is used to represent the human voice signal in the first time-frequency domain signal. The human voice signal is the human voice component of the song in the time-frequency domain, including the singer's voice characteristics, pitch, timbre, volume and other information.

[0056] The time-frequency mask is a two-dimensional matrix whose size is the same as the input time-frequency domain signal. In the time-frequency mask, the value of each element represents the intensity or existence of the human voice signal at the corresponding time and frequency point.

[0057] Exemplarily, the value of each element in the time-frequency mask may be normalized to between 0 and 1, wherein 1 indicates that it completely belongs to a human voice signal, and 0 indicates that it does not belong to a human voice signal at all.

[0058] Step 108: Adjust the first time-frequency domain signal based on time-frequency masking to obtain a second time-frequency domain signal.

[0059] In this step, the terminal device adjusts the first time-frequency domain signal based on time-frequency masking to suppress the human voice volume and retain only the human voice, thereby obtaining an adjusted second time-frequency domain signal. The second time-frequency domain signal is a second time-frequency domain signal of the human voice for the first time-frequency domain signal.

[0060] Exemplarily, the terminal device can apply the time-frequency masking output by the human voice extraction model to the first time-frequency domain signal, thereby eliminating the accompaniment in the first time-frequency domain signal and retaining only the human voice signal in the first time-frequency domain signal. The second time-frequency domain signal retaining only the human voice is obtained according to the following formula (1):

[0061] (1)

[0062] in, Indicates that only the second time-frequency domain signal of the human voice is retained. represents the time-frequency masking output by the human voice extraction model, represents the first time-frequency domain signal, k is the frequency band index, and τ is the data block index.

[0063] The terminal device can also reduce the time-frequency masking output by the human voice extraction model, apply the reduced time-frequency masking to the first time-frequency domain signal corresponding to the song, and adjust the human voice signal in the first time-frequency domain signal, thereby suppressing the volume of the human voice in the song. The second time-frequency domain signal after the human voice volume is suppressed is obtained according to the following formula (2):

[0064] (2)

[0065] in, Represents the second time-frequency domain signal after the human voice volume is suppressed, represents the time-frequency masking output by the human voice extraction model, represents the first time-frequency domain signal. Indicates the magnitude of the suppression for time-frequency masking, The range is [0, 1].

[0066] Taking the range of the suppression of the human voice volume in the song as [-40, 0] dB as an example, when When it is 0, the suppression amplitude of time-frequency masking reaches the highest level, the suppression amplitude of human voice volume is -40dB, and only the accompaniment is retained in the adjusted second time-frequency domain signal; when When it is 1, the suppression amplitude of the time-frequency masking reaches the minimum, the suppression amount of the human voice volume is 0dB, and the human voice volume in the adjusted second time-frequency domain signal is not suppressed.

[0067] It is understandable that the terminal device can also perform operations such as enhancing the time-frequency masking according to the actual business scenario to obtain a second time-frequency domain signal after the human voice signal is enhanced, and the embodiments of the present application are not limited to this.

[0068] In one embodiment, the terminal device may receive a human voice adjustment ratio set by a user through an interface of the terminal device, and suppress time-frequency masking according to the human voice adjustment ratio set by the user.

[0069] The vocal adjustment ratio is the ratio of adjusting the volume of the vocals in the song, and is used to identify the suppression amount of time-frequency masking.

[0070] Taking the above formula (2) as an example, the terminal device sets the suppression amount amplitude for time-frequency masking according to the human voice adjustment ratio set by the user. Based on the suppression amount amplitude of the time-frequency masking, the human voice signal in the first time-frequency domain signal is suppressed.

[0071] For example, if the human voice adjustment ratio set by the user is reduced by 30%, the suppression amount of the time-frequency masking of the human voice can be reduced. Set to 0.7, thereby obtaining the second time-frequency domain signal with the human voice volume suppressed by 30%.

[0072] As described above, by suppressing the time-frequency masking according to the vocal adjustment ratio set by the user, and adjusting the vocal signal in the first time-frequency domain signal based on the suppressed time-frequency masking, the vocal volume in the song is adjusted according to user needs, providing users with a more personalized music experience.

[0073] Step 110: Convert the second time-frequency domain signal into a second time domain signal to obtain an adjusted song.

[0074] In this step, the terminal device uses methods such as Inverse Short-Time Fourier Transform (ISTFT), Overlap-Add Method, and Overlap-Save Method to convert the second time-frequency domain signal after the human voice adjustment into a second time domain signal to obtain this frame of the song after the human voice adjustment. The second time domain signal is a time domain signal obtained by converting the second time-frequency domain signal after the human voice adjustment back to the time domain.

[0075] The method for adjusting the vocals of a song in the present embodiment converts a first time domain signal in the song into a first time-frequency domain signal, extracts features from the first time-frequency domain signal to obtain time-frequency domain audio features, inputs the time-frequency domain audio features into a vocal extraction model, obtains a time-frequency mask output by the vocal extraction model, adjusts the first time-frequency domain signal based on the time-frequency mask to obtain a second time-frequency domain signal, converts the second time-frequency domain signal into a second time domain signal to obtain an adjusted song, and can separate and adjust the vocal signal in the song to achieve real-time adjustment of the vocal volume in the song on a terminal device.

[0076] In addition, streaming the first time domain signal of a frame and adjusting the vocal signal of the frame has a lower output delay, so that the vocals of the song can be adjusted in real time. At the same time, the computing power and storage resources consumed are low, and only limited hardware resources in the terminal device are needed to adjust the vocal components in the song.

[0077] In the above-mentioned embodiment, it is introduced that the time-frequency mask in the first time-frequency domain signal is extracted by the human voice extraction model, and the adjusted time-frequency mask is applied to the first time-frequency domain signal to adjust the human voice signal in the song. In the following embodiment, the application scenario of the human voice adjustment of the song will be described in more detail, and can be applied to any of the above embodiments.

[0078] In one embodiment, the terminal device may include: a system layer and an application layer. The system layer deploys a human voice extraction model, and the application layer deploys a music application. When playing a song, the music application in the application layer calls the interface of the system layer to play. The terminal device can receive the first time domain signal of each frame sent by the music application from the interface of the system layer.

[0079] The system layer can convert the received first time domain signal into a first time-frequency domain signal, extract features from the first time-frequency domain signal, and obtain time-frequency domain audio features. The system layer uses a human voice extraction model to obtain a time-frequency mask corresponding to the first time-frequency domain signal.

[0080] The system layer adjusts the human voice signal in the first time-frequency domain signal based on time-frequency masking to obtain an adjusted second time-frequency domain signal. The second time-frequency domain signal is converted into a second time domain signal to obtain an adjusted song.

[0081] See also Figure 2 , Figure 2 The schematic diagram of the structure of the application layer and the system layer of a terminal device is shown. The terminal device may include an application layer 20, a system layer 21, a microphone 22 and a speaker 23.

[0082] The application layer 20 may include a first music application 201 , a second music application 202 , a third music application 203 and multiple music applications.

[0083] The system layer 21 may include: an adjustment module 211 , an environmental human voice extraction module 212 , a mixing module 213 and a frequency division module 214 .

[0084] When the user uses the third music application 203 to play a song, the third music application 203 calls the system layer 30, and the adjustment module 211 receives a first time domain signal with a sampling rate of 48kHz of a frame of the song sent by the third music application 203.

[0085] The adjustment module 211 converts the first time domain signal into a first time-frequency domain signal, extracts the time-frequency mask in the first time-frequency domain signal, adjusts the human voice signal in the first time-frequency domain signal based on the time-frequency mask, and then converts it into the time domain to obtain a second time domain signal with a sampling rate of 48kHz.

[0086] As described above, by receiving the first time domain signal sent by the music application in the application layer, the terminal device does not need to obtain song resources in advance, and can achieve song vocal adjustment by only using the songs played by the music application in the application layer.

[0087] In addition, by receiving the first time domain signal in the song sent by the music application, the vocals of the song are adjusted, and then the adjusted second time domain signal continues to be played through the audio channel. This is imperceptible to the music application at the application layer, and the music application does not need to be modified.

[0088] In one embodiment, the terminal device can collect the ambient audio signal of the terminal device through a microphone. The ambient audio signal collected from the terminal device is analyzed, and audio processing such as echo cancellation, howling suppression, noise reduction, reverberation, sound effects, and upsampling is performed to obtain a clean sing-along signal in the ambient audio signal. The second time domain signal and the sing-along signal are mixed to obtain a song that the user sings along with.

[0089] Among them, the music application is an application program in the terminal device with a song playing function.

[0090] The ambient audio signal is a sound wave signal from the environment around the terminal device captured by the microphone, which contains key information such as the frequency, amplitude and phase of the ambient sound. The sing-along signal is an audio signal formed by the user singing along with the song in the ambient audio signal, which is the human voice part of the ambient audio signal.

[0091] Please continue reading Figure 2 The terminal device collects the ambient audio signal around the terminal device through the microphone 22, and the ambient audio signal includes the singing-along human voice and echo at a sampling rate of 16kHz.

[0092] The ambient human voice extraction module 212 performs audio processing such as echo cancellation, howling suppression, noise reduction, reverberation, sound effects, and upsampling on the ambient audio signal to obtain a clean sing-along signal with a sampling rate of 48kHz in the ambient audio signal.

[0093] The mixing module 213 mixes the second time domain signal and the sing-along signal in the ambient audio signal to obtain a song with human voice singing along, wherein the song with human voice singing along includes the sing-along signal with a sampling rate of 48kHz, the song accompaniment and the adjusted song vocals.

[0094] The frequency division module 214 divides the song sung by human voice into multiple frequency bands according to the frequency range, for example, bass, mid-range, and treble, and plays them through different speakers 23 respectively to provide a richer and more realistic listening experience.

[0095] As described above, by analyzing the ambient audio signal collected by the terminal device, the sing-along signal in the ambient audio signal is obtained, the second time domain signal and the sing-along signal are mixed to obtain the time domain signal of the user singing along, which has real-time performance and low resource consumption, and realizes the karaoke function of the original singer leading the song in a low voice.

[0096] In the above-mentioned embodiment, it is introduced that the karaoke function of the song vocals leading the song quietly is realized by adjusting the first time domain signal of the song in the music application. In the following embodiment, each module for realizing the song vocals adjustment in the terminal device will be described in detail, and can be applied to any of the above embodiments.

[0097] In one embodiment, see Figure 3 , Figure 3 The schematic diagram of the structure of the adjustment module of the first time domain signal is shown. The adjustment module 211 may include: an STFT module 2110, an audio feature extraction module 2111, a human voice extraction model 2112, a human voice ratio adjustment module 2113 and an ISTFT module 2114.

[0098] The STFT module 2110 can perform a short-time Fourier transform on the first time domain signal in the song to obtain a first time-frequency domain signal. The first time-frequency domain signal is input into the audio feature extraction module 2111 and the vocal ratio adjustment module 2113, and the audio feature extraction module 2111 extracts the time-frequency domain audio features in the first time-frequency domain signal.

[0099] The time-frequency domain audio features are input to the human voice extraction model 2112 , and the human voice extraction model 2112 sends the extracted time-frequency mask to the human voice ratio adjustment module 2113 .

[0100] The vocal ratio adjustment module 2113 adjusts the time-frequency masking based on the vocal adjustment ratio set by the user or preset, applies the adjusted time-frequency masking to the first time-frequency domain signal, and adjusts the vocal signal in the first time-frequency domain signal.

[0101] The ISTFT module 2114 performs inverse short-time Fourier transform on the second time-frequency domain signal after the vocal adjustment, converts it into a second time domain signal, and obtains the song after the vocal adjustment.

[0102] As described above, by performing a short-time Fourier transform on the first time domain signal and then performing an inverse short-time Fourier transform on the second time-frequency domain signal after the vocal adjustment to obtain the second time domain signal, the characteristics of the song in both time and frequency dimensions are captured. In addition, the short-time Fourier transform and the inverse short-time Fourier transform can improve the computational efficiency of converting from the time domain to the time-frequency domain and then back to the time domain, thereby further ensuring the real-time adjustment of the vocals of the song.

[0103] In one embodiment, the time-frequency domain audio features may include: FBank features. The audio feature extraction module 2111 first determines the number of Mel filters in the Mel filter bank, and the preferred range of the number is between twenty and forty Mel filters. According to the sampling rate of the first time-frequency domain signal, the frequency range to be analyzed is determined, and the Mel filter bank is set.

[0104] Convert the linear frequency range to the Mel frequency range, and divide the intervals evenly on the Mel scale according to the number of selected filters. Each interval corresponds to a Mel filter. Then convert each Mel interval to the corresponding linear frequency range, and each Mel filter corresponds to a linear frequency band to obtain a Mel filter bank.

[0105] The linear spectrum of the first time-frequency domain signal is filtered using the set Mel filter bank, and the energy of the output of each Mel filter is calculated to obtain the Mel spectrum energy of the first time-frequency domain signal. The natural logarithm of the Mel spectrum energy is calculated to obtain the filter bank feature.

[0106] The Mel filter bank includes a series of triangular Mel filters distributed according to the Mel scale, which are used to map the linear spectrum into a nonlinear Mel spectrum based on auditory perception.

[0107] The Mel scale can simulate the nonlinear characteristics of human hearing. The human ear has a higher resolution for low-frequency components, and as the frequency increases, the perception of resolution gradually decreases.

[0108] Mel spectrum energy is obtained by calculating the output energy of each Mel filter, which is used to simulate the nonlinear perception of frequency by the human auditory system.

[0109] Exemplarily, the audio feature extraction module 2111 may convert the frequency between the linear scale h (Hz) and the Mel scale m (Mel) according to the following formula (3) and formula (4):

[0110] (3)

[0111] (4)

[0112] in, represents the Mel scale, Represents a linear scale.

[0113] See also Figure 4 , Figure 4 A schematic diagram of FBank feature is shown. Taking the sampling rate of the first time-frequency domain signal as 48kHz as an example, according to formula (3) and formula (4), a Mel filter bank covering the entire frequency range of 0 to 24 kHz can be set.

[0114] Figure 4 The starting and ending frequencies of the 120-dimensional FBank feature setting are = 0 and = 24000. According to formula (3), the starting and ending points on the Mel scale can be obtained: = 0, = 4016.

[0115] Since the 120-dimensional FBank feature is equivalent to a total of 120 peaks of the triangular window, and half of the adjacent triangular windows overlap, it is necessary to [ , ] The interval is evenly divided into 121 parts, requiring a total of 122 demarcation points, each corresponding to half a triangular window, and its coverage range is .

[0116] from[ , ] The first dividing point in the interval, the three adjacent dividing points correspond to the starting, center, and cutoff frequencies of a triangular window respectively. Then, according to formula (4), the dividing points are converted back to the linear scale to obtain the non-uniform interval. Then, the triangular window is set with three dividing points as a group. Each triangular window is a Mel filter, and a series of Mel filters constitute a Mel filter bank.

[0117] Apply each Mel filter in the Mel filter bank to the linear spectrum, calculate the output energy of each filter, and obtain the Mel spectrum energy of the first time-frequency domain signal. Take the natural logarithm of the Mel spectrum energy to get Figure 4 FBank features of the window structure in .

[0118] As described above, by setting the Mel filter group based on the sampling rate of the first time-frequency domain signal, using the Mel filter group to filter the linear spectrum of the first time-frequency domain signal, the Mel spectrum energy of the first time-frequency domain signal is obtained, and the natural logarithm of the Mel spectrum energy is calculated to obtain the filter group characteristics. Based on the nonlinear perception characteristics of the human auditory system to frequency, the extracted features are more in line with the actual human hearing and the accuracy of vocal adjustment of the song is improved.

[0119] In one embodiment, the human voice extraction model may include: a nonlinear transformation unit, a multi-layer convolution unit and a masking processing unit.

[0120] The terminal device can splice the time-frequency domain audio features of multiple channels such as stereo channels by means of concatenation, fusion, etc. to obtain the spliced ​​features. The time-frequency domain audio features are feature data of the first time-frequency domain signal on multiple channels.

[0121] The terminal device inputs the concatenated features into the nonlinear transformation unit, which performs nonlinear transformation on the concatenated features to obtain nonlinear transformation data. The nonlinear transformation data is used to enhance the nonlinear expression ability of the model, which helps the model learn more complex audio feature representations.

[0122] The nonlinear transformation unit inputs the nonlinear transformation data into the multi-layer convolution unit, and the multi-layer convolution unit performs convolution processing on the nonlinear transformation data. A deeper network structure is constructed by stacking multiple convolution layers to extract local features and reduce the spatial dimension of the features to obtain filtered data.

[0123] The multi-layer convolution unit inputs the filtered data to the mask processing unit, which calculates and outputs the time-frequency mask based on the filtered data. In the fully connected layer, each input node of the mask processing unit is connected to the output node through a weight and a bias term is added.

[0124] Through forward propagation, the values ​​of each output node are calculated, which correspond to the real and imaginary parts of the time-frequency mask. The activation function is applied to the output of the fully connected layer to obtain the final time-frequency mask consisting of the real and imaginary parts.

[0125] The time-frequency mask is in complex form, consisting of a real part and an imaginary part. The real part and the imaginary part in the time-frequency mask jointly describe the characteristics of the first time-frequency domain signal in the time-frequency domain.

[0126] See also Figure 5 , Figure 5 The schematic diagram of the structure of a human voice extraction model is shown. The human voice extraction model may include: a nonlinear transformation unit 50, a multi-layer convolution unit 51 and a masking processing unit 52.

[0127] The nonlinear transformation unit 50 may include a fully connected layer (Affine) and a rectified linear unit (ReLU) activation function. The masking processing unit 52 may include a fully connected layer (Affine) and a tanh activation function.

[0128] The multi-layer convolution unit 51 may include a multi-layer feedforward sequential memory network (FSMN) unit, and each layer of the FSMN unit may include: a fully connected layer 510, a feedforward sequential memory network layer 511 and an activation layer 512.

[0129] The fully connected layer 510 is a linear layer, which is used to receive input data and generate output through linear transformation. The activation layer 512 is a ReLU (Affine) layer, which is used to perform nonlinear transformation on the output of the feedforward sequence memory network layer 511.

[0130] The feedforward sequence memory network layer 511 is composed of a convolutional structure, which is equivalent to the frequency domain signal processing algorithm frequency band filtering operation. A one-dimensional convolution is performed on each feature dimension.

[0131] The nonlinear transformation unit 50 uses the ReLU activation function in the fully connected layer to perform nonlinear transformation on the concatenated features to obtain nonlinear transformation data. The multi-layer convolution unit 51 uses multi-layer FSMN units to perform convolution processing on the nonlinear transformation data to obtain filtered data.

[0132] The feedforward sequence memory network layer 511 can obtain the convolution-processed data according to the following formula (5): (5)

[0133] Wherein, x represents the input data of the feedforward sequence memory network layer 511, y represents the output data of the feedforward sequence memory network layer 511, w and v represent the filter coefficients learned by the neural network, Indicates the data dimension, represents the frame index, T represents the convolution length, and t represents the time index.

[0134] The mask processing unit 52 in the fully connected layer calculates the value of each output node through forward propagation based on the filtered data, and uses the tanh activation function to output a time-frequency mask consisting of a real part and an imaginary part.

[0135] Understandably, Figure 5 The structure of the vocal extraction model shown is only an example, and other neural network structures may also be used. As long as the vocal extraction model can implement the song vocal adjustment method described in the embodiment of the present application, the embodiment of the present application does not impose any restrictions on this.

[0136] As described above, by splicing the time-frequency domain audio features to obtain the spliced ​​features, the nonlinear transformation unit performs nonlinear transformation on the spliced ​​features to obtain nonlinear transformation data, and the multi-layer convolution unit performs convolution processing on the nonlinear transformation data to obtain filtered data, the model calculation speed can be further accelerated and the calculation complexity can be reduced. The masking processing unit calculates and outputs the time-frequency mask composed of the real part and the imaginary part based on the filtered data. The time-frequency mask composed of the real part and the imaginary part can more accurately distinguish the time-frequency domain audio features of different sound sources, more effectively reduce the influence of interference components such as background noise and accompaniment music on the time-frequency domain audio features, and improve the accuracy of human voice extraction.

[0137] In one embodiment, the terminal device can use the constructed training data to train the initial human voice extraction model. The training data is input into the initial human voice extraction model, and the initial human voice extraction model performs calculations through its network structure and outputs the predicted time-frequency mask. The predicted time-frequency mask is the human voice signal predicted by the initial human voice extraction model for the training data.

[0138] The loss value of the loss function is calculated based on the time domain prediction loss and frequency domain prediction loss of the initial human voice extraction model for the training data.

[0139] Through optimization algorithms such as Stochastic Gradient Descent (SGD) and Adam, the parameters of the initial human voice extraction model are adjusted to reduce the loss value of the loss function. The above steps of time-frequency masking prediction, loss value calculation and parameter update are repeated until the loss value is minimized to obtain the human voice extraction model.

[0140] The training data includes multiple audio samples, each of which includes mixed audio (such as a mixture of accompaniment and vocals) and a corresponding vocal signal, which is used to train the model to learn to extract the vocal signal from the mixed audio.

[0141] The time domain prediction loss is the difference in the time domain between the vocal signal in the training data and the predicted time-frequency mask output by the initial vocal extraction model. The frequency domain prediction loss is the difference in the frequency domain between the vocal signal and the predicted time-frequency mask.

[0142] For example, the loss value of the loss function can be calculated according to the following formula (6):

[0143] (6)

[0144] in, represents the loss value of the loss function, represents the predicted time-frequency masking, represents the real human voice signal in the training data, represents the average value of the time domain prediction loss, represents the average value of the frequency domain prediction loss. k is the frequency band index, τ is the data block index, and t is the time index.

[0145] The above loss function also implicitly includes the loss of the phase information of the predicted time-frequency mask. The phase information of the predicted time-frequency mask can also be continuously optimized with the model training. When the loss value is minimized, the phase information of the predicted time-frequency mask will be more consistent with the phase information of the human voice signal in the training data.

[0146] As described above, the initial human voice extraction model is trained by using the constructed training data, and the loss value of the loss function is calculated according to the time domain prediction loss and the frequency domain prediction loss. The loss value is minimized by adjusting the parameters of the initial human voice extraction model to obtain the human voice extraction model. Since the loss function considers the prediction losses in both the time domain and the frequency domain at the same time, and the phase information of the predicted time-frequency masking output when the loss value is minimized is also more accurate, the trained human voice extraction model can extract more accurate time-frequency masking.

[0147] In one embodiment, in order to enhance the stability and generalization of the human voice extraction model to cover more and more comprehensive application scenarios, it is necessary to use a variety of massive data to train the initial human voice extraction model.

[0148] The training data may include at least one of the following: a simulated audio signal, and an audio signal synchronized with an audio track.

[0149] By splicing the various previously stored human voice sources and accompaniment sound sources, and then adding reverberation, noise, nonlinear processing, adjusting the signal-to-noise ratio and other human voice sources and accompaniment sound sources, analog audio signals collected under various conditions in different environments are simulated.

[0150] See also Figure 6 , Figure 6 A schematic diagram of a simulated audio signal is shown. The randomly selected human voice source and accompaniment sound source are respectively subjected to audio processing such as voice change, speech speed change, short sentence splicing, etc. to generate long audio of the human voice source and long audio of the accompaniment sound source.

[0151] The long audio of the human voice source and the long audio of the accompaniment sound source are convolved using the transfer function in the transfer function library or the transfer function simulated by the transfer function to obtain the human voice signal and the accompaniment signal.

[0152] The transfer function library refers to a library containing a variety of transfer functions, which can be directly applied to modify the long audio of the human voice source and the long audio of the accompaniment sound source.

[0153] Transfer function simulation refers to the use of mathematical models or simulation software to simulate a transfer function, which supports the simulation of audio effects without actually building a physical system. By adjusting the parameters in the simulation, it is ensured that the long audio of the human voice source and the long audio of the accompaniment source achieve the expected audio effect in actual applications.

[0154] The human voice signal and the accompaniment signal are adjusted by random signal-to-noise ratio and volume adjustment to obtain a simulated audio signal that simulates the long audio of a multi-channel microphone. The initial human voice extraction model can be trained using the simulated audio signal through model training tools such as TensorFlow and PyTorch.

[0155] The terminal device can also use the Teacher-Student mode to select a Teacher model (such as a high-quality pre-trained model such as Demucs) to process a large amount of audio resources and separate various instrument components. The audio resources can be audio without separate tracks.

[0156] The separated instrumental components can then be synchronized by comparing and analyzing the time code of the audio signal or by utilizing the phase information of the audio signal to achieve time synchronization of the separated instrumental components. Rhythm analysis can also be used to ensure that the rhythms of the various tracks match each other, thereby generating a track-synchronized audio signal in which the various tracks in the track-synchronized audio signal maintain precise synchronization in time and rhythm.

[0157] The initial vocal extraction model is used as the Student model, and the initial vocal extraction model is trained using the audio signal synchronized with the audio track.

[0158] The training data can also include open source datasets such as MUSDB18-HQ that contain song data with synchronized audio tracks.

[0159] As described above, by using the simulated audio signal obtained by audio processing the human voice source and the accompaniment sound source to train the initial human voice extraction model, it is possible to obtain diverse training data more economically and efficiently; by using the audio signal synchronized with the audio track generated by separating and synchronizing the instrument components of the audio resources to train the initial human voice extraction model, it helps the human voice extraction model to extract the human voice more accurately in a complex music environment.

[0160] To further introduce the process of adjusting the vocals of songs, Figure 7 A flow chart of another method for adjusting the vocals of a song is shown. The method for adjusting the vocals of a song may include the following steps:

[0161] Step 702: Use the constructed training data to train the initial voice extraction model.

[0162] In this step, the terminal device uses the constructed simulated audio signal, the audio signal synchronized with the audio track and other training data to train the initial human voice extraction model.

[0163] Step 704: Calculate the loss value of the loss function according to the time domain prediction loss and the frequency domain prediction loss.

[0164] In this step, the terminal device calculates the loss value of the loss function based on the time domain prediction loss and the frequency domain prediction loss between the human voice signal in the training data and the predicted time-frequency mask output by the initial human voice extraction model.

[0165] Step 706: The terminal device adjusts the parameters of the initial human voice extraction model to minimize the loss value and obtain the human voice extraction model.

[0166] Step 708: Receive a first time domain signal sent by a music application.

[0167] In this step, the system layer in the terminal device receives the first time domain signal in the song sent by the music application of the application layer.

[0168] Step 710: Perform short-time Fourier transform on the first time domain signal to obtain a first time-frequency domain signal.

[0169] In this step, the terminal device performs short-time Fourier transform on the first time domain signal to obtain a first time-frequency domain signal of the first time domain signal in the time-frequency domain.

[0170] Step 712: The terminal device extracts features from the first time-frequency domain signal to obtain FBank features.

[0171] In this step, the terminal device sets the Mel filter group based on the sampling rate of the first time-frequency domain signal, uses the Mel filter group to filter the linear spectrum of the first time-frequency domain signal, obtains the Mel spectrum energy of the first time-frequency domain signal, calculates the natural logarithm of the Mel spectrum energy, and obtains the FBank feature.

[0172] Step 714: The terminal device splices the FBank features of multiple channels to obtain spliced ​​features.

[0173] Step 716: Input the concatenated features into the human voice extraction model to obtain the time-frequency mask output by the human voice extraction model.

[0174] In this step, the nonlinear transformation unit in the human voice extraction model performs nonlinear transformation on the splicing features to obtain nonlinear transformation data, the multi-layer convolution unit performs convolution processing on the nonlinear transformation data to obtain filtered data, and the masking processing unit calculates and outputs the time-frequency mask composed of the real part and the imaginary part based on the filtered data.

[0175] Step 718: According to the human voice adjustment ratio set by the user, the first time-frequency domain signal is adjusted based on time-frequency masking.

[0176] In this step, the terminal device adjusts the time-frequency mask corresponding to the first time-frequency domain signal according to the human voice adjustment ratio set by the user, applies the adjusted time-frequency mask to the first time-frequency domain signal, adjusts the human voice signal in the first time-frequency domain signal, and generates a second time-frequency domain signal after the human voice adjustment.

[0177] Step 720: The terminal device performs an inverse short-time Fourier transform on the second time-frequency domain signal to obtain a second time domain signal.

[0178] Step 722: Analyze the ambient audio signal collected from the terminal device to obtain a sing-along signal in the ambient audio signal.

[0179] In this step, the terminal device performs audio processing such as echo cancellation, noise reduction, and reverberation on the ambient audio signal collected by the terminal device to obtain a sing-along signal in the ambient audio signal.

[0180] Step 724: Mix the second time domain signal and the sing-along signal to obtain a song that the user sings along with.

[0181] In this step, the terminal device mixes the second time domain signal and the sing-along signal to obtain the song that the user sings along with, thereby achieving real-time adjustment of the vocal volume of the song played in the music application.

[0182] Figure 8 1 is a schematic diagram of a terminal device according to an exemplary embodiment of the present application. The terminal device may be, for example, a mobile phone, a computer, a digital broadcast terminal, a message transceiver, a game console, a tablet device, a personal digital assistant, a server, a smart home appliance, a car computer, etc. Figure 8 At the hardware level, the terminal device includes a processor 802, an internal bus 804, a network interface 806, a memory 808, and a non-volatile memory 810, and may also include hardware required for other services. The processor 802 reads the corresponding computer program from the non-volatile memory 810 into the memory 808 and then runs it, forming a song vocal adjustment device at the logical level. Of course, in addition to the software implementation, this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0183] Fig. 9 is a block diagram of a device for adjusting the vocals of a song according to an exemplary embodiment of the present application. Fig. 9 The device may include: a time-frequency domain conversion module 902, a feature extraction module 904, a human voice extraction module 906, a human voice adjustment module 908 and a time domain conversion module 910, wherein:

[0184] The time-frequency domain conversion module 902 is used to convert the first time domain signal in the song into a first time-frequency domain signal;

[0185] The feature extraction module 904 is used to extract features from the first time-frequency domain signal to obtain time-frequency domain audio features;

[0186] The human voice extraction module 906 is used to input the time-frequency domain audio features into the human voice extraction model to obtain the time-frequency mask output by the human voice extraction model; the time-frequency mask is used to represent the human voice signal in the first time-frequency domain signal;

[0187] The human voice adjustment module 908 is used to adjust the first time-frequency domain signal based on the time-frequency masking to obtain a second time-frequency domain signal;

[0188] The time domain conversion module 910 is used to convert the second time-frequency domain signal into a second time domain signal to obtain an adjusted song.

[0189] In one example, the terminal device includes: a system layer and an application layer, the system layer is deployed with the human voice extraction model, and the application layer is deployed with a music application; the time-frequency domain conversion module 902, before being used to convert the first time domain signal in the song into the first time-frequency domain signal, also includes: the system layer receives the first time domain signal sent by the music application.

[0190] In one example, the time domain conversion module 910, when used to convert the second time-frequency domain signal into a second time domain signal to obtain an adjusted song, includes: analyzing the ambient audio signal collected from the terminal device to obtain a sing-along signal in the ambient audio signal; mixing the second time domain signal and the sing-along signal to obtain a song that the user sings along with.

[0191] In one example, the human voice extraction module 906, before being used to input the time-frequency domain audio features into the human voice extraction model to obtain the time-frequency mask output by the human voice extraction model, also includes: using the constructed training data to train the initial human voice extraction model; calculating the loss value based on the time domain prediction loss and the frequency domain prediction loss; wherein the time domain prediction loss is the difference in the time domain between the human voice signal in the training data and the predicted time-frequency mask output by the initial human voice extraction model, and the frequency domain prediction loss is the difference in the frequency domain between the human voice signal and the predicted time-frequency mask; by adjusting the parameters of the initial human voice extraction model, the loss value is minimized to obtain the human voice extraction model.

[0192] In one example, the training data includes at least one of the following: a simulated audio signal, which is obtained by splicing and simulating a human voice source and an accompaniment sound source; and a track-synchronized audio signal, which is an audio signal generated by separating and synchronizing instrument components of audio resources.

[0193] In one example, the time-frequency domain audio features include: filter group features; the feature extraction module 904, when used to extract features from the first time-frequency domain signal to obtain the time-frequency domain audio features, includes: setting a Mel filter group based on the sampling rate of the first time-frequency domain signal; using the Mel filter group to filter the linear spectrum of the first time-frequency domain signal to obtain the Mel spectrum energy of the first time-frequency domain signal; and taking the natural logarithm of the Mel spectrum energy to obtain the filter group features.

[0194] In one example, the time-frequency domain conversion module 902, when used to convert a first time domain signal in a song into a first time-frequency domain signal, includes: performing a short-time Fourier transform on the first time domain signal to obtain the first time-frequency domain signal; the time domain conversion module 910, when used to convert the second time-frequency domain signal into a second time domain signal to obtain an adjusted song, includes: performing an inverse short-time Fourier transform on the second time-frequency domain signal to obtain the second time domain signal.

[0195] In one example, the human voice extraction model includes: a nonlinear transformation unit, a multi-layer convolution unit and a masking processing unit; the human voice extraction module 906, when used to input the time-frequency domain audio features into the human voice extraction model to obtain the time-frequency mask output by the human voice extraction model, includes: splicing the time-frequency domain audio features to obtain spliced ​​features; wherein the time-frequency domain audio features are feature data of the first time-frequency domain signal on multiple channels; the nonlinear transformation unit performs nonlinear transformation on the spliced ​​features to obtain nonlinear transformation data; the multi-layer convolution unit performs convolution processing on the nonlinear transformation data to obtain filtered data; the masking processing unit calculates and outputs the time-frequency mask based on the filtered data; the time-frequency mask consists of a real part and an imaginary part.

[0196] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0197] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0198] In an exemplary embodiment, a non-temporary computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of a song vocal adjustment device to implement a method as described in any of the above embodiments.

[0199] The non-temporary computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc., and the present application does not limit this.

[0200] In an exemplary embodiment, a computer program product including a computer program / instructions is also provided. The computer program / instructions can be executed by a processor of a song vocal adjustment device to implement any of the methods described in the above embodiments.

[0201] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0202] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the inventions claimed herein. The present application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0203] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for adjusting the vocals of a song, characterized in that: The method is applied to a terminal device, and the method comprises: Converting a first time domain signal in the song into a first time-frequency domain signal; Performing feature extraction on the first time-frequency domain signal to obtain time-frequency domain audio features; Inputting the time-frequency domain audio features into a human voice extraction model to obtain a time-frequency mask output by the human voice extraction model; the time-frequency mask is used to represent the human voice signal in the first time-frequency domain signal; Adjusting the first time-frequency domain signal based on the time-frequency mask to obtain a second time-frequency domain signal; Converting the second time-frequency domain signal into a second time domain signal to obtain an adjusted song; Before inputting the time-frequency domain audio features into the human voice extraction model to obtain the time-frequency mask output by the human voice extraction model, the method further includes: The initial vocal extraction model is trained using the constructed training data to obtain the trained vocal extraction model; the training data includes at least one of the following: An analog audio signal, wherein the analog audio signal is obtained by simulating a human voice source and an accompaniment sound source; The audio signal synchronized with the audio track is an audio signal generated by separating and synchronizing the instrument components of the audio resource.

2. The method according to claim 1, characterized in that: The terminal device comprises: a system layer and an application layer, the system layer is deployed with the human voice extraction model, and the application layer is deployed with a music application; Before converting the first time domain signal in the song into the first time-frequency domain signal, the method further includes: The system layer receives the first time domain signal sent by the music application.

3. The method according to claim 1, characterized in that The step of converting the second time-frequency domain signal into a second time domain signal to obtain an adjusted song includes: Analyze the ambient audio signal collected from the terminal device to obtain a sing-along signal in the ambient audio signal; The second time domain signal and the sing-along signal are mixed to obtain a song that the user sings along with.

4. The method according to claim 1, characterized in that: The method of using the constructed training data to train the initial human voice extraction model to obtain the trained human voice extraction model includes: Using the training data, training the initial vocal extraction model; Calculate the loss value according to the time domain prediction loss and the frequency domain prediction loss; wherein the time domain prediction loss is the difference between the human voice signal in the training data and the predicted time-frequency mask output by the initial human voice extraction model in the time domain, and the frequency domain prediction loss is the difference between the human voice signal and the predicted time-frequency mask in the frequency domain; The loss value is minimized by adjusting the parameters of the initial human voice extraction model to obtain the human voice extraction model.

5. The method according to claim 1, characterized in that The time-frequency domain audio features include: filter bank features; The extracting features of the first time-frequency domain signal to obtain time-frequency domain audio features includes: Setting a Mel filter bank based on a sampling rate of the first time-frequency domain signal; Using the Mel filter bank, filtering the linear spectrum of the first time-frequency domain signal to obtain Mel spectrum energy of the first time-frequency domain signal; The natural logarithm of the Mel spectrum energy is calculated to obtain the filter bank characteristics.

6. The method according to claim 1, characterized in that The step of converting the first time domain signal in the song into a first time-frequency domain signal comprises: Performing short-time Fourier transform on the first time domain signal to obtain the first time-frequency domain signal; The step of converting the second time-frequency domain signal into a second time domain signal to obtain an adjusted song includes: Perform an inverse short-time Fourier transform on the second time-frequency domain signal to obtain the second time domain signal.

7. The method according to claim 1, characterized in that The human voice extraction model includes: a nonlinear transformation unit, a multi-layer convolution unit and a masking processing unit; The step of inputting the time-frequency domain audio features into a human voice extraction model to obtain a time-frequency mask output by the human voice extraction model comprises: Splicing the time-frequency domain audio features to obtain spliced ​​features; wherein the time-frequency domain audio features are feature data of the first time-frequency domain signal on multiple channels; The nonlinear transformation unit performs nonlinear transformation on the splicing features to obtain nonlinear transformation data; The multi-layer convolution unit performs convolution processing on the nonlinear transformation data to obtain filtered data; The mask processing unit calculates and outputs the time-frequency mask based on the filtered data; the time-frequency mask consists of a real part and an imaginary part.

8. A device for adjusting the vocals of a song, characterized in that: The device is applied to a terminal device, and the device includes: A time-frequency domain conversion module, used for converting a first time-domain signal in a song into a first time-frequency domain signal; A feature extraction module, used to extract features from the first time-frequency domain signal to obtain time-frequency domain audio features; A human voice extraction module, used for inputting the time-frequency domain audio features into a human voice extraction model to obtain a time-frequency mask output by the human voice extraction model; the time-frequency mask is used to represent the human voice signal in the first time-frequency domain signal; A human voice adjustment module, used for adjusting the first time-frequency domain signal based on the time-frequency masking to obtain a second time-frequency domain signal; A time domain conversion module, used for converting the second time-frequency domain signal into a second time domain signal to obtain an adjusted song; The human voice extraction module, before inputting the time-frequency domain audio features into the human voice extraction model to obtain the time-frequency mask output by the human voice extraction model, further includes: using the constructed training data to train the initial human voice extraction model to obtain the trained human voice extraction model; the training data includes at least one of the following: An analog audio signal, wherein the analog audio signal is obtained by simulating a human voice source and an accompaniment sound source; The audio signal synchronized with the audio track is an audio signal generated by separating and synchronizing the instrument components of the audio resource.

9. A terminal device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 7 by running the executable instructions.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product having a computer program / instructions stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images