Song adaptation method, computer device, and storage medium

Abstract

The application relates to a song adaptation method, a computer device and a storage medium. The method comprises the following steps: acquiring a sub-track signal of a to-be-adapted song and melody information of the sub-track signal; performing timbre rendering on the melody information of the sub-track signal to obtain a rendered sub-track signal of the to-be-adapted song; the timbre of the rendered sub-track signal is different from the timbre of the sub-track signal; performing loudness calibration on the rendered sub-track signal according to the sub-track signal to obtain an adapted sub-track signal of the to-be-adapted song; and performing sound mixing on the adapted sub-track signal to obtain a target adapted song of the to-be-adapted song. The method can improve the song adaptation efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, in particular to a song adaptation method, a computer device, a storage medium and a computer program product. BACKGROUND

[0002] With the gradually lowered threshold of music creation, more and more users want to create their own songs, and adapting existing songs has become the most common way of creation.

[0003] Currently, the original song is mainly listened to by artificial audition, and then the performance of the original song is analyzed and recorded, and the original song is adapted and created according to the performance of the original song. However, the artificial song adaptation method consumes high production cost, and the artificial production cycle is long, which leads to the defect that the artificial song adaptation method has low efficiency. SUMMARY

[0004] Therefore, it is necessary to provide a song adaptation method, a computer device and a computer readable storage medium capable of improving the efficiency of song adaptation.

[0005] In a first aspect, the present application provides a song adaptation method. The method comprises:

[0006] obtaining a track signal of a song to be adapted and melody information of the track signal;

[0007] rendering the melody information of the track signal in timbre to obtain a rendered track signal of the song to be adapted; the timbre of the rendered track signal is different from the timbre of the track signal;

[0008] performing loudness calibration on the rendered track signal according to the track signal to obtain an adapted track signal of the song to be adapted;

[0009] performing mixing processing on the adapted track signal to obtain a target adapted song of the song to be adapted.

[0010] In one of the embodiments, performing loudness calibration on the rendered track signal according to the track signal to obtain an adapted track signal of the song to be adapted comprises:

[0011] summing the square of the track signal in different time dimensions to obtain a to-be-processed track signal;

[0012] summing the square of the rendered track signal in different time dimensions to obtain a to-be-processed rendered track signal;

[0013] The loudness calibration parameter of the rendering track signal is obtained by dividing the to-be-processed track signal and the to-be-processed rendering track signal through a calibration parameter prediction model.

[0014] The rendering track signal of the to-be-adapted song is obtained by calibrating the loudness of the rendering track signal according to the loudness calibration parameter.

[0015] In one of the embodiments, the melody information of the track signal is timbre rendered to obtain the rendering track signal of the to-be-adapted song, including:

[0016] The target instrument timbre of the to-be-adapted song is determined according to the instrument type corresponding to the track signal; the instrument type corresponding to the target instrument timbre is the same as the instrument type corresponding to the track signal;

[0017] The melody information of the track signal is timbre rendered according to the target instrument timbre to obtain the rendering track signal of the to-be-adapted song.

[0018] In one of the embodiments, the melody information of the track signal is timbre rendered according to the target instrument timbre to obtain the rendering track signal of the to-be-adapted song, including:

[0019] The target renderer is set according to the target instrument timbre to obtain a target renderer.

[0020] The melody information of the track signal is audio signal synthesized according to the target instrument timbre through the target renderer to obtain the rendering track signal of the to-be-adapted song.

[0021] In one of the embodiments, the track signal of the to-be-adapted song and the melody information of the track signal are obtained, including:

[0022] The to-be-adapted song is input into the trained audio track separation model to obtain the track signal of the to-be-adapted song.

[0023] The melody information of the track signal is extracted from the track signal.

[0024] In one of the embodiments, the trained audio track separation model is trained in the following way:

[0025] The plurality of sample track signals are fused to obtain a sample mixed signal.

[0026] The sample mixed signal is input into the to-be-trained audio track separation model to obtain a predicted track signal of the sample mixed signal.

[0027] The loss value of the audio track separation model to be trained is obtained based on the time-domain norm between the sample track separation signal and the predicted track separation signal, and the number of predicted track separation signals.

[0028] Based on the loss value, the audio track separation model to be trained is iteratively trained to obtain the trained audio track separation model.

[0029] In one embodiment, inputting the sample mixture signal into the audio track separation model to be trained to obtain the predicted track separation signal of the sample mixture signal includes:

[0030] The temporal convolutional network in the audio track separation model to be trained is used to perform convolution processing on the sample mixed signal in the temporal dimension to obtain the temporal convolutional features of the sample mixed signal.

[0031] The frequency domain convolutional network in the audio track separation model to be trained is used to perform convolution processing on the sample mixed signal in the frequency domain dimension to obtain the frequency domain convolutional features of the sample mixed signal.

[0032] The temporal convolutional features and the frequency convolutional features are fused to obtain fused convolutional features.

[0033] The fused convolutional features are processed by convolution in the temporal dimension through the temporal convolutional network to obtain the target temporal features corresponding to the fused convolutional features.

[0034] The frequency domain convolutional network is used to perform convolution processing on the fused convolutional features in the frequency domain dimension to obtain the target frequency domain features corresponding to the fused convolutional features.

[0035] The predicted track-splitting signal is obtained by fusing the target's time-domain features and the target's frequency-domain features.

[0036] In one embodiment, extracting the melody information of the track-segmented signal from the track-segmented signal includes:

[0037] The track-splitting signal is subjected to Fourier transform processing to obtain the Mel spectrum of the track-splitting signal;

[0038] Using a music transcription model, the Mel spectrogram is transcribed using a multi-task, multi-track method to obtain the vocabulary marker sequence of the Mel spectrogram and the temporal information of the vocabulary marker sequence.

[0039] Based on the word tag sequence and the time information of the word tag sequence, the melody information of the track-separated signal is obtained.

[0040] In one embodiment, the Mel spectrogram is transcribed using a music transcription model through multi-task, multi-track music transcription to obtain a vocabulary marker sequence of the Mel spectrogram and temporal information of the vocabulary marker sequence, including:

[0041] The Mel spectrogram and the next word tag whose occurrence probability satisfies the preset probability condition are used as the input sequence of the music transcription model;

[0042] The input sequence is encoded and decoded using the music transcription model to output the vocabulary tag sequence of the Mel spectrogram and the time information of the vocabulary tag sequence; wherein the vocabulary tag sequence contains multiple vocabulary tags.

[0043] Secondly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0044] Obtain the track-by-track signals of the song to be adapted, as well as the melody information of the track-by-track signals;

[0045] The melody information of the track-segmented signal is rendered with timbre to obtain the rendered track-segmented signal of the song to be adapted; the timbre of the rendered track-segmented signal is different from that of the track-segmented signal.

[0046] Based on the track segmentation signal, the loudness of the rendered track segmentation signal is calibrated to obtain the adapted track segmentation signal of the song to be adapted.

[0047] The modified track signals are mixed to obtain the target modified version of the song to be modified.

[0048] Thirdly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0049] Obtain the track-by-track signals of the song to be adapted, as well as the melody information of the track-by-track signals;

[0050] The melody information of the track-segmented signal is rendered with timbre to obtain the rendered track-segmented signal of the song to be adapted; the timbre of the rendered track-segmented signal is different from that of the track-segmented signal.

[0051] Based on the track segmentation signal, the loudness of the rendered track segmentation signal is calibrated to obtain the adapted track segmentation signal of the song to be adapted.

[0052] The modified track signals are mixed to obtain the target modified version of the song to be modified.

[0053] Fourthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0054] Obtain the track-by-track signals of the song to be adapted, as well as the melody information of the track-by-track signals;

[0055] The melody information of the track-segmented signal is rendered with timbre to obtain the rendered track-segmented signal of the song to be adapted; the timbre of the rendered track-segmented signal is different from that of the track-segmented signal.

[0056] Based on the track segmentation signal, the loudness of the rendered track segmentation signal is calibrated to obtain the adapted track segmentation signal of the song to be adapted.

[0057] The modified track signals are mixed to obtain the target modified version of the song to be modified.

[0058] The aforementioned song adaptation method, computer equipment, and storage medium acquire the track-by-track signals of the song to be adapted, along with the melodic information of these signals. Then, the melodic information of the track-by-track signals is rendered to obtain the rendered track-by-track signals. Next, based on these rendered track-by-track signals, loudness calibration is performed to obtain the adapted track-by-track signals. Finally, the adapted track-by-track signals are mixed to obtain the target adapted song. This method enables timbre rendering based on the melodic information of the track-by-track signals, simulating the process of manually adapting a song without manual adaptation. This improves the efficiency of song adaptation while reducing costs. Furthermore, the loudness calibration process ensures that while there are differences between the adapted track-by-track signals and the original track-by-track signals, the similarity between them is also maintained. By mixing the adapted track-by-track signals to obtain the target adapted song, automatic song adaptation is achieved without manual adaptation, further improving efficiency and reducing costs. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating the song adaptation method in one embodiment.

[0060] Figure 2 This is a flowchart illustrating the steps for obtaining the adaptation track-splitting signal of the song to be adapted in one embodiment;

[0061] Figure 3 This is a schematic diagram of the audio track separation model in one embodiment;

[0062] Figure 4This is a schematic diagram illustrating the acquisition of melody information through multi-task, multi-track music transcription in one embodiment.

[0063] Figure 5 This is a flowchart illustrating the song adaptation method in another embodiment;

[0064] Figure 6 This is a flowchart illustrating the song adaptation method in yet another embodiment;

[0065] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0067] In one embodiment, such as Figure 1 As shown, a song adaptation method is provided. This embodiment illustrates the method applied to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. The server can be a standalone server or a server cluster consisting of multiple servers. In this embodiment, the method includes the following steps:

[0068] Step S101: Obtain the track-by-track signals of the song to be adapted, as well as the melody information of the track-by-track signals.

[0069] In this context, "song to be adapted" refers to a piece of music that needs to be adapted or re-performed. This can be an accompaniment extracted from a song or purely instrumental music. "Separate track signal" refers to the audio signal of a single (instrument) track. "Melody information" describes the pitch, notes, numerical control parameters of the notes, performance time, and end time of the separate track signal; this melody information can be a MIDI (Musical Instrument Digital Interface) file, a computer-understandable music standard format.

[0070] Specifically, the terminal separates the various tracks in the song to be adapted to obtain the track signals of different instruments corresponding to the song; then the terminal obtains the melody information of each track signal.

[0071] Step S102: The melody information of the track-by-track signal is rendered with timbre to obtain the rendered track-by-track signal of the song to be adapted; the timbre of the rendered track-by-track signal is different from that of the track-by-track signal.

[0072] Among them, the rendered track signal refers to the audio signal with a different timbre obtained after being rendered by the same type of instrument as the track signal.

[0073] Specifically, the terminal renders the timbre of the melody information of the track signals. This can be achieved by synthesizing audio using the same instrument type but different timbres based on the melody information of the track signals, resulting in a new track signal. This new track signal is then used as the rendered track signal for the song to be adapted. It's understandable that since the rendered track signal is based on the melody information of the track signals, the melody information of the rendered track signal is the same as that of its corresponding track signal, maintaining a certain degree of similarity. Furthermore, since the instrument type used for rendering is the same as that of the track signal, the similarity (or proximity) between the rendered track signal and the track signal is further enhanced. Moreover, the timbre used for rendering is different from the timbre of the track signal, allowing the rendered track signal to approach the track signal while maintaining a difference, thus achieving the effect of song adaptation.

[0074] For example, suppose the song to be adapted is played on multiple instruments such as guitar, piano, and bass. We can first extract the individual track signals for each instrument (to distinguish them, we can call them guitar track signal, piano track signal, and bass track signal, etc.). Taking the guitar track signal as an example, the original guitar track signal is played on a classical guitar, which has a relatively mellow and bright tone. The terminal can then select the tone of an electric guitar and render the melody information of the track signal to obtain a rendered track signal. Although the rendered track signal has the same melody information as the guitar track signal, its tone is rougher and has a metallic feel, while the guitar track signal has a more mellow and bright tone.

[0075] Step S103: Based on the track segmentation signal, perform loudness calibration on the rendered track segmentation signal to obtain the adapted track segmentation signal of the song to be adapted.

[0076] Among them, the adapted track-segment signal refers to the audio signal obtained after the track-segment signal has been adapted using the song adaptation methods described above. The adapted track-segment signal is close to the original track-segment signal, but the two retain their differences. Both the adapted track-segment signal and the track-segment signal are audio signals of a single (instrument) track.

[0077] Specifically, the terminal can perform loudness calibration on the rendered track signals based on the track signals. This can be done by scaling the loudness of the rendered track signals according to the loudness of the track signals themselves, in order to avoid significant changes in the loudness of the rendered track signals due to differences in timbre. If there is too much difference between the loudness of the rendered track signals and the original track signals, the terminal obtains the adapted track signals corresponding to the track signals, that is, it obtains the adapted track signals of the song to be adapted. This ensures that the loudness of each adapted track signal is consistent with the original loudness of its corresponding track signal, thus ensuring the similarity between the adapted song and the original song to be adapted.

[0078] Step S104: Mix the adapted track signals to obtain the target adapted song.

[0079] The target adapted song refers to the song obtained after the original song has been adapted. The target adapted song is similar to the original song, but the two maintain their differences.

[0080] Specifically, after acquiring the adapted track signals corresponding to each track signal, the terminal can also perform mixing processing on all the adapted track signals, which can be done by superimposing all the adapted track signals into an audio track, thus obtaining the target adapted song of the song to be adapted.

[0081] Understandably, the song adaptation method in this paper involves modifying or re-performing the original song, making the adapted song (or adapted track signal) closely resemble the original song (or track signal) in terms of melodic information (such as pitch and rhythm) and loudness. However, differences exist (for example, the adapted track signal has a fuller and brighter timbre). It does not completely rewrite the original song (or track signal) in terms of melody, loudness, timbre, and arrangement, thus avoiding significant differences between the adapted song (or adapted track signal) and the original song. For example, in karaoke scenarios, using a song's accompaniment often requires considering copyright ownership, making it inconvenient to use the original accompaniment. The song adaptation method described above modifies the original song's accompaniment into a target accompaniment that is similar to the original but differs from it. Using this target accompaniment for karaoke not only maintains the user experience but also reduces operating costs.

[0082] In the aforementioned song adaptation method, the process involves acquiring the track-by-track signals of the song to be adapted, along with their melodic information. Then, the melodic information of the track-by-track signals is rendered to obtain the rendered track-by-track signals. Next, based on these rendered track-by-track signals, loudness calibration is performed to obtain the adapted track-by-track signals. Finally, the adapted track-by-track signals are mixed to obtain the target adapted song. This method allows for timbre rendering and loudness calibration based on the melodic information of the track-by-track signals, simulating the process of manually adapting a song without manual intervention. This improves adaptation efficiency and reduces costs. Furthermore, the loudness calibration process ensures that while the adapted track-by-track signals differ from the original signals, they also maintain a degree of similarity. By mixing the adapted track-by-track signals to obtain the target adapted song, automatic song adaptation is achieved without manual intervention, further improving efficiency and reducing costs.

[0083] In one embodiment, such as Figure 2 As shown, step S103 above, based on the track-by-track signal, performs loudness calibration on the rendered track-by-track signal to obtain the adapted track-by-track signal of the song to be adapted, specifically including the following:

[0084] Step S201: Sum the squares of the track-segmented signals under different time dimensions to obtain the track-segmented signals to be processed.

[0085] Step S202: Sum the squares of the rendering track signals under different time dimensions to obtain the rendering track signals to be processed.

[0086] Step S203: Using the calibration parameter prediction model, divide the signal to be processed and the signal to be rendered by the signal to be processed to obtain the loudness calibration parameters of the rendered signal.

[0087] The calibration parameter prediction model refers to the model used to calculate the loudness calibration parameters of the rendered track signals. The loudness calibration parameters are used to adjust the loudness of the rendered track signals.

[0088] To make the adapted track signals more closely resemble the original track signals of the song to be adapted, loudness calibration can be performed on the rendered track signals. This ensures that the loudness of each adapted track signal is consistent with the original track signals, while also making the target adapted song more similar to the original song. Specifically, the terminal inputs the track signals and the rendered track signals into a calibration parameter prediction model. By summing the squares of the track signals at different time dimensions, the track signal to be processed is obtained; similarly, by summing the squares of the rendered track signals at different time dimensions, the rendered track signal to be processed is obtained. The calibration parameter prediction model can then calculate the loudness calibration parameters of the rendered track signal based on both the track signal to be processed and the rendered track signal to be processed.

[0089] In practical applications, the calibration parameter prediction model can be the formula shown in formula (1). The terminal can input the track split signal and the rendering track split signal into formula (1), and then calculate the loudness calibration parameters of the rendering track split signal through the calibration parameter prediction model shown in formula (1).

[0090]

[0091] Where α represents the loudness calibration parameter; y(t) represents the track-segmented signal; x(t) represents the rendered track-segmented signal; M represents the total duration of the track-segmented signal (or the rendered track-segmented signal); and t represents the time dimension.

[0092] Step S204: According to the loudness calibration parameters, perform loudness calibration on the rendered track signal to obtain the adapted track signal of the song to be adapted.

[0093] Specifically, the terminal performs loudness calibration on the rendered track signal based on loudness calibration parameters. This can be achieved by multiplying the loudness calibration parameters by the rendered track signal, thus scaling the loudness of the rendered track signal using the loudness calibration parameters. The terminal then obtains the adapted track signal for the song to be adapted. In practical applications, the adapted track signal x new (t) can also be calculated using the formula (2) shown below:

[0094] x new (t)=αx(t) (2)

[0095] In this embodiment, the loudness calibration parameters of the rendered track signal are first calculated using the track-splitting signal and the rendered track-splitting signal. Then, based on the loudness calibration parameters, the loudness of the rendered track signal is calibrated to obtain the adapted track signal of the song to be adapted. This avoids a large difference between the loudness of the processed track signal (such as the adapted track signal) and the loudness of the original track signal, ensuring that the loudness of the processed adapted track signal is consistent with the original loudness of its corresponding track signal. This improves the adaptation efficiency while also ensuring the similarity between the adapted track signal and the track signal of the song to be adapted.

[0096] In one embodiment, step S102 above involves rendering the melody information of the track-by-track signal to obtain the rendered track-by-track signal of the song to be adapted. The timbre of the rendered track-by-track signal is different from that of the track-by-track signal, specifically including the following: determining the target instrument timbre of the song to be adapted based on the instrument type corresponding to the track-by-track signal; the instrument type corresponding to the target instrument timbre is the same as the instrument type corresponding to the track-by-track signal; and rendering the melody information of the track-by-track signal based on the target instrument timbre to obtain the rendered track-by-track signal of the song to be adapted.

[0097] The target instrument timbre refers to the selected timbre used to render melody information.

[0098] Specifically, the terminal determines the instrument type corresponding to the track-by-track signal, and then selects an instrument of the same type but with a different timbre as the target instrument. The timbre of this target instrument becomes the target instrument timbre for the song to be adapted. The terminal uses the target instrument timbre to render the melody information of the track-by-track signal. This can be achieved by using the target instrument timbre to perform audio synthesis according to the melody information of the track-by-track signal, thereby obtaining a rendered track-by-track signal that has the same melody information as the track-by-track signal but a different timbre.

[0099] In this embodiment, the target instrument timbre of the song to be adapted is first determined according to the instrument type corresponding to the track signal; then, the melody information of the track signal is rendered according to the target instrument timbre, thereby obtaining a rendered track signal that is close to the track signal but maintains the timbre difference, thus realizing the reasonable adaptation of the track signal of the song to be adapted.

[0100] In one embodiment, the melody information of the track-by-track signal is rendered according to the timbre of the target instrument to obtain the rendered track-by-track signal of the song to be adapted. Specifically, this includes: setting a renderer according to the timbre of the target instrument to obtain a target renderer; and using the target renderer to synthesize audio signals from the melody information of the track-by-track signal according to the timbre of the target instrument to obtain the rendered track-by-track signal of the song to be adapted.

[0101] In this context, the target renderer refers to the tool used to render or further process audio signals. Both the renderer and the target renderer can be FluidSynth (a real-time MIDI synthesizer), which can synthesize audio and freely control and adjust the effects of the synthesized audio.

[0102] Specifically, after determining the timbre of the target instrument, the terminal can set the relevant parameters of the target instrument timbre in the renderer. After setting, a usable target renderer is obtained. Then, the melody information of the track-by-track signal is input to the target renderer. Based on the melody information, the target renderer synthesizes the audio signal according to the timbre of the target instrument, and the terminal obtains the rendered track-by-track signal of the song to be adapted.

[0103] For example, assuming the target instrument's timbre is that of an electric guitar A, the terminal can use FluidSynth to select the timbre of electric guitar A from the MIDI file of the multi-track signal to synthesize the audio signal, thereby obtaining the rendered multi-track information with the timbre of electric guitar A.

[0104] In this embodiment, a renderer is first set according to the timbre of the target instrument to obtain a configured target renderer. Then, the target renderer synthesizes the audio signal of the melody information of the track signal according to the timbre of the target instrument to obtain the rendered track signal of the song to be adapted. The renderer can automatically realize the reasonable adaptation of the track signal of the song to be adapted without having to manually replay it with instruments of different timbres, thus improving the adaptation efficiency of the track signal.

[0105] In one embodiment, step S101 above, obtaining the track-separated signal of the song to be adapted and the melody information of the track-separated signal, specifically includes the following: inputting the song to be adapted into the trained track separation model to obtain the track-separated signal of the song to be adapted; extracting the melody information of the track-separated signal from the track-separated signal.

[0106] Among them, the track separation model refers to the model used to separate the track signals of different musical instruments.

[0107] Specifically, the terminal pre-trains a track separation model. After acquiring the song to be adapted, the song can be input into the trained track separation model, which separates the track signals of different instruments. Then, the terminal extracts the melody information of each track signal from the track signals of different instruments.

[0108] In this embodiment, the track-separation signal of the song to be adapted is obtained by inputting the song to be adapted into the trained track separation model; then the melody information of the track-separation signal is extracted from the track-separation signal, so as to realize the reasonable acquisition of the track-separation signal and melody information of the song to be adapted, so that the track-separation signal and melody information can be used as the basis for processing to perform subsequent song adaptation steps.

[0109] In one embodiment, the trained audio track separation model is trained as follows: multiple sample track separation signals are fused to obtain a sample mixed signal; the sample mixed signal is input into the audio track separation model to be trained to obtain the predicted track separation signal of the sample mixed signal; the loss value of the audio track separation model to be trained is obtained based on the time-domain norm between the sample track separation signal and the predicted track separation signal, and the number of predicted track separation signals; the audio track separation model to be trained is iteratively trained based on the loss value to obtain the trained audio track separation model.

[0110] The audio track separation model can be constructed based on an end-to-end time-frequency domain model. The audio track separation model can also be a hybrid demucs model (a music source separation model).

[0111] The sample track-separated signal is also an audio signal of a single (instrument) track, but it is used as training data for training the track separation model. The sample mixed signal is an audio signal that combines multiple (instrument) tracks, and it is also used as training data for training the track separation model.

[0112] The terminal can first construct training data for training the track separation model. Specifically, the terminal can obtain multiple sample track separation signals from a dataset or database; then, the terminal fuses these multiple sample track separation signals. This fusion can be achieved by fusing multiple sample track separation signals of the same instrument type but different timbres, or by fusing multiple sample track separation signals of different instrument types, resulting in a sample mixed signal. Further, the terminal inputs the sample mixed signal into the track separation model to be trained, and the model separates the predicted track separation signals from the sample mixed signal. Based on the time-domain norm between the sample track separation signals and the predicted track separation signals, and the number of predicted track separation signals, the terminal obtains the loss value of the track separation model to be trained. The terminal then iteratively updates the model parameters of the track separation model using the loss value. When a preset training termination condition is met, the terminal obtains the trained track separation model. The loss value of the track separation model to be trained... It can be calculated using the formula (3) shown below:

[0113]

[0114] Where J represents the number of predicted track-separated signals separated by the track separation model; the subscript 1 represents the first-order norm in the time domain. y represents the j-th predicted track splitting signal; j (t) represents the sample track segmentation signal corresponding to the j-th predicted track segmentation signal; t represents the time dimension.

[0115] In this embodiment, the fused sample mixed signal is input into the track separation model to be trained to obtain the predicted track separation signal of the sample mixed signal; then, based on the time-domain norm between the sample track separation signal and the predicted track separation signal, and the number of predicted track separation signals, the loss value of the track separation model to be trained is obtained; then, based on the loss value, the track separation model to be trained is iteratively trained to obtain the trained track separation model, thus realizing the training of the track separation model so that the subsequent song adaptation steps can use the trained track separation model to process the obtained track separation signal.

[0116] In one embodiment, the sample mixed signal is input into the track separation model to be trained to obtain the predicted track-separated signal of the sample mixed signal. Specifically, this includes: using a temporal convolutional network in the track separation model to be trained, convolving the sample mixed signal in the temporal dimension to obtain temporal convolutional features of the sample mixed signal; using a frequency-domain convolutional network in the track separation model to be trained, convolving the sample mixed signal in the frequency domain dimension to obtain frequency-domain convolutional features of the sample mixed signal; fusing the temporal and frequency-domain convolutional features to obtain fused convolutional features; using a temporal convolutional network to convolve the fused convolutional features in the temporal dimension to obtain target temporal features corresponding to the fused convolutional features; using a frequency-domain convolutional network to convolve the fused convolutional features in the frequency domain dimension to obtain target frequency-domain features corresponding to the fused convolutional features; and fusing the target temporal and target frequency-domain features to obtain the predicted track-separated signal.

[0117] Figure 3 The diagram shows the structure of the audio track separation model, which includes multiple temporal convolutional networks (TDecorder, TEncorder), multiple frequency convolutional networks (ZDecorder, ZEncorder), a shared encoder (Eecoder6), and a shared decoder (Decoder6). The temporal and frequency convolutional networks can be constructed from the encoder and decoder. Figure 3 In this context, `timesteps` represents the time step, `freq` represents the sampling frequency, and `C` represents the sampling frequency. in Indicates the size of the input data, C out Indicates the size of the output data.

[0118] Specifically, such asFigure 3 As shown, the terminal can first perform a Short-Time Fourier Transform (STFT) on the sample mixed signal to obtain its spectrum. Then, multiple time-domain convolutional networks constructed by the encoder are used to convolve the spectrum in the time domain to obtain its time-domain convolutional features. Simultaneously, multiple frequency-domain convolutional networks constructed by the encoder are used to convolve the spectrum in the frequency domain to obtain its frequency-domain convolutional features. Next, the time-domain and frequency-domain convolutional features are fused to obtain the fused convolutional features. Finally, the shared encoder and shared decoder in the track separation model to be trained are used to convolve the fused convolutional features to obtain the shared convolutional features. The terminal uses multiple temporal convolutional networks constructed by the decoder to perform convolution processing on shared convolutional features in the temporal dimension. Simultaneously, it uses multiple frequency-domain convolutional networks constructed by the decoder to perform convolution processing on the shared convolutional features in the frequency domain dimension. The terminal then obtains the target temporal features output by the last temporal convolutional network and the target frequency-domain features output by the last frequency-domain convolutional network. By fusing the target temporal and frequency-domain features, the terminal obtains a predicted spectrogram. Finally, the terminal performs an inverse short-time Fourier transform (ISTFT) on the predicted spectrogram to obtain the predicted track-separated signal. It can be understood that the processing steps of the trained track separation model for the song to be adapted are the same as the processing steps of the track separation model to be trained for the sample mixed signal.

[0119] In this embodiment, the sample mixed signal is convolved in the time domain dimension through the temporal convolutional network in the track separation model to be trained, and the sample mixed signal is convolved in the frequency domain dimension through the frequency domain convolutional network in the track separation model to be trained. This can uncover richer features of the sample mixed signal in the frequency and time domains, which is beneficial to improving the track separation capability of the trained track separation model.

[0120] In one embodiment, extracting the melody information of the track-segmented signal from the track-segmented signal specifically includes the following: performing Fourier transform processing on the track-segmented signal to obtain the Mel spectrogram of the track-segmented signal; performing multi-task multi-track music transcription on the Mel spectrogram using a music transcription model to obtain the vocabulary tag sequence of the Mel spectrogram and the time information of the vocabulary tag sequence; and obtaining the melody information of the track-segmented signal based on the vocabulary tag sequence and the time information of the vocabulary tag sequence.

[0121] Specifically, the terminal obtains the Mel spectrogram of the track-by-track signal by performing Fourier transform processing on the track-by-track signal; then, through a music transcription model, it performs multi-task multitrack music transcription (MT3) on the Mel spectrogram. This can be achieved by inputting the Mel spectrogram into a music transcription model trained on MT3, and then outputting the vocabulary tag sequence of the Mel spectrogram, as well as the time information of the vocabulary tag sequence, through the music transcription model; obtaining the preset mapping relationship between the vocabulary tag sequence and notes, and converting the vocabulary tag sequence and the time information of the vocabulary tag sequence into the melody information of the track-by-track signal according to the preset mapping relationship; wherein, the melody information can be a MIDI file containing instrument type, notes, pitch, note start and end, time information, and slur information. It is understood that the melody information in this method does not itself contain waveform data, but rather records sound information, which is a set of instructions used to instruct the target renderer on how to reproduce the timbre of the target instrument in the above-mentioned target renderer.

[0122] Figure 4 This is a schematic diagram illustrating the acquisition of melodic information through multi-task, multi-track music transcription, such as... Figure 4 As shown, the performance of the music transcription model trained on MT3 can be validated using six datasets of different sizes, recording processes, instruments, and types. These six datasets are: MAESTRO, whose audio and detailed MIDI data are collected from performers playing on a Disklavier piano; Slakh2100, composed of audio generated by rendering MIDI files using professional-grade, sample-based synthesis software; Cerberus4, derived from the Slakh2100 dataset, composed of four instruments (guitar, bass, drums, and piano) mixed in tracks where the instruments are active; GuitarSet, composed of live guitar performances of various types, rhythms, and styles, recorded using a high-precision six-tone pickup that captures the sound of each guitar string individually; MusicNet, recordings from classical instruments such as classical guitar, drum kit, and ensemble instruments; and URMP, composed of classic works with multiple instruments, such as cello and violin. The terminal inputs six datasets—MAESTRO, Cerberus4, GuitarSet, MusicNet, Slakh2100, and URMP—into the music transcription model, which then outputs the corresponding MIDI file for each dataset.

[0123] In this embodiment, the Mel spectrogram of the track-segmented signal is obtained by performing Fourier transform on the track-segmented signal; the Mel spectrogram is then transcribed using a music transcription model to obtain the lexical tag sequence and the temporal information of the lexical tag sequence; and the melody information of the track-segmented signal is obtained based on the lexical tag sequence and the temporal information of the lexical tag sequence. This allows the melody information to be used as the basis for subsequent timbre rendering steps, and the track-segmented signal can be adapted in conjunction with the target renderer.

[0124] In one embodiment, a music transcription model is used to perform multi-task, multi-track music transcription on a Mel spectrogram to obtain a lexical marker sequence of the Mel spectrogram and the temporal information of the lexical marker sequence. Specifically, this includes: using the Mel spectrogram and the next lexical marker whose occurrence probability satisfies a preset probability condition as the input sequence of the music transcription model; using the music transcription model to encode and decode the input sequence to output the lexical marker sequence of the Mel spectrogram and the temporal information of the lexical marker sequence; wherein, the lexical marker sequence contains multiple lexical markers.

[0125] The preset probability condition refers to the judgment condition set for the probability of the next word token appearing. The predicted probability condition can be the highest probability of occurrence. A word token refers to a token vocabulary built based on a MIDI file, which can represent the corresponding melody information. The time information refers to the start and end times of the corresponding word token.

[0126] Specifically, upon initial input, the terminal uses the Mel spectrogram as the input sequence. The music transcription model encodes and decodes the input sequence, outputting a word tag and its temporal information. For subsequent inputs, the terminal selects the next word tag with the highest probability from the output word tags and uses it as the next word tag. The terminal then uses the Mel spectrogram and the next word tag as the input sequence for the music transcription model. The music transcription model encodes and decodes the input sequence, outputting a word tag and its temporal information. The word tag sequence is formed from the word tags output each time, and the temporal information of the word tag sequence is determined.

[0127] In this embodiment, the Mel spectrogram and the next word tag whose occurrence probability satisfies the preset probability condition are used as the input sequence of the music transcription model. The music transcription model encodes and decodes the input sequence to output the word tag sequence of the Mel spectrogram and the time information of the word tag sequence. The accurate acquisition of the word tag sequence is achieved through the music transcription model MT3, so that the melody information of the track-separated signal can be obtained in subsequent steps based on the word tag sequence and its time information.

[0128] In one embodiment, such as Figure 5 As shown, another method for song adaptation is provided. Taking the application of this method to a terminal as an example, the steps include:

[0129] Step S501: Input the song to be adapted into the trained audio track separation model to obtain the track separation signal of the song to be adapted.

[0130] Step S502: Perform Fourier transform processing on the track-segmented signal to obtain the Mel spectrum of the track-segmented signal; use the Mel spectrum and the next word tag whose occurrence probability satisfies the preset probability condition as the input sequence of the music transcription model.

[0131] Step S503: The input sequence is encoded and decoded using a music transcription model to output the vocabulary marker sequence of the Mel spectrogram and the time information of the vocabulary marker sequence.

[0132] The lexical tag sequence contains multiple lexical tags.

[0133] Step S504: Based on the word tag sequence and the time information of the word tag sequence, obtain the melody information of the track-by-track signal.

[0134] Step S505: Determine the target instrument timbre of the song to be adapted based on the instrument type corresponding to the track-separated signal.

[0135] The instrument type corresponding to the target instrument timbre is the same as the instrument type corresponding to the track-splitting signal.

[0136] Step S506: Set the renderer according to the timbre of the target instrument to obtain the target renderer; use the target renderer to synthesize the audio signal of the melody information of the track signal according to the timbre of the target instrument to obtain the rendered track signal of the song to be adapted.

[0137] Step S507: Sum the squares of the track-segmented signals under different time dimensions to obtain the track-segmented signals to be processed; sum the squares of the rendered track-segmented signals under different time dimensions to obtain the rendered track-segmented signals to be processed.

[0138] Step S508: Using the calibration parameter prediction model, divide the signal to be processed and the signal to be rendered by the signal to be processed to obtain the loudness calibration parameters of the rendered signal.

[0139] Step S509: According to the loudness calibration parameters, perform loudness calibration on the rendered track signal to obtain the adapted track signal of the song to be adapted.

[0140] Step S510: Mix the adapted track signals to obtain the target adapted song.

[0141] The aforementioned song adaptation method achieves the following beneficial effects: It can render the timbre and perform loudness calibration based on the melodic information of the track-by-track signals to simulate the process of manually adapting a song, eliminating the need for manual adaptation and thus improving efficiency while reducing costs. Furthermore, the loudness calibration process ensures that while there are differences between the adapted and original track-by-track signals, the similarity between them is also maintained. By mixing the adapted track-by-track signals, the target adapted song is obtained, achieving automatic song adaptation without manual intervention, further improving efficiency and reducing costs.

[0142] To more clearly illustrate the song adaptation method provided in this disclosure, a specific embodiment will be used to describe the above-mentioned song adaptation method in detail below. For example... Figure 6 As shown, another method for song adaptation is provided, which can be applied to the terminal, and specifically includes the following:

[0143] If the music to be adapted is the accompaniment of a song, the terminal can separate the tracks of the accompaniment to obtain the piano track signal and the guitar track signal. Then, the terminal extracts the piano MIDI file from the piano track signal and the guitar MIDI file from the guitar track signal. Next, the terminal determines the target piano timbre and the target guitar timbre respectively, and renders the piano MIDI file according to the target piano timbre to obtain the rendered piano track signal of the accompaniment. At the same time, it renders the guitar MIDI file according to the target guitar timbre to obtain the rendered guitar track signal of the accompaniment. The terminal performs loudness calibration on the rendered piano track signal according to the piano track signal to obtain the adapted piano track signal of the accompaniment. At the same time, it performs loudness calibration on the rendered guitar track signal according to the guitar track signal to obtain the adapted guitar track signal of the accompaniment. Finally, the adapted guitar track signal and the adapted piano signal are mixed to obtain the target adapted accompaniment.

[0144] In this embodiment, the terminal automatically completes the accompaniment adaptation process based on the extracted track-by-track signals and melody information, without requiring manual re-performance of the accompaniment. This improves the efficiency of song adaptation while reducing adaptation costs. Furthermore, loudness calibration processing ensures that while there are differences between the target adapted accompaniment and the original accompaniment, the similarity between the target adapted accompaniment and the original accompaniment is also maintained.

[0145] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0146] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a song adaptation method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0147] Those skilled in the art will understand thatFigure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0148] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0149] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0150] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0151] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0152] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0153] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for adapting a song, characterized in that, The method includes: Obtain the track-by-track signals of the song to be adapted, as well as the melody information of the track-by-track signals; The target instrument timbre of the song to be adapted is determined based on the instrument type corresponding to the track splitting signal; the instrument type corresponding to the target instrument timbre is the same as the instrument type corresponding to the track splitting signal. Based on the timbre of the target instrument, the melody information of the track-by-track signal is rendered to obtain the rendered track-by-track signal of the song to be adapted; the timbre of the rendered track-by-track signal is different from the timbre of the track-by-track signal. Based on the track segmentation signal, the loudness of the rendered track segmentation signal is calibrated to obtain the adapted track segmentation signal of the song to be adapted. The modified track signals are mixed to obtain the target modified version of the song to be modified.

2. The method according to claim 1, characterized in that, The step of performing loudness calibration on the rendered track-segment signal based on the track-segment signal to obtain the adapted track-segment signal of the song to be adapted includes: The squares of the track-segmented signals under different time dimensions are summed to obtain the track-segmented signals to be processed; The squares of the rendering track signals under different time dimensions are summed to obtain the rendering track signals to be processed. By using a calibration parameter prediction model, the loudness calibration parameters of the rendered track signal are obtained by dividing the track signal to be processed and the rendered track signal to be processed. Based on the loudness calibration parameters, the loudness of the rendered track signal is calibrated to obtain the adapted track signal of the song to be adapted.

3. The method according to claim 1, characterized in that, The step of rendering the melody information of the track-by-track signal with timbre based on the timbre of the target instrument to obtain the rendered track-by-track signal of the song to be adapted includes: The renderer is set according to the timbre of the target musical instrument to obtain the target renderer; The target renderer synthesizes the melody information of the track-by-track signal according to the timbre of the target instrument to obtain the rendered track-by-track signal of the song to be adapted.

4. The method according to claim 1, characterized in that, The acquisition of the track-by-track signals of the song to be adapted, and the melody information of the track-by-track signals, includes: The song to be adapted is input into the trained audio track separation model to obtain the track-separated signal of the song to be adapted. The melody information of the track-segmented signal is extracted from the track-segmented signal.

5. The method according to claim 4, characterized in that, The trained audio track separation model was obtained through the following method: Multiple sample track signals are fused to obtain a sample mixed signal; The sample mixed signal is input into the audio track separation model to be trained to obtain the predicted track separation signal of the sample mixed signal; The loss value of the audio track separation model to be trained is obtained based on the time-domain norm between the sample track separation signal and the predicted track separation signal, and the number of predicted track separation signals. Based on the loss value, the audio track separation model to be trained is iteratively trained to obtain the trained audio track separation model.

6. The method according to claim 5, characterized in that, The step of inputting the sample mixture signal into the audio track separation model to be trained to obtain the predicted track separation signal of the sample mixture signal includes: The temporal convolutional network in the audio track separation model to be trained is used to perform convolution processing on the sample mixed signal in the temporal dimension to obtain the temporal convolutional features of the sample mixed signal. The frequency domain convolutional network in the audio track separation model to be trained is used to perform convolution processing on the sample mixed signal in the frequency domain dimension to obtain the frequency domain convolutional features of the sample mixed signal. The temporal convolutional features and the frequency convolutional features are fused to obtain fused convolutional features. The fused convolutional features are processed by convolution in the temporal dimension through the temporal convolutional network to obtain the target temporal features corresponding to the fused convolutional features. The frequency domain convolutional network is used to perform convolution processing on the fused convolutional features in the frequency domain dimension to obtain the target frequency domain features corresponding to the fused convolutional features. The predicted track-splitting signal is obtained by fusing the target's time-domain features and the target's frequency-domain features.

7. The method according to claim 4, characterized in that, The step of extracting the melody information of the track-segmented signal from the track-segmented signal includes: The track-splitting signal is subjected to Fourier transform processing to obtain the Mel spectrum of the track-splitting signal; Using a music transcription model, the Mel spectrogram is transcribed using a multi-task, multi-track method to obtain the vocabulary marker sequence of the Mel spectrogram and the temporal information of the vocabulary marker sequence. Based on the word tag sequence and the time information of the word tag sequence, the melody information of the track-separated signal is obtained.

8. The method according to claim 7, characterized in that, The process involves using a music transcription model to perform multi-task, multi-track music transcription on the Mel spectrogram to obtain the vocabulary marker sequence of the Mel spectrogram and the temporal information of the vocabulary marker sequence, including: The Mel spectrogram and the next word tag whose occurrence probability satisfies the preset probability condition are used as the input sequence of the music transcription model; The input sequence is encoded and decoded using the music transcription model to output the vocabulary tag sequence of the Mel spectrogram and the time information of the vocabulary tag sequence; wherein the vocabulary tag sequence contains multiple vocabulary tags.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images