Music style transfer method and device, computer device and storage medium

Abstract

The application discloses a music style migration method, which comprises the following steps: obtaining to-be-combined music and to-be-migrated music; inputting the to-be-combined music and the to-be-migrated music into a preset style decoupling model respectively to obtain a first pitch content code, a second pitch content code and a rhythm code; inputting the first pitch content code and the rhythm code into a first decoder to obtain migrated music; inputting the second pitch content code and the rhythm code into a second decoder to obtain reconstructed music; detecting the migrated music and the reconstructed music by using a discriminator to obtain a style migration detection result; and determining the migrated music as target music when the style migration detection result represents successful migration. The preset style decoupling model is used for completely decoupling music, thereby avoiding mutual mixing between pitch content and rhythm style. The discriminator is used for detecting the migrated music and the reconstructed music, thereby ensuring successful rhythm style migration and improving the effect of music style migration.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and in particular to a method, apparatus, computer device, and storage medium for music style transfer. Background Technology

[0002] With the continuous development of science and technology, speech style transfer technology has been gradually applied in various fields, such as voice changing systems and voice chat. Speech style transfer refers to generating speech with the characteristics of a given speaker from any text sequence.

[0003] In existing technologies, speech style transfer often involves decoupling and recombining speech component features. Speech decoupling breaks down the original speech into multiple audio components, but pitch content and rhythm features are often still mixed, failing to completely decouple the original speech's audio components. Recombination refers to converting pitch content features into speech of any style, given any rhythmic feature. This method is ineffective for music style transfer, falling far short of user needs and severely impacting the user experience. Summary of the Invention

[0004] This invention provides a music style transfer method, apparatus, computer device, and storage medium to solve the problem of poor music style transfer results in the prior art.

[0005] A method for music style transfer includes:

[0006] Obtain the music to be synthesized and the music to be transferred;

[0007] Obtain a preset style decoupling model, input the music to be synthesized and the music to be transferred into the preset style decoupling model respectively, obtain the first pitch content code corresponding to the music to be synthesized, and obtain the second pitch content code and rhythm code corresponding to the music to be transferred;

[0008] The first pitch content encoding and the rhythm encoding are input into the first decoder to obtain the transferred music;

[0009] The second pitch content encoding and the rhythm encoding are input into the second decoder to obtain the reconstructed music;

[0010] The style transfer detection results are obtained by detecting the transferred music and the reconstructed music using a discriminator.

[0011] When the style transfer detection result indicates that the rhythm style transfer is successful, the transferred music is identified as the target music.

[0012] A music style transfer device, comprising:

[0013] The music acquisition module is used to acquire the music to be synthesized and the music to be transferred; the music to be synthesized and the music to be transferred are different pieces of music.

[0014] The encoding acquisition module is used to acquire a preset style decoupling model, input the music to be synthesized and the music to be transferred into the preset style decoupling model respectively, acquire the first pitch content code corresponding to the music to be synthesized, and acquire the second pitch content code and rhythm code corresponding to the music to be transferred;

[0015] The first decoding module is used to input the first pitch content encoding and the rhythm encoding into the first decoder to obtain the transferred music;

[0016] The second decoding module is used to input the second pitch content encoding and the rhythm encoding into the second decoder to obtain the reconstructed music;

[0017] The detection result determination module is used to detect the transferred music and the reconstructed music based on the discriminator to obtain the style transfer detection result;

[0018] The migration success determination module is used to determine the migrated music as the target music when the style migration detection result indicates that the rhythm style migration is successful.

[0019] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described music style transfer method.

[0020] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described music style transfer method.

[0021] This invention provides a method, apparatus, computer device, and storage medium for music style transfer. The method uses a preset style decoupling model to decouple the synthesized music and the music to be transferred, thereby achieving complete decoupling of pitch content and rhythmic style in the music. This avoids mutual mixing between pitch content and rhythmic style, simplifying the music style decoupling process. A discriminator detects the transferred music after style transfer and the reconstructed music after reconstruction of the music to be transferred, thus determining the style transfer detection result and judging whether the music style transfer was successful. When the style transfer detection result indicates successful transfer, the transferred music is identified as the target music, ensuring successful music style transfer, improving the effect of music style transfer, and enhancing the user experience. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the application environment of the music style transfer method in one embodiment of the present invention;

[0024] Figure 2 This is a flowchart of a music style transfer method according to an embodiment of the present invention;

[0025] Figure 3 This is a flowchart of step S20 in a music style transfer method according to an embodiment of the present invention;

[0026] Figure 4 This is a flowchart of step S201 in a music style transfer method according to an embodiment of the present invention;

[0027] Figure 5 This is a schematic block diagram of a music style transfer device according to an embodiment of the present invention;

[0028] Figure 6 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] The music style transfer method provided in this embodiment of the invention can be applied to, for example... Figure 1 The application environment is shown. Specifically, this music style transfer method is applied in a music style transfer device, which includes, as shown in the example, a music style transfer device. Figure 1The client and server shown communicate over a network to address the issue of poor music style migration performance in existing technologies. The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The client, also known as the user terminal, refers to the program that provides categorized services to customers, corresponding to the server. The client can be installed on, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices.

[0031] In one embodiment, such as Figure 2 As shown, a music style transfer method is provided, which is then applied to... Figure 1 Taking the client as an example, the following steps are included:

[0032] S10: Obtain the music to be synthesized and the music to be transferred.

[0033] Understandably, the music to be synthesized is music whose rhythmic style has been altered, that is, music with a different rhythmic style has been transferred to it. The music to be transferred is music whose rhythmic style has been transferred to it. For example, in the application of a voice changer, a pop song can be transformed into a rock or hip-hop song; the pop song is the music to be synthesized, and the rock or hip-hop song is the music to be transferred. The music to be synthesized and the music to be transferred can be music from two completely different people (i.e., music from two different people), or music from the same person with different styles. Furthermore, the music to be synthesized and the music to be transferred can be obtained through web scraping technology from different websites or clients, or from different databases, or even sent by the user from the client to the server.

[0034] S20: Obtain a preset style decoupling model, input the music to be synthesized and the music to be transferred into the preset style decoupling model respectively, obtain the first pitch content code corresponding to the music to be synthesized, and obtain the second pitch content code and rhythm code corresponding to the music to be transferred.

[0035] Understandably, the predefined style decoupling model includes a rhythm category encoder, a pitch content encoder, a decoder, and a discriminator. This model is used to decouple pitch content and rhythmic style in music. The rhythm category encoder encodes the rhythmic style in the music, converting it to a hidden state. The pitch content encoder encodes the pitch content in the music, converting it to a hidden state. The decoder decodes and fuses the pitch content encoding and rhythm encoding.

[0036] Specifically, after obtaining the music to be synthesized and the music to be transferred, a preset style decoupling model is retrieved from the server. The music to be synthesized and the music to be transferred are input into the preset style decoupling model respectively. First, a rhythm category encoder is used to decouple the rhythm style of the music to be synthesized and the music to be transferred, that is, to determine the rhythm style based on the note values ​​in the music to be synthesized and the music to be transferred, thus obtaining the rhythm style corresponding to the music to be synthesized and the rhythm style corresponding to the music to be transferred. Then, the rhythm style is encoded to obtain the rhythm code corresponding to the music to be transferred. Next, a pitch content encoder is used to decouple the pitch content of the music to be synthesized and the music to be transferred, respectively, to obtain the first pitch content corresponding to the music to be synthesized and the second pitch content corresponding to the music to be transferred. The first pitch content and the second pitch content are encoded to obtain the first pitch content code corresponding to the music to be synthesized and the second pitch content code corresponding to the music to be transferred.

[0037] S30: Input the first pitch content encoding and the rhythm encoding into the first decoder to obtain the transferred music.

[0038] Understandably, the first decoder is a hardware or software device used to reconstruct an audio file from the encoded pitch and rhythm content. The first decoder includes a Tacotron2 architecture and an LPCNet network. The Tacotron2 architecture acts as a synthesizer to synthesize the encoded pitch and rhythm content and convert it into a spectrogram. The LPCNet network acts as a neural vocoder to reconstruct the waveform from the spectrogram. The transferred music is obtained by transferring the rhythmic style of the music to be transferred to the pitch content of the music to be synthesized.

[0039] Specifically, after obtaining the first pitch content code, the second pitch content code, and the rhythm code, the first pitch content code and the rhythm code are input into the first decoder, that is, the pitch content code of the music to be synthesized and the rhythm style code of the music to be transferred are input into the first decoder. First, the pitch content code of the music to be synthesized and the rhythm style code of the music to be transferred are decoded, that is, the first pitch content code and the rhythm code are converted from the hidden state to the spectrograms corresponding to the pitch content and rhythm style through the Tacotron2 architecture. Then, the LPCNet network is used to reconstruct the waveforms of the decoded spectrograms corresponding to the pitch content and rhythm style, thereby obtaining the transferred music.

[0040] S40, input the second pitch content encoding and the rhythm encoding into the second decoder to obtain the reconstructed music.

[0041] Understandably, the second decoder is a hardware or software device used to encode and restore the second pitch content and rhythm into an audio file, and its structure is the same as that of the first decoder. The reconstructed music is music rebuilt from the music to be transferred using a preset style decoupling model.

[0042] Specifically, after obtaining the target music, the second pitch content code and rhythm code corresponding to the music to be transferred are input into the second decoder. First, the rhythm style code and pitch content code of the music to be transferred are decoded. That is, the second pitch content code and rhythm code are converted from the hidden state to the spectrogram corresponding to the pitch content and rhythm style through the Tacotron2 architecture. The LPCNet network is used to reconstruct the waveform of the decoded pitch content and rhythm style spectrogram to obtain the reconstructed music.

[0043] S50: The style transfer detection result is obtained by detecting the transferred music and the reconstructed music according to the discriminator.

[0044] S60, when the style transfer detection result indicates that the rhythm style transfer is successful, the transferred music is identified as the target music.

[0045] Understandably, the style transfer detection result is used to characterize whether the rhythmic style of the music has been successfully transferred. The discriminator is used to detect whether the rhythmic style transfer in the target music has been successful, that is, to detect whether the rhythmic style of the target music is close to the rhythmic style of the music to be transferred. The target music is obtained by successfully transferring the rhythmic style of the music to be transferred to the pitch content of the music to be synthesized.

[0046] Specifically, after obtaining the target music and the reconstructed music, both are simultaneously input into the discriminator. The discriminator detects the loss between the target music and the reconstructed music, that is, it calculates the loss between the target music and the reconstructed music using a preset loss function, thus obtaining a style transfer loss value. The style transfer loss value is compared with a preset threshold. When the style transfer loss value is greater than the preset threshold, a style transfer detection result indicating a failure is obtained. When the style transfer loss value is less than or equal to the preset threshold, a style transfer detection result indicating a successful transfer is obtained. When the style transfer detection result indicates a successful rhythm style transfer, the target music is identified as the transferred music. When the style transfer detection result indicates a failed rhythm style transfer, the rhythm style transfer is performed again on the music to be synthesized and the music to be transferred, that is, the rhythm style of the music to be transferred is transferred to the pitch content of the music to be synthesized, and the judgment is repeated.

[0047] This invention provides a music style transfer method. The method uses a preset style decoupling model to decouple the synthesized music and the music to be transferred, thereby achieving complete decoupling of pitch content and rhythmic style in the music. This avoids the mixing of pitch content and rhythmic style, simplifying the music style decoupling process. The decoder in the preset style decoupling model performs style transfer and style reconstruction on the synthesized music and the music to be transferred, thus acquiring the transferred and reconstructed music. A discriminator detects the transferred music and the reconstructed music after style transfer, thereby determining the style transfer detection result and judging whether the music style transfer was successful. When the style transfer detection result indicates successful rhythmic style transfer, the transferred music is identified as the target music, thus achieving music style transfer, improving the effect of music style transfer, and enhancing the user experience.

[0048] In one embodiment, such as Figure 3 As shown, step S20, namely obtaining the first pitch content code corresponding to the music to be synthesized, and obtaining the second pitch content code and rhythm code corresponding to the music to be transferred, includes:

[0049] S201, The music to be transferred is input into the rhythm category encoder, and the rhythm category encoder performs rhythm style encoding on the music to be transferred to obtain the rhythm code.

[0050] Understandably, the rhythm category encoder is used to decouple the rhythm style of the music to be transferred. The rhythm encoding is the encoded result of the rhythm style of the music to be transferred.

[0051] Specifically, after obtaining the music to be transferred, it is input into a preset style decoupling model. The rhythm category encoder in this model decouples the rhythm style from the music to be transferred, obtaining the corresponding rhythm style. The rhythm category encoder encodes the rhythm style using a reference encoder and a diffusion model. Specifically, it first encodes the rhythm style using six 2D convolutional layers and a unidirectional 128-unit GRU layer, making the rhythm style encoding closer to the actual rhythm style of the music to be transferred. Then, the diffusion model performs noise diffusion and de-diffusion processing on the rhythm style encoding to remove noise components.

[0052] S202, the music to be migrated is input into the pitch content encoder, and the pitch content encoder performs pitch content encoding on the music to be migrated to obtain the second pitch content encoding.

[0053] Understandably, the pitch content encoder is used to decouple the pitch content of the music to be transferred and the music to be synthesized. The second pitch content encoder is the encoded result of the pitch content of the music to be transferred.

[0054] Specifically, after obtaining the music to be transferred, it is simultaneously input into a preset style decoupling model. The pitch content encoder in the preset style decoupling model decouples the pitch content of the music to be transferred, obtaining the second pitch content corresponding to the music to be transferred. The second pitch content is then encoded in the pitch content encoder using three long short-term memory networks, making the pitch content encoding closer to the actual pitch content of the music to be transferred, thus obtaining the second pitch content encoding.

[0055] S203, the music to be synthesized is input into the pitch content encoder, and the pitch content encoder performs pitch content encoding on the music to be synthesized to obtain the first pitch content encoding.

[0056] Understandably, the first pitch content is encoded as the result of encoding the pitch content of the music to be synthesized.

[0057] Specifically, after obtaining the second pitch content encoding, the music to be synthesized is input into a preset style decoupling model. The pitch content encoder in the preset style decoupling model decouples the pitch content in the music to be synthesized, obtaining the second pitch content corresponding to the music to be synthesized. In the pitch content encoder, the first pitch content is encoded through three long short-term memory networks, making the pitch content encoding closer to the actual pitch content of the music to be synthesized, thus obtaining the first pitch content encoding.

[0058] This invention decouples the rhythm category encoder and pitch content encoder in a preset style decoupling model to the music to be transferred, thereby determining the rhythm style and the second pitch content, and ultimately achieving complete decoupling of the rhythm style and pitch content in the music to be transferred. Then, the encoding functions in the rhythm category encoder and pitch content encoder encode the decoupled rhythm style, second pitch content, and first pitch content, thereby converting the rhythm style, second pitch content, and first pitch content into a hidden state, thus simplifying the music style decoupling process.

[0059] In one embodiment, step S20, namely obtaining the preset style decoupling model, includes:

[0060] S203, Obtain a sample training dataset, the sample training dataset including at least one sample training data; one sample training data is associated with one sample style feature.

[0061] Understandably, the sample training data includes a set of source music and target music. This sample training data can be collected from different websites or databases using web scraping techniques, or it can be sent by the user from the client to the server. The sample training dataset is then constructed based on all the acquired sample training data. A sample style feature is assigned to each sample training data point; this feature is used to characterize the standard music within the sample training data.

[0062] S204, Obtain a preset training model, and use the preset training model to perform prediction processing on the sample training data to obtain predicted style features.

[0063] Understandably, the preset training model is a model pre-set for style transfer on the sample training data. The predicted style features are obtained by transferring the rhythmic style of the standard music to the pitch content of the source music through the preset training model.

[0064] Specifically, a pre-set training model is retrieved from the server, and all sample training data is input into it. The rhythm category encoder and pitch content encoder in the pre-set training model decouple the source music and standard music included in the sample training data, respectively, to obtain the rhythm style and pitch content corresponding to the source music and the standard music. The rhythm category encoder and pitch content encoder in the pre-set training model encode the decoupled rhythm style and pitch content, respectively, thus transitioning them to the hidden state. The decoder in the pre-set training model decodes the pitch content encoding corresponding to the source music and the rhythm encoding corresponding to the standard music, thus transitioning them from the hidden state. The decoded pitch content corresponding to the source music and the rhythm style corresponding to the standard music are then fused to obtain the predicted style features.

[0065] S205, determine the prediction loss value of the sample training data based on the sample style features and the prediction style features corresponding to the same sample training data.

[0066] Understandably, the prediction loss is generated during the prediction of the predicted style features of the sample training data and is used to characterize the difference between the sample style features and the predicted style features.

[0067] Specifically, after obtaining the predicted style features, all sample style features corresponding to the sample training data are arranged according to the order of the sample training data in the sample training dataset. Then, the predicted style features associated with the sample training data are compared with the sample style features of the sample training data with the same sequence. That is, according to the sample training data, the sample style feature corresponding to the first sample training data is compared with the predicted style feature corresponding to the first sample training data, and the loss value between the sample style feature and the predicted style feature is determined by the loss function. Then, the sample style feature corresponding to the second sample training data is compared with the predicted style feature corresponding to the second sample training data, until all sample style features and predicted style features have been compared, and the predicted loss value of the sample training data can be determined.

[0068] S206, when the predicted loss value does not reach the preset convergence condition, iteratively update the initial parameters in the preset training model until the predicted loss value reaches the convergence condition, and record the converged preset training model as a preset style decoupling model.

[0069] Understandably, the convergence condition can be the condition that the predicted loss value is less than a set threshold, that is, when the predicted loss value is less than the set threshold, training stops; the convergence condition can also be the condition that the predicted loss value is very small after 500 calculations and will not decrease further, that is, when the predicted loss value is very small after 500 calculations and will not decrease further, training stops, and the preset training model after convergence is recorded as the preset style decoupling model.

[0070] Specifically, after determining the predicted loss value of the sample training data, if the predicted loss value does not reach the preset convergence condition, the initial parameters of the preset training model are adjusted according to the predicted loss value, and all sample training data are re-input into the preset training model after adjusting the initial parameters. When the predicted loss value of the sample training data reaches the preset convergence condition, other sample training data in the sample training dataset are selected, and the above steps S203 to S205 are executed to obtain the predicted loss value corresponding to the preset training model with adjusted initial parameters. If the predicted loss value does not reach the preset convergence condition, the initial parameters of the preset training model are adjusted again according to the predicted loss value, so that the predicted loss value of the preset training model with adjusted initial parameters reaches the preset convergence condition.

[0071] In this way, after training the preset training model with all sample training data, the output of the preset training model can continuously approach the accurate result, making the prediction accuracy higher and higher, until the prediction loss value of all sample training data reaches the preset convergence condition, and the converged preset training model is recorded as the preset style decoupling model.

[0072] This invention trains a pre-defined training model using a large amount of sample training data and determines the predicted loss value between the predicted style features and the sample style features using a pre-defined loss function. The initial parameters of the pre-defined training model are adjusted based on the predicted loss value until the model converges, thereby obtaining a pre-defined style decoupling model and ensuring that the pre-defined style decoupling model has a high prediction accuracy.

[0073] In one embodiment, such as Figure 4 As shown, in step S201, that is, the rhythm style encoding of the music to be transferred is performed by the rhythm category encoder to obtain the rhythm encoding, including:

[0074] S2011, The music to be transferred is encoded according to the reference encoder to obtain the reference rhythm code.

[0075] Understandably, the reference rhythm code is obtained by encoding the rhythmic style of the music to be transferred using the reference encoder. The reference encoder consists of six 2-bit convolutional layers and a unidirectional 128-unit GRU layer.

[0076] Specifically, after obtaining the music to be transferred, the rhythmic style of the music is encoded using the reference encoder in the rhythm category encoder. This means that the decoupled rhythmic style is first represented digitally, and then the digitized rhythmic style is sequentially processed by six two-dimensional convolutional layers to obtain the rhythmic convolution result. Then, a unidirectional 128-unit GRU layer encodes the rhythmic convolution result. Specifically, the GRU layer takes the output of the previous time step and the current input, outputs the information for the next time step, and uses the output of the GRU layer as the reference rhythmic code. The GRU layer includes a reset gate and an update gate. The reset gate is used to calculate the output of the candidate hidden layers, controlling how much of the previous hidden layer's information is retained. The update gate controls how much of the candidate hidden layer's output information is added, thus obtaining the output of the current hidden layer.

[0077] S2012, Based on the diffusion model, noise processing is performed on the reference rhythm code to obtain the rhythm code.

[0078] Understandably, the diffusion model is trained using a UNet network architecture on a large amount of image training data.

[0079] Specifically, Gaussian noise is added to the reference rhythm code in the first time step, and according to a specific diffusion step, the added Gaussian noise is diffused to the entire reference rhythm code in T-1 time steps. That is, the noise addition process lasts for T times, producing a series of noisy audio values ​​x1, x2, ..., x... t In the case of x t-1 Add noise to x t During the process, the standard deviation / variance of the noise is a fixed value β within the interval (0, 1). t The mean is determined by a fixed value β. t And the noise audio data at the current moment x t-1 This is determined by [the formula]. That is, the formula for this diffusion step is [formula]. q(x t |x t-1 ) is Gaussian noise from x t-1 To x t During the process, Gaussian noise follows a distribution. (referring to) β is the mean. t (The variance is a Gaussian distribution). Thus, it is only necessary to determine a fixed value β for each step based on the reference rhythm encoding. t This allows us to obtain the noise audio at each time step until the diffusion is complete at the last time step, at which point we can obtain the noise audio data.

[0080] Furthermore, a reverse diffusion process is performed on the noisy audio data, which is to denoise the noisy audio data, or to predict the noisy audio data based on the reference rhythm code. From q(x t-1 |x t Sampling from a random Gaussian distribution N(0,I) allows for the reconstruction of a true rhythm code from such a distribution, i.e., obtaining a true rhythm code from completely chaotic noisy audio data. Since it's necessary to find the data distribution within the noisy audio data, a model p is required. θ This approximates the conditional probability to complete the reverse diffusion process. The formula for this reverse diffusion is p. θ (x t-1 |x t ):=N(x t-1 μ θ (x t ,t),∑ θ (x t ,t)), where, This is a parameterized representation of the predicted average value. When performing denoising and prediction processing on noisy audio data, x needs to be parameterized according to the reference rhythm code (x0). t To x t-1 The prediction (q(x) t-1 |x t We perform conditional probability calculations on x0) to obtain rhythm codes.

[0081] This invention employs a reference encoder to encode the rhythmic style in the music to be transferred, thereby obtaining a reference rhythm code corresponding to the rhythmic style in the music to be transferred. By performing noise diffusion and de-diffusion processing on the reference rhythm code using a diffusion model, pitch content in the rhythmic style is removed, making the decoupled rhythmic style closer to the actual rhythmic style, thus determining the rhythm code.

[0082] In one embodiment, step S30, which involves inputting the first pitch content encoding and the rhythm encoding into the first decoder to obtain the transferred music, includes:

[0083] S301, the first pitch content encoding is processed cyclically using the cyclic consistency loss function to obtain the cyclic pitch content encoding.

[0084] Understandably, the cycle consistency loss function is used to preserve the pitch content in pitch content encoding and the rhythmic style in rhythm encoding. The expression for the cycle consistency loss function is: X s For the music to be synthesized, X g For music to be migrated, r s For the rhythmic style of the music to be synthesized, rg For the rhythmic style of the music to be transferred, For parameters of the rhythm category encoder, η is a parameter of the pitch content encoder. De These are the parameters for the decoder. To perform cyclic processing on the encoding of the first pitch content by a decoder with output probability p, and to retain the pitch content encoded in the first pitch content, The rhythm encoding is processed cyclically by a decoder with output probability p, preserving the rhythmic style of the rhythm encoding. The cyclic pitch content encoding is obtained by iteratively processing the first pitch content encoding.

[0085] Specifically, after obtaining the first pitch content code corresponding to the music to be synthesized, and the second pitch content code and rhythm code corresponding to the music to be transferred, the first pitch content is input into the first decoder. The first pitch content code is then processed cyclically using the cycle-consistent loss function in the first decoder. This means the cycle-consistent loss function is used to process the pitch content in the music to be synthesized, allowing the pitch content in the second pitch content code to be retained through a cyclic process. The cyclically processed first pitch content code is then regularized to avoid overfitting due to excessive iterations, and this cyclically processed first pitch content code is determined as the cyclic pitch content code.

[0086] S302, the rhythm code is processed cyclically using the cyclic consistency loss function to obtain the cyclic rhythm code.

[0087] Understandably, the cyclic rhythm encoding is obtained by iterating the rhythm encoding.

[0088] Specifically, after obtaining the cyclic pitch content encoding, the rhythm encoding is input into the first decoder. The rhythm encoding is then processed cyclically using the cycle-consistent loss function in the first decoder. This means the cycle-consistent loss function is used to process the rhythmic style of the music being transferred, allowing the rhythmic style in the rhythm encoding to be preserved through cyclic processing. Furthermore, the first pitch content encoding after cyclic processing is regularized to avoid overfitting due to excessive iterations, and the resulting cyclic rhythm encoding is determined as the cyclic rhythm encoding.

[0089] S303, perform style transfer based on the cyclic pitch content encoding and the cyclic rhythm encoding to obtain the transferred music.

[0090] Specifically, after obtaining the cyclic rhythm encoding, the cyclic pitch content encoding and cyclic rhythm encoding are processed by a spectrogram prediction network in the decoder to perform feature prediction, converting the input cyclic pitch content encoding and cyclic rhythm encoding into a frame sequence of Mel spectrograms. That is, the input cyclic pitch content encoding and cyclic rhythm encoding are converted into a 512-dimensional vector by the encoder, and then passed through a three-layer convolution, each layer containing 512 5x1 convolutional kernels, i.e., each kernel spans 5 characters. The convolutional layers model the context of the input vector and activate it using the ReLU function. The output of the last convolutional layer is fed into a bidirectional 256-unit LSTM layer to generate encoded features. A content attention mechanism is then used to maintain consistency as the encoded features move forward, while a positional attention mechanism uses 32 one-dimensional convolutional kernels of length 31 to obtain positional features. The encoded features and positional features are then projected onto a 128-dimensional hidden layer representation to calculate the attention weights. Finally, the output of a PreNet network with two fully connected layers consisting of 256 hidden ReLU units is concatenated with the attention context vector and passed to a stack of two unidirectional LSTM layers containing 1024 units. The LSTM output is then concatenated with the attention context vector and projected through a linear transformation to predict the target spectrogram frame. In other words, the target spectrogram frame passes through a 5-layer convolutional post-processing network, and its output is then concatenated with the residual output of the linear projection to obtain the Mel spectrogram.

[0091] Furthermore, the Mel spectrogram is input into the LPCNet network, which reconstructs the waveform of the Mel spectrogram. The input for feature calculation is the BFCC feature and two-dimensional pitch information; the output can be called frame features. The input for the sampling part is the output of the feature calculation part, the linear part of the current sampling point calculated by LPC, the previous sampling point, and the sampling network output (non-linear part) of the previous sampling point. The output is the non-linear part of the current point. Finally, the target music is obtained by adding the LPC-calculated value and the sampling network output value.

[0092] This invention employs a cyclic consistency loss function to perform cyclic processing on the first pitch content encoding and rhythm encoding, thereby preserving both pitch content and rhythmic style. Then, a spectrogram prediction network and an LPCNet network are used to perform feature prediction and waveform reconstruction on the cyclic pitch content encoding and cyclic rhythm encoding, respectively, thus acquiring the target music and achieving music rhythm style transfer. This further enhances the music style transfer effect and improves the user experience.

[0093] In one embodiment, step S40, which involves inputting the second pitch content code and the rhythm code into the second decoder to obtain the reconstructed music, includes:

[0094] S401, the second pitch content code is reconstructed using a reconstruction loss function to obtain the reconstructed pitch content code.

[0095] Understandably, the reconstruction loss function is X s For the music to be synthesized, X g For music to be migrated, z s For the pitch content of the music to be synthesized, r s For the rhythmic style of the music to be synthesized, z g For the pitch content of the music to be transferred, r g For the rhythmic style of the music to be transferred, To predict the second pitch content encoding using a decoder with output probability p, in order to reconstruct the pitch content of the music to be transferred, A decoder with output probability p predicts rhythm codes to reconstruct the rhythmic style of the music to be transferred. A reconstruction loss function is used to reconstruct the pitch content and rhythmic style of the music to be transferred. The reconstructed pitch content code is obtained by reconstructing the second pitch content code.

[0096] Specifically, after obtaining the second pitch content code and rhythm code corresponding to the music to be transferred, the second pitch content code is input into the second decoder. The second pitch content code is reconstructed by the reconstruction loss function in the second decoder. That is, the reconstruction loss function is used to process the pitch content in the music to be transferred, so that the difference between the pitch content in the music to be transferred and the reconstructed pitch content can be measured by reconstruction, thereby obtaining the reconstructed pitch content code.

[0097] S402, the rhythm code is reconstructed using a reconstruction loss function to obtain the reconstructed rhythm code.

[0098] S403, the reconstructed pitch content encoding and the reconstructed rhythm encoding are fused to obtain the reconstructed music.

[0099] Understandably, the reconstructed rhythm code is obtained by reconstructing the rhythm code.

[0100] Specifically, after obtaining the reconstructed pitch content code, the rhythm code is input into the second decoder. The rhythm code is then reconstructed using the reconstruction loss function in the second decoder. This reconstruction loss function applies a loss to the rhythmic style of the music to be transferred, allowing the difference between the rhythmic style in the music to be transferred and the reconstructed rhythm code to be measured through reconstruction, thus obtaining the reconstructed rhythm code. Further, the reconstructed pitch content code and the reconstructed rhythm code are processed by a spectrogram prediction network in the decoder, with the specific process being the same as step S303 above, and will not be repeated here. Further, the Mel spectrogram is input into the LPCNet network, and the waveform of the Mel spectrogram is reconstructed using the LPCNet network to obtain the reconstructed music, with the specific process being the same as step S303 above, and will not be repeated here.

[0101] This invention recursively processes the second pitch content encoding and rhythm encoding using a reconstruction loss function, thereby reconstructing the music to be transferred. Then, a spectrogram prediction network and an LPCNet network are used to perform feature prediction and waveform reconstruction on the reconstructed pitch content encoding and reconstructed rhythm encoding, respectively, thus acquiring the reconstructed music and further improving the effect of music style transfer, enhancing the user experience.

[0102] In one embodiment, step S50, namely, detecting the target music and the reconstructed music using a discriminator to obtain style transfer detection results, includes:

[0103] S501, the style transfer loss value is obtained by predicting the loss of the transferred music and the reconstructed music using an adversarial loss function.

[0104] Understandably, the style transfer loss value is used to characterize the difference between the target music and the reconstructed music. The adversarial loss function is used to detect the loss between the target music and the reconstructed music. The expression for the adversarial loss function is: X s For the music to be synthesized, X g For music to be migrated, z s For the pitch content of the music to be synthesized, r s For the rhythmic style of the music to be synthesized, z g For the pitch content of the music to be transferred, r g For the rhythmic style of the music to be transferred, η D For the parameters of the discriminator, The discriminator's loss detection results for the reconstructed music. This represents the loss detection result of the discriminator on the target music.

[0105] Specifically, after obtaining the target music and the reconstructed music, the target music and the reconstructed music are input into the discriminator. The discriminator uses the adversarial loss function to predict the loss between the target music and the reconstructed music. That is, first, the similarity of rhythm and style between the target music and the reconstructed music is calculated, such as Euclidean distance or cosine similarity. Then, the adversarial loss function is used to determine the loss between the target music and the reconstructed music, thereby obtaining the loss between the target music and the reconstructed music. The predicted loss value is determined as the style transfer loss value.

[0106] S502, compare the style transfer loss value with a preset threshold. When the style transfer loss value is less than or equal to the preset threshold, obtain a style transfer detection result that indicates successful transfer.

[0107] S503, when the style transfer loss value is greater than the preset threshold, a style transfer detection result representing the failure of the transfer is obtained.

[0108] Understandably, the preset threshold is a pre-set value used to determine whether the migration was successful.

[0109] Specifically, after obtaining the style transfer loss value, it is compared with a preset threshold. When the style transfer loss value is less than or equal to the preset threshold, it indicates that the rhythm style transfer of the music to be transferred is successful, thus obtaining a style transfer detection result indicating successful transfer. When the style transfer loss value is greater than the preset threshold, it indicates that the rhythm style transfer of the music to be transferred has failed, thus obtaining a style transfer detection result indicating failed transfer. Further, after obtaining the style transfer detection result indicating failed transfer, the music to be synthesized and the music to be transferred are decoupled again, and the pitch content of the music to be synthesized and the rhythm style of the music to be transferred are fused by the decoder to obtain updated music. The discriminator then performs loss detection again on the updated music and the reconstructed music to obtain the style transfer detection result.

[0110] Furthermore, if a style transfer detection result indicating successful transfer is not obtained more than twice, the preset style decoupling model is transmitted to a third-party platform. Staff obtain the preset style decoupling model from the third-party platform, perform tests on the preset style decoupling model, update the initial parameters of the preset style decoupling model, and feed the updated preset style decoupling model back to the server through the third-party platform. The rhythm style transfer is then performed again on the music to be synthesized and the music to be transferred, thereby obtaining a style transfer detection result indicating successful transfer.

[0111] This invention employs an adversarial loss function to detect the loss between the target music and the reconstructed music, thereby obtaining the style transfer loss value generated during rhythmic style transfer. By comparing the style transfer loss value with a preset threshold, the style transfer detection result is obtained, and thus, the success of the style transfer detection is determined.

[0112] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0113] In one embodiment, a music style transfer device is provided, which corresponds one-to-one with the music style transfer method described in the above embodiments. For example... Figure 5 As shown, the music style transfer device includes a music acquisition module 11, an encoding acquisition module 12, a first decoding module 13, a second decoding module 14, a detection result determination module 15, and a transfer success determination module 16. Detailed descriptions of each functional module are as follows:

[0114] The music acquisition module 11 is used to acquire music to be synthesized and music to be transferred; the music to be synthesized and the music to be transferred are different types of music.

[0115] The encoding acquisition module 12 is used to acquire a preset style decoupling model, input the music to be synthesized and the music to be transferred into the preset style decoupling model respectively, acquire the first pitch content code corresponding to the music to be synthesized, and acquire the second pitch content code and rhythm code corresponding to the music to be transferred.

[0116] The first decoding module 13 is used to input the first pitch content encoding and the rhythm encoding into the first decoder to obtain the transferred music;

[0117] The second decoding module 14 is used to input the second pitch content encoding and the rhythm encoding into the second decoder to obtain the reconstructed music;

[0118] The detection result determination module 15 is used to detect the transferred music and the reconstructed music according to the discriminator to obtain the style transfer detection result;

[0119] The migration success determination module 16 is used to determine the migrated music as the target music when the style migration detection result indicates that the rhythm style migration is successful.

[0120] Specific limitations regarding the music style transfer device can be found in the limitations of the music style transfer method described above, and will not be repeated here. Each module in the aforementioned music style transfer device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0121] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data used in the music style transfer method described in the above embodiments. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a music style transfer method.

[0122] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the music style transfer method described in the above embodiment.

[0123] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the music style transfer method described above.

[0124] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0125] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0126] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for music style transfer, characterized in that, include: Obtain the music to be synthesized and the music to be transferred; A preset style decoupling model is obtained. The music to be synthesized and the music to be transferred are respectively input into the preset style decoupling model to obtain the first pitch content code corresponding to the music to be synthesized, and the second pitch content code and rhythm code corresponding to the music to be transferred. The preset style decoupling model includes a rhythm category encoder, a pitch content encoder, a decoder and a discriminator. The first pitch content encoding and the rhythm encoding are input into the first decoder to obtain the transferred music; The second pitch content encoding and the rhythm encoding are input into the second decoder to obtain the reconstructed music; The style transfer detection results are obtained by detecting the transferred music and the reconstructed music using a discriminator. When the style transfer detection result indicates that the rhythm style transfer is successful, the transferred music is identified as the target music.

2. The music style transfer method as described in claim 1, characterized in that, The step of obtaining the first pitch content code corresponding to the music to be synthesized, and obtaining the second pitch content code and rhythm code corresponding to the music to be transferred, includes: The music to be transferred is input into the rhythm category encoder, and the rhythm category encoder performs rhythm style encoding on the music to be transferred to obtain the rhythm code; The music to be transferred is input into the pitch content encoder, and the pitch content encoder performs pitch content encoding on the music to be transferred to obtain the second pitch content encoding; The music to be synthesized is input into the pitch content encoder, and the pitch content encoder encodes the music to be synthesized to obtain the first pitch content code.

3. The music style transfer method as described in claim 2, characterized in that, The rhythm category encoder includes a reference encoder and a diffusion model; The step of performing rhythm style encoding on the music to be transferred using the rhythm category encoder to obtain the rhythm encoding includes: The music to be transferred is encoded according to the reference encoder to obtain the reference rhythm code; The reference rhythm code is processed for noise based on the diffusion model to obtain the rhythm code.

4. The music style transfer method as described in claim 1, characterized in that, The step of inputting the first pitch content encoding and the rhythm encoding into the first decoder to obtain the transferred music includes: The first pitch content code is processed cyclically using the cycle consistent loss function to obtain the cyclic pitch content code; The rhythm code is processed cyclically using a cycle-consistent loss function to obtain a cyclic rhythm code; The transferred music is obtained by performing style transfer based on the cyclic pitch content encoding and the cyclic rhythm encoding.

5. The music style transfer method as described in claim 1, characterized in that, The step of inputting the second pitch content encoding and the rhythm encoding into the second decoder to obtain the reconstructed music includes: The second pitch content code is reconstructed by using a reconstruction loss function to obtain the reconstructed pitch content code; The rhythm code is reconstructed by using a reconstruction loss function to obtain the reconstructed rhythm code; The reconstructed pitch content encoding and the reconstructed rhythm encoding are fused to obtain the reconstructed music.

6. The music style transfer method as described in claim 1, characterized in that, The step of detecting the transferred music and the reconstructed music using a discriminator to obtain style transfer detection results includes: The style transfer loss value is obtained by predicting the loss of the transferred music and the reconstructed music using an adversarial loss function. The style transfer loss value is compared with a preset threshold. When the style transfer loss value is less than or equal to the preset threshold, a style transfer detection result representing successful transfer is obtained. When the style transfer loss value is greater than the preset threshold, a style transfer detection result indicating a failed transfer is obtained.

7. The music style transfer method as described in claim 1, characterized in that, The process of obtaining the preset style decoupling model includes: Obtain a sample training dataset, wherein the sample training dataset includes at least one sample training data; One sample training data point is associated with one sample style feature; Obtain a preset training model, and use the preset training model to perform prediction processing on the sample training data to obtain predicted style features; The prediction loss value of the sample training data is determined based on the sample style features and the prediction style features corresponding to the same sample training data. When the predicted loss value does not reach the preset convergence condition, the initial parameters in the preset training model are iteratively updated until the predicted loss value reaches the convergence condition. Then, the preset training model after convergence is recorded as the preset style decoupling model.

8. A music style transfer device, characterized in that, include: The music acquisition module is used to acquire the music to be synthesized and the music to be transferred; the music to be synthesized and the music to be transferred are different pieces of music. The encoding acquisition module is used to acquire a preset style decoupling model, input the music to be synthesized and the music to be transferred into the preset style decoupling model respectively, acquire the first pitch content code corresponding to the music to be synthesized, and acquire the second pitch content code and rhythm code corresponding to the music to be transferred; wherein, the preset style decoupling model includes a rhythm category encoder, a pitch content encoder, a decoder and a discriminator; The first decoding module is used to input the first pitch content encoding and the rhythm encoding into the first decoder to obtain the transferred music; The second decoding module is used to input the second pitch content encoding and the rhythm encoding into the second decoder to obtain the reconstructed music; The detection result determination module is used to detect the transferred music and the reconstructed music based on the discriminator to obtain the style transfer detection result; The migration success determination module is used to determine the migrated music as the target music when the style migration detection result indicates that the rhythm style migration is successful.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the music style transfer method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the music style transfer method as described in any one of claims 1 to 7.

Images