Song-to-music-score conversion method, song-to-music-score conversion device and storage medium

Abstract

The embodiment of the invention discloses a song-to-music-score conversion method, a song-to-music-score conversion device and a storage medium, and is used for the technical field of audio processing. In the embodiment of the invention, a to-be-converted music score song audio is acquired; performing pitch prediction, audio boundary prediction and strength prediction on the audio of the song to be subjected to spectrum conversion through the target song spectrum conversion model to obtain a predicted pitch, a predicted audio boundary and a predicted strength; and synthesizing the voice score of the song audio to be converted based on the predicted pitch, the predicted audio boundary and the predicted strength. Synthesizing the voice score of the song audio to be converted into the music score through the prediction pitch, the prediction audio boundary and the prediction strength obtained through prediction of the target song music score conversion model; the prediction strength in the human voice score of the audio of the song to be converted can reflect the rhythm information of the audio of the song to be converted, so that the human voice score can have more rhythm, and the accuracy of the converted human voice score is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio processing technology, and in particular to a song music transcription method, a song music transcription device, and a storage medium. Background Art

[0002] Existing song score conversion technology refers to separating the human voice from the song audio, performing pitch detection on the separated human voice to determine the pitch value of each note, and determining the boundary information of the note (that is, the duration of the note); based on the pitch value of the note and the boundary information of the note, the human voice in the song audio is converted into discrete note representation to form the corresponding music score.

[0003] Existing song-to-score conversion technologies often use a song-to-score model. This model employs a multi-task learning approach, learning both note boundaries and corresponding pitch values ​​for each audio frame. This model then outputs predicted pitch values ​​and note boundaries during song conversion. However, existing song-to-score conversion models struggle to incorporate rhythmic information during multi-task learning, resulting in poor accuracy in the resulting vocal scores. Summary of the Invention

[0004] The embodiments of the present application provide a song music score conversion method, a song music score conversion device, and a storage medium, which effectively improve the accuracy of the converted vocal music score.

[0005] The present application provides a method for converting a song into music score, comprising:

[0006] Get the audio of the song to be transcribed;

[0007] Using the target song transcription model, the pitch prediction, audio boundary prediction and dynamics prediction are performed on the audio of the song to be transcribed, thereby obtaining the predicted pitch, predicted audio boundary and predicted dynamics;

[0008] Based on the predicted pitch, the predicted audio boundary and the predicted dynamics, a vocal score of the song audio to be transcribed is synthesized.

[0009] Furthermore, the method also includes

[0010] Obtaining a sample pitch value sequence and a sample velocity value sequence corresponding to a plurality of notes in a sample song audio, and obtaining annotation information of the sample song audio; the annotation information includes a pitch label, an audio boundary label, and a velocity label;

[0011] encoding and decoding the sample pitch value sequence through the first branch of the initial song transcription model to obtain a first sample predicted intensity, and determining a first loss value based on the first sample predicted intensity and the sample intensity value sequence;

[0012] The second branch of the initial song-to-song conversion model performs pitch prediction, audio boundary prediction, and dynamics prediction on the sample song audio to obtain a sample predicted pitch, a sample predicted audio boundary, and a second sample predicted dynamics;

[0013] Determine a second loss value based on the sample predicted pitch, the sample predicted audio boundary, the second sample predicted intensity, and the annotation information;

[0014] The initial song-to-score conversion model is optimized based on the first loss value and the second loss value until a convergence condition is reached, thereby obtaining a target song-to-score conversion model.

[0015] Furthermore, the first branch of the initial song-to-song conversion model includes: an autoencoder; encoding and decoding the sample pitch value sequence by the first branch of the initial song-to-song conversion model to obtain a first sample predicted intensity, and determining a first loss value based on the first sample predicted intensity and the sample intensity value sequence, including:

[0016] Inputting the sample pitch value sequence into the autoencoder to obtain a first sample predicted strength output by the autoencoder after encoding and decoding;

[0017] Calculate the mean square error loss between the first sample predicted strength and the sample strength value sequence to obtain the first loss value.

[0018] Furthermore, the autoencoder includes: an encoder and a decoder;

[0019] The step of inputting the sample pitch value sequence into the autoencoder to obtain the first sample predicted strength output by the autoencoder after encoding and decoding includes:

[0020] Inputting the sample pitch value sequence into the encoder to obtain an encoding vector output by the encoder;

[0021] The encoding vector is input into the decoder to obtain the decoded first sample prediction strength output by the decoder.

[0022] Furthermore, the second branch of the initial song-to-score conversion model includes: a preset network model; the second branch of the initial song-to-score conversion model performs pitch prediction, audio boundary prediction, and dynamics prediction on the sample song audio to obtain a sample predicted pitch, a sample predicted audio boundary, and a second sample predicted dynamics, including:

[0023] Extracting sample vocal audio from the sample song audio;

[0024] The audio features of the sample human voice audio are input into the preset network model to obtain the sample predicted pitch, sample predicted audio boundary and second sample predicted intensity corresponding to each audio frame in the sample human voice audio predicted and output by the preset network model.

[0025] Furthermore, determining the second loss value based on the sample predicted pitch, the sample predicted audio boundary, the second sample predicted intensity, and the annotation information includes:

[0026] Calculating a multi-classification loss between sample predicted pitches and pitch labels in a plurality of audio frames of the sample human voice audio to obtain a first multi-classification loss value;

[0027] Calculating a multi-classification loss between the predicted intensity of the second samples and the intensity labels in the plurality of audio frames to obtain a second multi-classification loss value;

[0028] Calculating a binary classification loss between sample predicted audio boundaries and audio boundary labels in the multiple audio frames to obtain a binary classification loss value;

[0029] The second loss value is obtained based on the first multi-class loss value, the second multi-class loss value, and the binary loss value.

[0030] Furthermore, the initial song-to-score conversion model is optimized based on the first loss value and the second loss value until a convergence condition is reached to obtain a target song-to-score conversion model, including:

[0031] Performing weighted fusion on the first loss value and the second loss value to obtain a target loss value;

[0032] The initial song-to-score conversion model is optimized based on the target loss value until the initial song-to-score conversion model converges, and it is determined that the training of the initial song-to-score conversion model is completed, thereby obtaining a target song-to-score conversion model.

[0033] Furthermore, the target song transcription model is used to perform pitch prediction, audio boundary prediction, and dynamics prediction on the song audio to be transcribed, to obtain predicted pitch, predicted audio boundary, and predicted dynamics, including:

[0034] Extracting the target vocal audio of the song to be transcribed;

[0035] The audio features of the target vocal audio are input into the target song spectrum conversion model to obtain the predicted pitch, predicted audio boundary and predicted intensity corresponding to each audio frame in the target vocal audio predicted and output by the target song spectrum conversion model.

[0036] The present application also provides a song music transcription device, comprising:

[0037] An acquisition unit, used to acquire the audio of the song to be transcribed;

[0038] A prediction unit, configured to perform pitch prediction, audio boundary prediction, and dynamics prediction on the audio of the song to be transcribed using a target song transcription model, to obtain a predicted pitch, a predicted audio boundary, and a predicted dynamics;

[0039] A synthesis unit is used to synthesize the vocal score of the song audio to be transcribed based on the predicted pitch, the predicted audio boundary and the predicted dynamics.

[0040] The present application also provides a training device for a song transcription model, including:

[0041] Processor, memory, and input and output interfaces;

[0042] The memory is a transient storage memory or a persistent storage memory;

[0043] The processor is configured to communicate with the memory and execute instructions in the memory to perform the above method.

[0044] An embodiment of the present application further provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute the method described above.

[0045] An embodiment of the present application further provides a computer program product comprising instructions or a computer program, wherein when the computer program product is run on a computer, the computer is caused to execute the method described above.

[0046] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0047] It can be seen that in the embodiment of the present application, the audio of the song to be transcribed is obtained; the pitch prediction, audio boundary prediction and dynamics prediction of the audio of the song to be transcribed are respectively performed on the audio of the song to be transcribed by the target song transcription model to obtain the predicted pitch, predicted audio boundary and predicted dynamics; based on the predicted pitch, predicted audio boundary and predicted dynamics, the vocal score of the audio of the song to be transcribed is synthesized. The predicted pitch, predicted audio boundary and predicted dynamics obtained by the target song transcription model are used to synthesize the vocal score of the audio of the song to be transcribed; the predicted dynamics in the vocal score of the audio of the song to be transcribed can reflect the rhythm information of the audio of the song to be transcribed, and can make the vocal score more rhythmic, so as to improve the accuracy of the transcribed vocal score. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0049] Figure 1 A schematic diagram of a communication architecture disclosed in an embodiment of the present application;

[0050] Figure 2 A flowchart of a song score conversion disclosed in an embodiment of the present application;

[0051] Figure 3 A training flow chart of a song transcription model disclosed in an embodiment of the present application;

[0052] Figure 4 A training flow chart of another song transcription model disclosed in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of a training method for a song transcription model disclosed in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of an autoencoder disclosed in an embodiment of the present application;

[0055] Figure 7 A schematic diagram of a song music transcription device disclosed in an embodiment of the present application;

[0056] Figure 8 This is a schematic diagram of another song transcription device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0058] In the description of the embodiments of the present application, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limiting the embodiments of the present application.

[0059] In the description of the embodiments of the present application, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in the embodiments of the present application based on the specific circumstances.

[0060] Song transcription technology Figure 1 As shown, the song-to-score transcribing device 101 can be connected to one or more song players 102. The song-to-score transcribing device 101 can separate the human voice from the song audio, perform pitch detection on the separated human voice to determine the pitch value of each note, and determine the boundary information of the note. Based on the pitch values ​​and boundary information of the note, the human voice in the song audio is converted into discrete note representations to form a corresponding music score, and the music score is provided to the song player 102. The music score can be a MIDI (Musical Instrument Digital Interface) music file containing information such as note boundaries and pitch values. The song player 102 can use the MIDI music file for singing synthesis, song scoring, etc.

[0061] In the existing song transcription device, a song transcription model is often used. The song transcription model adopts a multi-task learning method. For each audio frame of the song audio, the boundary information of the note is learned, and the corresponding pitch value information is learned at the same time. When the song is transcribed, the predicted pitch value and the boundary of the note are output through the song transcription model. However, it is difficult for the existing song transcription model to learn rhythm information in multi-task learning, and the accuracy of the transcribed vocal music score is poor. Therefore, a song transcription method is provided in the embodiment of the present application, which can effectively improve the accuracy of the transcribed vocal music score, such as Figure 2 As shown, the specific steps include:

[0062] 201. Obtain the audio of the song to be transcribed.

[0063] In an embodiment of the present application, the song transcription device can obtain the audio of the song to be transcribed, wherein the audio of the song to be transcribed is audio data containing at least human voice. The user can input the audio of the song to be transcribed, or the song audio of the song player can be collected through a network connection or the song audio sent by the song player can be received to obtain the audio of the song to be transcribed. The specific details are not limited here.

[0064] 202. Using the target song transcription model, pitch prediction, audio boundary prediction, and dynamics prediction are performed on the audio of the song to be transcribed, thereby obtaining predicted pitch, predicted audio boundary, and predicted dynamics.

[0065] After obtaining the audio of the song to be transcribed, the song transcription device can obtain the target song transcription model, such as by connecting to a network or pre-storing the target song transcription model, which is not limited here. The song transcription device can perform pitch prediction, audio boundary prediction and intensity prediction on the song audio to be transcribed through the target song transcription model, and obtain predicted pitch, predicted audio boundary and predicted intensity. Among them, the target song transcription model can predict the pitch value, note boundary and intensity value of each audio frame in the song audio to be transcribed, and obtain the predicted pitch, predicted audio boundary and predicted intensity of each audio frame. Among them, the pitch value refers to the height of the note sound, the intensity value refers to the strength of the note sound, and the note boundary is used to indicate whether the audio frame is the boundary of the note.

[0066] Specifically, the song transcription device can extract the target vocal audio from the song audio to be transcribed; for example, the dry sound extraction of the song audio to be transcribed can be performed through Adobe Audition software or a spleeter model to obtain the vocal audio; the vocal audio is the vocal singing audio in the song audio to be transcribed, and does not include the accompaniment of the song audio to be transcribed; wherein the duration of the extracted vocal audio is the same as the duration of the song audio to be transcribed.

[0067] After extracting the dry sound from the audio of the song to be transcribed, audio features are extracted, where the audio features can be mel-spectrogram features (mel-spectrogram features) or CQT (Constant Q Transform) spectrum features, which are not specifically limited here. The audio features of the target vocal audio can be input into the target song transcription model to obtain the predicted pitch, predicted audio boundary, and predicted dynamics corresponding to each audio frame in the target vocal audio output by the target song transcription model; that is, the target song transcription model predicts the pitch value, note boundary, and dynamics value of each audio frame of the target vocal audio in the song to be transcribed.

[0068] 203. Based on the predicted pitch, the predicted audio boundary, and the predicted dynamics, synthesize the vocal score of the song audio to be transcribed.

[0069] After obtaining the predicted pitch, predicted audio boundaries, and predicted dynamics of the song audio to be transcribed, the song-to-score model can synthesize a vocal score for the song audio to be transcribed based on the predicted pitch, predicted audio boundaries, and predicted dynamics. In other words, the song-to-score model can synthesize the pitch values, note boundaries, and dynamics values ​​of each audio frame of the target vocal audio in the song audio to be transcribed into a score for the corresponding audio frame, thereby obtaining a vocal score for the target vocal audio in the song audio to be transcribed.

[0070] It can be seen that in the embodiment of the present application, the audio of the song to be transcribed is obtained; the pitch prediction, audio boundary prediction and dynamics prediction of the audio of the song to be transcribed are respectively performed on the audio of the song to be transcribed by the target song transcription model to obtain the predicted pitch, predicted audio boundary and predicted dynamics; based on the predicted pitch, predicted audio boundary and predicted dynamics, the vocal score of the audio of the song to be transcribed is synthesized. The predicted pitch, predicted audio boundary and predicted dynamics obtained by the target song transcription model are used to synthesize the vocal score of the audio of the song to be transcribed; the predicted dynamics in the vocal score of the audio of the song to be transcribed can reflect the rhythm information of the audio of the song to be transcribed, which can make the vocal score more rhythmic, so as to improve the accuracy of the transcribed vocal score. In the output prediction data, the target song transcription model includes dynamics values ​​in addition to note boundaries and pitch values. Adding the dynamics values ​​output by the target song transcription model to the vocal score can make the vocal score more rhythmic, and the application scenarios of the vocal score are also more extensive, such as song adaptation and creation based on the dynamics values ​​in the vocal score.

[0071] Furthermore, it is difficult for existing song-to-score conversion models to learn rhythm information in multi-task learning, which easily leads to poor rhythmicity of the pitch values ​​and note boundaries predicted and output by the song-to-score conversion model. Therefore, the present application also provides a training method for a song-to-score conversion model, which can make the pitch values ​​and note boundaries predicted and output by the trained song-to-score conversion model more rhythmic, such as Figure 3 As shown, the details are as follows:

[0072] 301. Obtain sample pitch value sequences and sample velocity value sequences corresponding to multiple notes in the sample song audio, and obtain annotation information of the sample song audio.

[0073] In an embodiment of the present application, the song-to-score transcription device can obtain sample song audio, which is audio data containing at least human voice. The sample song audio can be obtained by collecting song audio from a song player or receiving song audio sent by a song player, the specific details of which are not limited herein. The sample song audio is training data for training the initial song-to-score transcription model. The number of sample song audio can be 900 or 1000, the specific details of which are not limited herein. To ensure training accuracy, the number of sample song audio should not be too small.

[0074] After obtaining the sample song audio, the sample pitch value sequence and sample velocity value sequence corresponding to the multiple notes in the sample song audio can be obtained; it can be understood that notes are the basic elements that constitute the audio, and the sample song audio contains multiple notes, each of which has a corresponding pitch value and velocity value. The pitch value and velocity value corresponding to each note in the sample song audio can be obtained to form a corresponding sample pitch value sequence and sample velocity value sequence. Among them, the sample pitch value sequence and sample velocity value sequence corresponding to the sample song audio can be obtained through software (such as Ableton Live and Cubase) or through programming and audio processing library (such as Python+librosa library). For example, the sample pitch value sequence corresponding to the sample song audio is [60,60,64,65,67,60,60], where each value represents the pitch value of a note.

[0075] In an embodiment of the present application, the song-to-score transcription device can also obtain annotation information for the sample song audio. This annotation information is pre-labeled label information for the sample song audio. This annotation information includes a pitch label, an audio boundary label, and a dynamics label. The pitch label contains the actual pitch value, the audio boundary label contains the actual audio boundary data, and the dynamics label contains the actual dynamics. The song-to-score transcription device can obtain annotation information for each audio frame in the sample song audio, that is, obtain the pitch label, audio boundary label, and dynamics label for each audio frame in the sample song audio.

[0076] 302. Encode and decode the sample pitch value sequence through the first branch of the initial song transcription model to obtain a first sample predicted intensity, and determine a first loss value based on the first sample predicted intensity and the sample intensity value sequence.

[0077] The song-to-song transcription device can obtain an initial song-to-song transcription model, encode and decode a sample pitch value sequence using a first branch of the initial song-to-song transcription model to obtain a first sample predicted dynamics, and determine a first loss value based on the first sample predicted dynamics and the sample dynamics value sequence. Specifically, the first loss value can be determined by calculating a mean square error or a mean absolute error between the first sample predicted dynamics and the sample dynamics value sequence.

[0078] By encoding and decoding the sample pitch value sequence, a first sample predicted intensity is obtained. The first sample predicted intensity is more discriminative than the original sample pitch value sequence and can better reflect the essential characteristics of the pitch value. The first loss value is determined based on the first sample predicted intensity and the sample intensity value sequence. The song transcription model obtained by training with the first loss value can more accurately learn the relationship between pitch value and intensity value.

[0079] 303. Perform pitch prediction, audio boundary prediction, and intensity prediction on the sample song audio through the second branch of the initial song-to-song conversion model to obtain a sample predicted pitch, a sample predicted audio boundary, and a second sample predicted intensity.

[0080] The song-to-score device can use the second branch of the initial song-to-score model to perform pitch prediction, audio boundary prediction, and dynamics prediction on the sample song audio, respectively, to obtain a sample predicted pitch, a sample predicted audio boundary, and a second sample predicted dynamics. Specifically, the sample song audio can be input into the second branch of the initial song-to-score model to obtain the sample predicted pitch, the sample predicted audio boundary, and the second sample predicted dynamics corresponding to each audio frame of the vocal audio of the sample song audio output by the second branch.

[0081] 304. Determine a second loss value based on the sample predicted pitch, the sample predicted audio boundary, the second sample predicted intensity, and the annotation information.

[0082] The song transcription device can determine the second loss value based on the sample predicted pitch, the sample predicted audio boundary, the second sample predicted intensity and the annotation information. Specifically, the song transcription device can obtain the corresponding pitch value classification probability, note boundary classification probability and intensity value classification probability based on the sample predicted pitch, sample predicted audio boundary and second sample predicted intensity corresponding to each audio frame in the human voice audio. That is, the sample predicted pitch, sample predicted audio boundary and second sample predicted intensity corresponding to each audio frame can be mapped through the classification function to obtain the pitch value classification probability, note boundary classification probability and intensity value classification probability corresponding to each audio frame. The pitch value classification probability is the probability of the category to which the pitch value belongs, the note boundary classification probability is the probability of whether it is a note boundary, and the intensity value classification probability is the probability of the category to which the intensity value belongs.

[0083] Among them, the annotation information includes pitch labels, audio boundary labels and velocity labels. After obtaining the pitch value classification probability, note boundary classification probability and velocity value classification probability, the loss value of the pitch value classification probability and the pitch label, the loss value of the note boundary classification probability and the audio boundary label, and the loss value of the velocity value classification probability and the velocity label can be calculated to obtain the second loss value.

[0084] It is understandable that the execution order of step 303 and step 304 and step 302 is not specifically limited here, that is, the first loss value can be determined first, or the second loss value can be determined first.

[0085] 305. Optimize the initial song-to-score conversion model based on the first loss value and the second loss value until a convergence condition is reached, thereby obtaining a target song-to-score conversion model.

[0086] After obtaining the first loss value and the second loss value, the song-to-score conversion device can optimize the initial song-to-score conversion model based on the first loss value and the second loss value until convergence conditions are reached, thereby obtaining a target song-to-score conversion model. The target song-to-score conversion model is used to convert the song audio into a musical score.

[0087] Specifically, the first loss value and the second loss value can be weightedly fused to obtain a target loss value; the initial song-to-score conversion model can be optimized based on the target loss value until the initial song-to-score conversion model converges, and the training of the initial song-to-score conversion model is determined to be completed, thereby obtaining a target song-to-score conversion model.

[0088] Determine whether the initial song-to-score conversion model converges based on the target loss value. Among them, when the target loss value is less than a preset threshold, it can be determined that the initial song-to-score conversion model converges. The preset threshold can be understood as, when the target loss value is expressed using JS divergence, the preset threshold can be 0.4 or 0.5, which is not specifically limited here. If the initial song-to-score conversion model does not converge, another sample song audio can be used to continue training the initial song-to-score conversion model. If converged, the training is completed and the target song-to-score conversion model is obtained. The target song-to-score conversion model is used to input the song audio to be transcribed, output the predicted pitch, predicted audio boundary and predicted intensity to obtain the corresponding vocal music score. That is, the song-to-score conversion model learns rhythm information through the intensity value, so that the note boundaries and pitch values ​​predicted and output by the song-to-score conversion model are more regular and rhythmic, so that the obtained music score is more regular and rhythmic.

[0089] It can be seen that in the embodiment of the present application, the initial song transcription model is trained by using the first loss value determined by the sample pitch value sequence and the sample velocity value sequence corresponding to the sample song audio, and the second loss value determined by the sample predicted pitch, sample predicted audio boundary and second sample predicted velocity output by the initial song transcription model. This allows the song transcription model to learn the relationship between pitch value and velocity value when predicting note boundaries and pitch values. The velocity value can reflect the rhythmic information of the song audio, so that the pitch value and note boundary predicted and output by the trained song transcription model are more rhythmic.

[0090] In one practicable approach, existing music scores have high requirements for the accuracy of note boundaries and pitch values. If the accuracy is insufficient, singing synthesis will be out of tune, words and sounds will not align, and song scoring may result in users singing correctly but being scored incorrectly. In the embodiment of the present application, the song transcription model learns the relationship (i.e., the correspondence) between pitch values ​​and dynamic values ​​during training, which can effectively improve the accuracy of the pitch values ​​and note boundaries predicted and output by the trained song transcription model, and can better provide accurate music scores for singing synthesis and song scoring.

[0091] Furthermore, the following will be combined Figure 4, the training process of the song transcription model in this application is described in detail as follows:

[0092] 401. Obtain sample pitch value sequences and sample velocity value sequences corresponding to multiple notes in the sample song audio, and obtain annotation information of the sample song audio.

[0093] It is understandable that step 401 is similar to the above step 301 and will not be described in detail here.

[0094] 402. Input the sample pitch value sequence into the autoencoder, calculate the mean square error loss between the first sample predicted force output by the autoencoder and the sample force value sequence, and obtain a first loss value.

[0095] In an embodiment of the present application, the first branch of the initial song-to-score model includes: an autoencoder; after obtaining a sample pitch value sequence and a sample velocity value sequence of the sample song audio, the sample pitch value sequence can be input into the autoencoder, and the mean square error loss between the first sample predicted velocity output by the autoencoder and the sample velocity value sequence is calculated to obtain a first loss value. Wherein, the autoencoder can be a fully connected autoencoder network or a convolutional autoencoder network, which is not specifically limited here. By inputting the sample pitch value sequence into the autoencoder, the first sample predicted velocity output by the autoencoder is obtained. The first sample predicted velocity is more discriminative than the original pitch value sequence and can better reflect the essential characteristics of the pitch value; the mean square error loss between the first sample predicted velocity and the velocity value sequence is calculated to obtain a first loss value. The song-to-score model trained with the first loss value can more accurately learn the relationship between pitch value and velocity value.

[0096] Specifically, the autoencoder includes: an encoder and a decoder; Figure 6 As shown, the sample pitch value sequence can be input into the encoder to obtain the encoding vector (i.e., the intermediate vector Code) output by the encoder; that is, the encoder can map the input sample pitch value sequence to a low-dimensional embedding feature space, learn the characteristic distribution of the pitch value sequence, and achieve dimensionality reduction of the pitch value sequence to retain the key information of the pitch value sequence. Next, the encoding vector is input into the decoder to obtain the decoded first sample prediction intensity output by the decoder; that is, the decoder can restore the encoding vector to the first sample prediction intensity (i.e., the embedding vector) based on the learned characteristic distribution. The length of the first sample prediction intensity is the same as the length of the pitch value sequence.

[0097] Calculate the mean square error loss between the predicted strength of the first sample output by the encoder and the strength value sequence. The corresponding mean square error loss formula is:

[0098]

[0099] Among them, MSE loss is the first loss value of the mean square error loss. There are n notes in the song audio, i is one of the notes, decoder_i is the predicted velocity of the first sample of the i-th note output by the autoencoder, and velocity_i represents the velocity value of the corresponding i-th note.

[0100] 403. Input the audio features of the sample vocal audio of the sample song audio into a preset network model to obtain the sample predicted pitch, sample predicted audio boundary and second sample predicted intensity for each audio frame in the sample vocal audio predicted and output by the preset network model.

[0101] In an embodiment of the present application, the second branch of the initial song spectrum conversion model includes: a preset network model. The preset network model can be a convolutional recurrent neural network (CRNN), a residual network (ResNet), or a self-attention network (Transformers), and the specific details are not limited here. The audio features of the sample vocal audio of the sample song audio can be input into the preset network model to obtain the sample predicted pitch, sample predicted audio boundary, and second sample predicted intensity for each audio frame in the sample vocal audio predicted and output by the preset network model.

[0102] Specifically, the sample vocal audio of the sample song audio can be extracted, and the sample vocal audio is the vocal singing audio in the sample song audio, and does not include the accompaniment of the sample song audio. After extracting the dry sound from the sample song audio, the audio features are extracted to obtain the audio features of the sample vocal audio; the audio features can be Mel spectrum features (mel spectrum features) or CQT (Constant Q Transform) spectrum features, which are not limited here. The audio features of the sample vocal audio are input into the preset network model to obtain the sample predicted pitch, sample predicted audio boundary and second sample predicted intensity corresponding to each audio frame in the sample vocal audio predicted and output by the preset network model.

[0103] 404. Calculate a multi-classification loss between the sample predicted pitch and the pitch label, a multi-classification loss between the second sample predicted intensity and the intensity label, and a binary classification loss between the sample predicted audio boundary and the audio boundary label to determine a second loss value.

[0104] After the sample predicts pitch, the sample predicts audio boundary and the second sample predicts intensity, the song transcription device can calculate the multi-classification loss of the sample predicted pitch and the pitch label, the multi-classification loss of the second sample predicted intensity and the intensity label, and the binary classification loss of the sample predicted audio boundary and the audio boundary label to determine the second loss value.

[0105] Specifically, based on the sample predicted pitch, the second sample predicted intensity, and the sample predicted audio boundary corresponding to each audio frame in the sample human voice audio, the pitch value classification probability, intensity value classification probability, and note boundary classification probability corresponding to each audio frame in the sample human voice audio can be obtained. Among them, for the pitch value classification probability, a multi-classification function: Softmax function can be used for mapping to obtain the probability of the category to which the pitch value belongs; it can be understood that the pitch value category generally has 128 categories, and correspondingly, the probability that the pitch value of the audio frame output by the initial song transcription model belongs to these 128 categories can be obtained. For the intensity value classification probability, the intensity value generally includes 128 categories, and the predicted data of the intensity value can be mapped by the multi-classification function: Softmax function to obtain the probability of the category to which the intensity value belongs. For the note boundary classification probability, the note boundary includes: the note starting point and the note ending point; that is, the initial song transcription model can output the predicted data of the note starting point and the note ending point corresponding to each audio frame in the vocal audio, and the predicted data can be an embedding vector; for the predicted data of the note starting point and the note ending point, a binary classification function: sigmoid function can be used for mapping to obtain the probability of whether it is a note starting point and the probability of whether it is a note ending point.

[0106] Next, a multi-classification loss between the pitch value classification probability and the pitch label, a multi-classification loss between the velocity value classification probability and the velocity label, and a binary classification loss between the note boundary classification probability and the audio boundary label can be calculated to determine a second loss value. This specifically includes the following steps 4041 to 4044:

[0107] 4041. Calculate a multi-classification loss between sample predicted pitches and pitch labels in multiple audio frames of the sample human voice audio to obtain a first multi-classification loss value.

[0108] In this embodiment of the present application, the multi-classification loss between the predicted pitch and the pitch label in multiple audio frames of the sample human voice audio can be calculated, that is, the multi-classification loss between the pitch value classification probability and the pitch label in multiple audio frames can be calculated to obtain a first multi-classification loss value. The corresponding multi-classification loss formula is:

[0109]

[0110] Among them, Loss pitch is the first multi-classification loss value corresponding to the pitch value, where i in y_ij represents the i-th audio frame, there are n audio frames in total, j represents the j-th category of the pitch value, there are k categories of pitch values ​​in total; y_ij' represents the pitch value classification probability, and y_ij represents the pitch label.

[0111] 4042. Calculate a multi-classification loss between the predicted intensity of the second sample and the intensity label in the multiple audio frames to obtain a second multi-classification loss value.

[0112] In this embodiment of the present application, the multi-classification loss of the predicted intensity of the second sample in multiple audio frames and the intensity label can be calculated to obtain a second multi-classification loss value, and the multi-classification loss of the intensity value classification probability and the intensity label in multiple audio frames can be calculated to obtain a second multi-classification loss value. The corresponding multi-classification formula is:

[0113]

[0114] Among them, Loss velocity is the second multi-classification loss value corresponding to the intensity value, where i in y_ij represents the i-th audio frame, there are n audio frames in total, j represents the j-th category of the intensity value, and there are k categories of intensity values ​​in total; y_ij' represents the intensity value classification probability, and y_ij represents the intensity label.

[0115] 4043. Calculate the binary classification loss of the sample predicted audio boundary and the audio boundary label in multiple audio frames to obtain the binary classification loss value.

[0116] In an embodiment of the present application, the binary classification loss between the sample predicted audio boundary and the audio boundary label in multiple audio frames can be calculated to obtain a binary classification loss value; that is, the binary classification loss corresponding to the probability of whether the multiple audio frames are note starting points and the binary classification loss corresponding to the probability of whether the multiple audio frames are note ending points can be calculated to obtain a binary classification loss value. Among them, the binary classification loss between the probability of whether the multiple audio frames are note starting points and the label of whether the multiple audio frames are note starting points can be calculated to obtain a first binary classification loss value; the binary classification loss between the probability of whether the multiple audio frames are note ending points and the label of whether the multiple audio frames are note ending points can be calculated to obtain a second binary classification loss value; the first binary classification loss value and the second binary classification loss value are used as the final binary classification loss value.

[0117] Specifically, the binary classification loss between the probability of whether a note starts in multiple audio frames and the label of whether a note starts can be calculated. The corresponding binary classification loss formula is:

[0118]

[0119] Among them, Loss onset It is the first two-class classification loss value between the probability of whether it is the starting point of a note and the label of whether it is the starting point of a note. There are n audio frames in the human voice audio. y_i represents the label of the i-th audio frame, 1 represents the starting point, 0 represents not the starting point, and y_i' represents the probability of whether the i-th audio frame is the starting point of a note.

[0120] Specifically, the binary classification loss between the probability of whether a note ends in multiple audio frames and the label of whether a note ends is calculated. The corresponding binary classification loss formula is:

[0121]

[0122] Among them, Loss offset The second two-category loss value between the probability of whether it is a note end point and the label of whether it is a note end point, y_i represents the label of the i-th audio frame, 1 represents the end point, 0 represents not the end point, y_i' represents the probability of whether the i-th audio frame is a note end point.

[0123] 4044. Obtain a second loss value based on the first multi-class loss value, the second multi-class loss value, and the binary loss value.

[0124] In the embodiment of the present application, the second loss value can be obtained based on the first multi-classification loss value, the second multi-classification loss value and the binary classification loss value. pitch , the second multi-classification loss value corresponding to the strength value Loss velocity , the first and second classification loss values ​​corresponding to the note starting point Loss onset And the second classification loss value Loss corresponding to the end point of the note offset , as the second loss value.

[0125] 405. Perform weighted fusion on the first loss value and the second loss value to obtain a target loss value, until the initial song-to-score conversion model converges to obtain a target song-to-score conversion model.

[0126] In the embodiment of the present application, after obtaining the first loss value and the second loss value, the first loss value and the second loss value can be weighted and fused to obtain a target loss value. When the initial song-to-score conversion model converges, it is determined that the training of the initial song-to-score conversion model is completed and the target song-to-score conversion model is obtained. Specifically, the first loss value and the second loss value can be weighted and fused to obtain the target loss value. The corresponding formula is:

[0127] Loss total =a*MSE loss +b*Loss onset +c*Loss offset +d*Loss pitch +e*Loss_velocity

[0128] Among them, Loss totalis the target loss value after weighted fusion, and a, b, c, d, and e are the corresponding weights. The target loss value is trained until the initial song-to-score model converges to obtain the target song-to-score model.

[0129] In an practicable manner, the song-to-score model can also predict the output octave value to improve the accuracy of the predicted output of the song-to-score model. The corresponding song-to-score model training process is as follows: Figure 5 As shown in the figure, CRNNmodel (CRNN network) is the preset network model, FC is the fully connected layer, and the preset network model contains multiple fully connected layers. Different target parameters can be set in multiple fully connected layers to output different prediction data, such as predicted pitch, predicted audio boundary, predicted intensity, and predicted octave. Among them, the multi-classification loss corresponding to the predicted octave can be calculated. The corresponding multi-classification loss formula is:

[0130]

[0131] Among them, Loss octave is the third multi-class classification loss value corresponding to the octave value. In y_ij, i represents the i-th audio frame, there are n audio frames, j represents the j-th category of the octave value, and there are k categories of octave values. y_ij' represents the classification probability of the octave value, and y_ij represents the octave label.

[0132] Correspondingly, the first loss value, the second loss value, and the third multi-classification loss value can be weighted and fused to obtain the target loss value; the corresponding formula is:

[0133] Loss total =a*MSE loss +b*Loss onset +c*Loss offset +d*Loss pitch +e*Loss_velocity+f*Loss octave

[0134] Among them, Loss total is the target loss value after weighted fusion, and a, b, c, d, e, and f are the corresponding weights. The trained song transcription model can predict the output data of the note starting point, note ending point, octave value, pitch value, and velocity value.

[0135] The present application also provides a song music transcription device. Figure 7 As shown, including:

[0136] An acquisition unit 701 is used to acquire the audio of the song to be transcribed;

[0137] A prediction unit 702 is configured to perform pitch prediction, audio boundary prediction, and dynamics prediction on the audio of the song to be transcribed using a target song transcription model to obtain a predicted pitch, a predicted audio boundary, and a predicted dynamics;

[0138] The synthesis unit 703 is used to synthesize the vocal music score of the song audio to be transcribed based on the predicted pitch, the predicted audio boundary and the predicted dynamics.

[0139] The present application also provides a song music transcription device 800. Figure 8 As shown, the song transcription device 800 of the embodiment of the present application may include one or more central processing units (CPUs) 801 and a memory 802 , in which one or more application programs or data are stored.

[0140] Memory 802 may be volatile or persistent storage. The program stored in memory 802 may include one or more modules, each of which may include a series of instruction operations on the electronic device. Furthermore, the central processing unit 801 may be configured to communicate with memory 802 and execute the series of instruction operations in memory 802 on the song transcription device 800.

[0141] The song transcription device 800 may also include one or more power supplies 805, one or more wired or wireless network interfaces 804, one or more input and output interfaces 803, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0142] The central processing unit 801 can execute the operations performed by the aforementioned first aspect or any specific method embodiment of the first aspect, and the details will not be repeated here.

[0143] An embodiment of the present application further provides a computer-readable storage medium, which includes instructions. When the instructions are executed on a computer, the computer executes the method described above.

[0144] An embodiment of the present application further provides a computer program product comprising instructions or a computer program, which enables the computer to execute the method described above when the computer program product is run on the computer.

[0145] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0147] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0148] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0149] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A song music transcription method, characterized in that: include: Get the audio of the song to be transcribed; Using the target song transcription model, the pitch prediction, audio boundary prediction and dynamics prediction are performed on the audio of the song to be transcribed, thereby obtaining the predicted pitch, predicted audio boundary and predicted dynamics; Based on the predicted pitch, the predicted audio boundary and the predicted dynamics, a vocal score of the song audio to be transcribed is synthesized.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining a sample pitch value sequence and a sample velocity value sequence corresponding to a plurality of notes in a sample song audio, and obtaining annotation information of the sample song audio; the annotation information includes a pitch label, an audio boundary label, and a velocity label; encoding and decoding the sample pitch value sequence through the first branch of the initial song transcription model to obtain a first sample predicted intensity, and determining a first loss value based on the first sample predicted intensity and the sample intensity value sequence; The second branch of the initial song-to-song conversion model performs pitch prediction, audio boundary prediction, and dynamics prediction on the sample song audio to obtain a sample predicted pitch, a sample predicted audio boundary, and a second sample predicted dynamics; Determine a second loss value based on the sample predicted pitch, the sample predicted audio boundary, the second sample predicted intensity, and the annotation information; The initial song-to-score conversion model is optimized based on the first loss value and the second loss value until a convergence condition is reached, thereby obtaining a target song-to-score conversion model.

3. The method according to claim 2, characterized in that The first branch of the initial song-to-song transcription model includes an autoencoder. The first branch of the initial song-to-song transcription model encodes and decodes the sample pitch value sequence to obtain a first sample predicted intensity, and determines a first loss value based on the first sample predicted intensity and the sample intensity value sequence, including: Inputting the sample pitch value sequence into the autoencoder to obtain a first sample predicted strength output by the autoencoder after encoding and decoding; Calculate the mean square error loss between the first sample predicted strength and the sample strength value sequence to obtain the first loss value.

4. The method according to claim 3, characterized in that The autoencoder includes: an encoder and a decoder; The step of inputting the sample pitch value sequence into the autoencoder to obtain the first sample predicted strength output by the autoencoder after encoding and decoding includes: Inputting the sample pitch value sequence into the encoder to obtain an encoding vector output by the encoder; The encoding vector is input into the decoder to obtain the decoded first sample prediction strength output by the decoder.

5. The method according to claim 2, characterized in that The second branch of the initial song-to-score conversion model includes: a preset network model; performing pitch prediction, audio boundary prediction, and dynamics prediction on the sample song audio through the second branch of the initial song-to-score conversion model to obtain a sample predicted pitch, a sample predicted audio boundary, and a second sample predicted dynamics, including: Extracting sample vocal audio from the sample song audio; The audio features of the sample human voice audio are input into the preset network model to obtain the sample predicted pitch, sample predicted audio boundary and second sample predicted intensity corresponding to each audio frame in the sample human voice audio predicted and output by the preset network model.

6. The method according to claim 5, characterized in that The determining of the second loss value based on the sample predicted pitch, the sample predicted audio boundary, the second sample predicted intensity, and the annotation information includes: Calculating a multi-classification loss between sample predicted pitches and pitch labels in a plurality of audio frames of the sample human voice audio to obtain a first multi-classification loss value; Calculating a multi-classification loss between the predicted intensity of the second samples and the intensity labels in the plurality of audio frames to obtain a second multi-classification loss value; Calculating a binary classification loss between sample predicted audio boundaries and audio boundary labels in the multiple audio frames to obtain a binary classification loss value; The second loss value is obtained based on the first multi-class loss value, the second multi-class loss value, and the binary loss value.

7. The method according to claim 2, characterized in that The optimizing the initial song-to-score conversion model based on the first loss value and the second loss value until a convergence condition is reached to obtain a target song-to-score conversion model includes: Performing weighted fusion on the first loss value and the second loss value to obtain a target loss value; The initial song-to-score conversion model is optimized based on the target loss value until the initial song-to-score conversion model converges, and it is determined that the training of the initial song-to-score conversion model is completed, thereby obtaining a target song-to-score conversion model.

8. The method according to claim 1, characterized in that The method of performing pitch prediction, audio boundary prediction, and dynamics prediction on the audio of the song to be transcribed by using the target song transcription model to obtain predicted pitch, predicted audio boundary, and predicted dynamics includes: Extracting the target vocal audio of the song to be transcribed; The audio features of the target vocal audio are input into the target song spectrum conversion model to obtain the predicted pitch, predicted audio boundary and predicted intensity corresponding to each audio frame in the target vocal audio predicted and output by the target song spectrum conversion model.

9. A song music transcription device, characterized in that: include: An acquisition unit, used to acquire the audio of the song to be transcribed; A prediction unit, configured to perform pitch prediction, audio boundary prediction, and dynamics prediction on the audio of the song to be transcribed using a target song transcription model, to obtain a predicted pitch, a predicted audio boundary, and a predicted dynamics; A synthesis unit is used to synthesize the vocal score of the song audio to be transcribed based on the predicted pitch, the predicted audio boundary and the predicted dynamics.

10. A song music transcription device, characterized in that: include: Processor, memory, and input and output interfaces; The memory is a transient storage memory or a persistent storage memory; The processor is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that The method comprises instructions which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 8.

12. A computer program product comprising instructions or a computer program, characterized in that When the computer program product is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 8.

Images